Object Storage Configuration

Object storage is used for storing large binary files such as debugging symbols and other artifacts. zymtrace supports S3-compatible storage solutions including MinIO, AWS S3, and Google Cloud Storage.

Configuration Modes

Use Existing

Connect to existing S3-compatible storage

This mode connects to your existing MinIO, AWS S3, or Google Cloud Storage.

MinIO

MinIO Configuration

MinIO Configuration

MinIO is a high-performance, S3-compatible object storage solution that can be deployed on-premises or in the cloud. Configure your existing MinIO instance with the following settings:

storage:
  mode: "use_existing"
  use_existing:
    type: "minio"
    minio:
      endpoint: "" # must be a url, http/s
      user: ""
      password: ""
  buckets:
    symbols: "zymtrace-symdb"

Required fields:

• endpoint: Complete URL to your MinIO server (e.g., https://minio.example.com or http://192.168.1.100:9000)
• user: MinIO access key
• password: MinIO secret key

Bucket Creation

mc mb minio/zymtrace-symdb

Testing Connectivity

mc config host add minio http://minio.company.com:9000 ACCESS_KEY SECRET_KEY
mc ls minio/zymtrace-symdb

Configuration Example

Development Environment

storage:
  mode: "use_existing"
  use_existing:
    type: "minio"
    minio:
      endpoint: "http://minio.dev.company.com:9000"
      user: "dev-access-key"
      password: "dev-secret-key"
  buckets:
    symbols: "zymtrace-symdb-dev"

AWS S3

AWS S3 Configuration

AWS S3 Configuration

storage:
  mode: "use_existing"
  use_existing:
    type: "s3"
    s3:
      region: ""
      accessKey: ""
      secretKey: ""
  buckets:
    symbols: "zymtrace-symdb"

Required fields:

• region: AWS region where your S3 bucket is located (e.g., us-west-2)
• accessKey: AWS access key ID
• secretKey: AWS secret access key

IAM Permissions for S3

Your AWS user or role needs the following permissions for the specified bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::zymtrace-symdb",
                "arn:aws:s3:::zymtrace-symdb/*"
            ]
        }
    ]
}

Bucket Creation

aws s3 mb s3://zymtrace-symdb --region us-west-2

Testing Connectivity

aws s3 ls s3://zymtrace-symdb --region us-west-2

Configuration Example

Production Environment

storage:
  mode: "use_existing"
  use_existing:
    type: "s3"
    s3:
      region: "us-west-2"
      accessKey: "AKIA..."
      secretKey: "..."
  buckets:
    symbols: "zymtrace-symdb-prod"

Google Cloud Storage

Google Cloud Storage Configuration

Google Cloud Storage Configuration

storage:
  mode: "use_existing"
  use_existing:
    type: "gcs"
    gcs:
      endpoint: "https://storage.googleapis.com" # GCS endpoint, defaults to https://storage.googleapis.com
      accessKey: ""
      secretKey: ""
  buckets:
    symbols: "zymtrace-symdb"

Required fields:

• accessKey: GCP service account access key
• secretKey: GCP service account secret key

GCS Service Account Permissions

Your GCP service account needs the following permissions for the specified bucket:

• storage.objects.create
• storage.objects.delete
• storage.objects.get
• storage.objects.list
• storage.buckets.get

You can assign the Storage Object Admin role for the specific bucket, or create a custom role with the minimal required permissions.

Bucket Creation

gsutil mb -c STANDARD -l us-west1 gs://zymtrace-symdb

Testing Connectivity

gsutil ls gs://zymtrace-symdb

Configuration Example

Production Environment

storage:
  mode: "use_existing"
  use_existing:
    type: "gcs"
    gcs:
      endpoint: "https://storage.googleapis.com"
      accessKey: "[email protected]"
      secretKey: "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
  buckets:
    symbols: "zymtrace-symdb-prod"

Bucket Configuration

Bucket Names

zymtrace uses the following bucket for storing symbols:

• symbols: Stores debugging symbols and related metadata (default: zymtrace-symdb)

Bucket Creation

Automatic Creation: If you have appropriate permissions, zymtrace can automatically create the required bucket during initialization.

Manual Creation: For production environments, create the bucket manually using your cloud provider's console or CLI tools. See the specific instructions in each storage provider tab above.

Security Considerations

Access Control

• Principle of Least Privilege: Grant only the minimum required permissions
• Network Security: Use HTTPS/TLS for connections to object storage
• Access Keys: Use dedicated service accounts with limited scope

Data Encryption

• Encryption in Transit: Always use HTTPS for object storage connections
• Encryption at Rest: Enable server-side encryption on your buckets
• Key Management: Use managed encryption keys when possible

Troubleshooting

Common Issues

1. Connection Timeout: Check network connectivity and firewall rules
2. Access Denied: Verify credentials and permissions
3. Bucket Not Found: Ensure the bucket exists and is accessible
4. SSL/TLS Errors: Verify certificate configuration for HTTPS endpoints

Next Steps

After configuring object storage, proceed to configure the other storage components:

Configure ClickHouse | Configure Postgres

Create Mode

Deploy new MinIO instance

This mode deploys and manages MinIO within your cluster.

MinIO Create Mode Configuration

storage:
  mode: "create"
  create:
    image:
      repository: minio/minio
      tag: "RELEASE.2024-12-18T13-15-44Z"
    config:
      user: "minio"
      password: "minio123"
    service:
      api:
        port: 9000
      console:
        port: 9001
    replicas: 1
    resources:
      requests:
        cpu: "200m"
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "1Gi"
    storage:
      type: "persistent"
      size: 20Gi
      className: ""
  buckets:
    symbols: "zymtrace-symdb"