Storage configuration

zymtrace uses ClickHouse, Scylla and S3-Compatible Object Storage as its primary storage backends. For a deeper understanding of how these components interact, refer to the architecture section.

To accommodate different operational needs, zymtrace provides two deployment modes:

1. Create Mode (create): Deploys and manages the storage service within your cluster. This is ideal for quick setup and seamless integration with zymtrace.
2. Existing Mode (use_existing): Connects to your existing ClickHouse, Scylla, or S3 storage, eliminating the need to manage additional infrastructure.

This guide walks you through configuring ClickHouse, Scylla, and S3-compatible storage in both modes.

Configuring storage backends

1. ClickHouse

zymtrace supports two modes for ClickHouse integration:

• create: Deploy a new ClickHouse instance.
• use_existing: Connect to an existing ClickHouse instance.

Update the custom_values.yaml file as follows:

custom-values.yaml

clickhouse:
  mode: "create"  # "create" or "use_existing"
  create:
    image:
      repository: clickhouse/clickhouse-server
      tag: "24.12.2"
    config:
      user: "clickhouse"
      password: "clickhouse123"
      database: "zymtrace"
    service:
      http:
        port: 8123
      native:
        port: 9000
    storage:
      type: "persistent"  # "persistent" or "empty_dir"
      size: 30Gi
  use_existing:
    host: ""  # host:nativePort
    user: ""
    password: ""
    database: ""
    secure: false  # Enable TLS/secure connection

Set mode to either create or use_existing, depending on your setup.

---

2. Scylla

Scylla integration supports similar modes:

• create: Deploy a new Scylla instance.
• use_existing: Connect to an existing Scylla cluster.

Configuration in custom_values.yaml:

custom-values.yaml

scylla:
  mode: "create"  # "create" or "use_existing"
  create:
    image:
      repository: scylladb/scylla
      tag: "6.2.2"
    config:
      user: "scylla"
      password: "scylla"
    service:
      port: 9042
    storage:
      type: "persistent"
      size: 20Gi
  use_existing:
    hosts: []  # List of nodes in the format: "host1:port1;host2:port2"
    user: ""
    password: ""
    secure: false  # Enable TLS/secure connection

Ensure that the hosts field is populated for the use_existing mode with node details.

---

3. S3-Compatible Object Storage

Object storage is configurable for:

• MinIO (self-hosted storage)
• AWS S3 or any S3-compatible storage.

To enable S3-compatible storage, configure the custom_values.yaml file:

custom-values.yaml

storage:
  mode: "create"  # "create" or "use_existing"
  create:
    image:
      repository: minio/minio
      tag: "RELEASE.2024-12-18T13-15-44Z"
    config:
      user: "minio"
      password: "minio123"
    service:
      api:
        port: 9000
  use_existing:
    type: "minio"  # "minio" or "s3"
    minio:
      endpoint: "" # must be a url, http/s
      user: ""
      password: ""
    s3:
      region: ""
      accessKey: ""
      secretKey: ""
  buckets:
    symbols: "zymtrace-symdb"

For S3-compatible storage, ensure the accessKey and secretKey are set.

---

Applying the configuration

Once you’ve updated custom_values.yaml with the appropriate configuration, deploy the backend using Helm:

helm upgrade backend zymtrace/backend  -f custom_values.yaml