Skip to main content

On-premise symbolization

Overview​

The zymtrace global symbolization service mirrors and processes debuginfo from Linux distro repos and serves the processed symbols via our CDN at https://symbols.zymtrace.com. The backend services automatically look up native symbols through this CDN, enabling efficient symbolization with zero configuration required from users.

For customers who do not have direct access to the internet, you can either:

  1. Allow firewall access to https://symbols.zymtrace.com (the easiest approach), or
  2. Clone the entire bucket locally into an S3-compatible storage.

Since zymtrace already ships with MinIO, you can use this as your local s3-compatible blob storage. The majority of our customers operate in air-gapped environments, so we provide several methods to clone the symbol bucket.

Configuration​

To use on-premise symbolization, update the Helm configuration:

custom-values.yaml
globalSymbolization:
  enabled: false
  config:
    bucketName: ""
    accessKey: ""
    secretKey: ""
    region: ""
    endpoint: ""
info

region is specific to AWS S3. We will automatically construct the URL without explicitly setting an endpoint.

but first, you need to clone the bucket.

Clone the bucket​

This guide provides commands and procedures for cloning the zystem-symbolblobs bucket from Google Cloud Storage. The bucket contains symbol files used by zymtrace for providing function names and line numbers in application traces.

Source bucket: https://zystem-symbolblobs.storage.googleapis.com

Recommended destination bucket: zymtrace-symbols

info

This bucket contains ~1TB of data and growing. Ensure your destination storage has sufficient capacity (2-3 TB) and expect the transfer to take several hours depending on your network bandwidth.

Requirements​

  • Google Cloud SDK installed
  • Sufficient storage space

Overview​

gsutil is Google's command-line tool for working with Cloud Storage. This method is particularly efficient for GCS-to-GCS transfers.

Installation​

# Install Google Cloud SDK
curl https://sdk.cloud.google.com | bash

# Initialize and authenticate
gcloud init

Command​

# Create destination bucket if needed
gsutil mb -c standard -l us-central1 gs://zymtrace-symbols

# Clone with maximum performance
gsutil -o "GSUtil:parallel_process_count=2" \
  -o "GSUtil:parallel_thread_count=5" \
  -m rsync -r gs://zystem-symbolblobs gs://zymtrace-symbols

Performance Considerations​

  • The -m flag enables multi-threading, critical for TB-scale transfers
  • parallel_process_count spawns multiple processes for parallel operations
  • parallel_thread_count uses multiple threads per process
  • Adjust these values based on your system's CPU cores and available memory
  • For memory-constrained systems, reduce these values to prevent out-of-memory errors

Monitoring Progress​

During the transfer, gsutil will display progress statistics:

Copying gs://zystem-symbolblobs/file1.pb [1/1000 files][ 10.5 MiB/ 10.5 MiB]
...

You can also use a second terminal to check the destination bucket size:

gsutil du -sh gs://zymtrace-symbols

Error Handling​

If the transfer is interrupted, simply run the same command again. The rsync operation will resume where it left off, copying only files that weren't successfully transferred.

After Cloning​

Once you've successfully cloned the symbol bucket, update your custom-values.yaml with the appropriate bucket information and deploy your zymtrace instance. The system will now use your local symbol storage instead of the internet-hosted CDN.