Install Zymtrace Profiler

The Zymtrace profiler is a lightweight, OpenTelemetry-compliant agent that collects performance profiles from your applications and systems with minimal overhead (<1% CPU and ~256MB RAM ). It can be deployed in various ways to suit your infrastructure needs. This guide covers installation using Kubernetes manifests, Helm charts, Docker containers, or direct binary installation.

Install with an AI assistant

Prefer natural language? Use the Zymtrace skills in Claude Code, OpenAI Codex, or Cursor to install the profiler for you — e.g. "install the Zymtrace profiler with GPU support". The assistant runs the right workflow (Helm, Docker, or binary) and verifies the agent is reporting.

tip

Please review the prerequisites before beginning, particularly if you are working in an airgapped environment.

Installation methods

Choose the installation method that best suits your environment. Each method provides the same functionality with different deployment characteristics.

Kubectl

Install with Kubectl

The profiler agent is deployed as a DaemonSet.

1. Create a namespace

   kubectl create namespace zymtrace

2. Deploy

   kubectl apply -n zymtrace -f https://helm.zystem.io/k8s-manifests/profiler/deploy.yaml

tip

Important: If you need GPU profiling capabilities, we recommend using the Helm installation method instead, which provides more configuration options for enabling GPU profiling.

tip

Collection Agent Configuration

By default, the collection agent is set to zymtrace-gateway.zymtrace.svc.cluster.local:80.

Remember that the collection agent should point to the Zymtrace gateway service. If you're installing the profiling agent on a different cluster than the one hosting the backend services, you'll need to modify this setting. In this case, we recommend downloading the configuration file first.

curl -O https://helm.zystem.io/k8s-manifests/profiler/deploy.yaml

Helm

Install with Helm

Helm Chart Source

The Helm chart source code is available on GitHub: zystem-io/zymtrace-charts

# Add the zystem repository
helm repo add Zymtrace https://helm.zystem.io

# List available charts and versions
helm search repo Zymtrace --versions

# Install
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace Zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80" \
 --set profiler.args[1]="--disable-tls" \
 --set profiler.args[2]="--project=colossus" \
 --set profiler.args[3]="--tags=prod;us-east" \
 --set "profiler.env.HTTPS_PROXY=http://username:password@proxy:port"

Authenticating with a service token

If service token authentication is enabled on the backend, the profiler needs a token to report profiles. Rather than passing it as the -auth-token CLI arg (which stores the token in plain text in the pod spec), the chart sources it from a Kubernetes Secret and exposes it to the agent as the ZYMTRACE_AUTH_TOKEN environment variable.

Chart-managed Secret

Provide the token as a plain value. When profiler.authToken.value is set and no existing Secret is referenced, the chart creates a Secret named zymtrace-profiler-auth-token (key auth-token) to hold it and keeps the token out of the pod spec:

helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace Zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80" \
 --set profiler.authToken.value="your-service-token-here"

Existing Secret

Reference a Secret you create and manage yourself. profiler.authToken.existingSecret takes precedence over profiler.authToken.value:

# Create the Secret (key defaults to auth-token)
kubectl create secret generic my-zymtrace-token \
 --namespace Zymtrace \
 --from-literal=auth-token="your-service-token-here"

helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace Zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80" \
 --set profiler.authToken.existingSecret.name=my-zymtrace-token \
 --set profiler.authToken.existingSecret.key=auth-token

profiler.authToken.existingSecret.key defaults to auth-token, so you can omit it when your Secret uses that key.

warning

Avoid the -auth-token=<token> CLI arg (i.e. profiler.args) in Helm deployments — it stores the token in plain text in the rendered pod spec. Use profiler.authToken instead.

Enabling GPU Profiling

To enable GPU profiling capabilities with Helm, add the cudaProfiler.enabled setting. This deploys the profiling libraries for both NVIDIA (CUDA) and AMD (ROCm) workloads.

GPU metrics collection is enabled by default.

# Install with GPU profiling and metrics enabled
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace Zymtrace \
 --set profiler.cudaProfiler.enabled=true \
 --set profiler.args[0]="--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80" \
 --set profiler.args[1]="--disable-tls" \
 --set profiler.args[2]="--enable-vllm-metrics"

This will automatically:

• Deploy the necessary GPU profiling libraries
• Configure the volume mounts for sharing libraries between containers
• Extract the profiling libraries to the shared host path
• Make GPU profiling available to all containers that mount the shared volume
• Collect GPU metrics (power usage, memory utilization, temperature, and performance metrics), which is on by default

To connect your GPU workloads, see the GPU Profiler Quick Start: CUDA applications use injection, ROCm applications are started with the AMD profiling environment variables.

tip

GPU metrics collection is enabled by default; set -enable-gpu-metrics=false if you only want CUDA profiling without metrics collection. You can profile CUDA applications without collecting GPU metrics.

You can also collect only GPU metrics without profiling.

The --enable-vllm-metrics flag enables automatic metrics collection for vLLM-based LLM inference applications. The metrics are correlated with GPU Profilers.

NVML Library Path: Add --nvml-auto-scan to detect the NVML library automatically, or --nvml-path=/path/to/libnvidia-ml.so if you know the exact path. This is only required for GPU metrics collection.

Using custom-values.yaml

Using custom values

You can create a custom values file with your configurations:

profiler:
 args:
   - "--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80"  # Point to your gateway service
   - "--disable-tls"
   - "--enable-vllm-metrics"
   - "--project=my-project-123"
   - "--tags=prod;ha100;fp16"

 # Enable GPU profiling
 cudaProfiler:
   enabled: true
   hostMountPath: "/var/lib/zymtrace/profiler"  # Default path

 env:
   HTTPS_PROXY: "http://user:[email protected]:8080"

Then install using:

helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace Zymtrace \
 -f custom-values.yaml

Next Step: Hook up your CUDA application

After enabling the GPU profiling module in the profiler, connect it to your CUDA application by referring to the GPU Profiler documentation.

Docker

Install with Docker

docker run --cgroupns=host --pid=host --privileged --net=host \
-v /etc/machine-id:/etc/machine-id:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
--rm -d --name zymtrace-profiler ghcr.io/zystem-io/zymtrace-pub-profiler:26.9.2 \
--disable-tls --collection-agent <host>:8080

tip

Be sure to update the collection agent value to point to your gateway service.

Enabling GPU Profiling

To enable GPU profiling capabilities with Docker, mount an additional volume. GPU metrics are collected by default.

docker run --cgroupns=host --pid=host --privileged --net=host \
-v /etc/machine-id:/etc/machine-id:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/lib/zymtrace/profiler:/opt/zymtrace-cuda-profiler \
--rm -d --name zymtrace-profiler ghcr.io/zystem-io/zymtrace-pub-profiler:26.9.2 \
--disable-tls \
--collection-agent <host>:8080

This will automatically:

• Extract the necessary CUDA and ROCm profiler libraries to the host path (/var/lib/zymtrace/profiler)
• Make GPU profiling available to your GPU workloads
• Enable GPU metrics collection (on by default)

tip

GPU metrics collection is enabled by default; set -enable-gpu-metrics=false if you only want CUDA profiling without metrics collection. You can profile CUDA applications without collecting GPU metrics.

You can also collect only GPU metrics without profiling.

NVML Library Path: Add --nvml-auto-scan to detect the NVML library automatically, or --nvml-path=/path/to/libnvidia-ml.so if you know the exact path. This is only required for GPU metrics collection.

To verify the profiler libraries were extracted correctly:

# Check the contents of the shared volume directory
ls -la /var/lib/zymtrace/profiler

# You should see
# libzymtracecudaprofiler.so
# and, on x86_64 hosts, the AMD libraries:
# libzymtracerocmprofiler.so, librocprofiler-sdk.so.1

Next Step: Hook up your GPU application

After enabling the GPU profiling module in the profiler, connect your CUDA application by referring to the GPU Profiler documentation, or start your ROCm application with the AMD profiling environment variables.

Binary

Install with Binary

x86_64

x86_64 Installation

# Download the binaries
curl -LO https://dl.zystem.io/zymtrace/26.9.2/amd64/zymtrace-profiler.tar.gz

# Extract them
sudo tar -xzvf zymtrace-profiler.tar.gz -C / --no-same-owner

# Run the agent (GPU metrics are collected by default)
sudo /opt/zymtrace/profiler/zymtrace-profiler --collection-agent localhost:8080 --disable-tls

Enabling GPU Profiling

Unlike Helm and Docker installations, there are no explicit steps to enable GPU profiling with the binary installation. You can directly start profiling your CUDA application (for ROCm applications, use the AMD profiling environment variables instead):

env RUST_LOG="zymtracecudaprofiler=info" \
   CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
   python -u matmul.py

tip

GPU metrics collection is enabled by default; set -enable-gpu-metrics=false if you only want CUDA profiling without metrics collection. You can profile CUDA applications without collecting GPU metrics.

You can also collect only GPU metrics without profiling.

NVML Library Path: Add --nvml-auto-scan to detect the NVML library automatically, or --nvml-path=/path/to/libnvidia-ml.so if you know the exact path. This is only required for GPU metrics collection.

ARM64

ARM64 Installation

# Download the binaries
curl -LO https://dl.zystem.io/zymtrace/26.9.2/arm64/zymtrace-profiler.tar.gz

# Extract them
sudo tar -xzvf zymtrace-profiler.tar.gz -C / --no-same-owner

# Run the agent (GPU metrics are collected by default)
sudo /opt/zymtrace/profiler/zymtrace-profiler --collection-agent localhost:8080 --disable-tls

tip

GPU metrics collection (power, memory, temperature, performance) is enabled by default on systems with GPUs. Set -enable-gpu-metrics=false if you don't need GPU metrics collection.

NVML Library Path: Add --nvml-auto-scan to detect the NVML library automatically, or --nvml-path=/path/to/libnvidia-ml.so if you know the exact path. This is only required for GPU metrics collection.

tip

Be sure to update the --collection-agent value to point to your zymtrace gateway service.

Management

These commands help you monitor and maintain your zymtrace profiler installation. Use them to check the agent's status and logs.

Kubectl

Kubectl Management

# Check agent status
kubectl get pods -n Zymtrace -l app=zymtrace,component=profiler

# View agent logs
kubectl logs -f -n Zymtrace -l app=zymtrace,component=profiler

Helm

Helm Management

Preserve your values on upgrade

When using --set (e.g. updating image tags), always add --reset-then-reuse-values to avoid resetting your chart values to defaults.

• Use --reset-then-reuse-values when upgrading with --set (e.g. image tags)
• Use -f <values-file> when doing a full deployment — no reuse flag needed in that case

# Check profiler status (not agent)
helm status profiler -n Zymtrace

# Upgrade charts, force a new image pull
helm upgrade profiler zymtrace/profiler -n Zymtrace --set global.imagePullPolicy=Always --reset-then-reuse-values \
 --debug \
 --atomic

# Get pod status
kubectl get pods -n Zymtrace -l app=zymtrace,component=profiler

# Upgrade profiler (not agent)
helm upgrade profiler zymtrace/profiler -n Zymtrace --reset-then-reuse-values \
 --debug \
 --atomic

# View profiler values (not agent)
helm get values profiler -n Zymtrace

# Uninstall profiler (not agent)
helm uninstall profiler -n Zymtrace

Docker

Docker Management

# View running agents
docker ps | grep zymtrace/profiler

# View agent logs
docker logs <container-id>

# Stop agent
docker stop <container-id>

# Remove agent container
docker rm <container-id>

Systemd

Setting up systemd service

To manage the profiler as a systemd service:

1. Create the zymtrace directory:

   sudo mkdir -p /opt/zymtrace

2. Move the downloaded agent binary to the installation directory:

   sudo mv zymtrace-profiler /opt/zymtrace/

3. Make the binary executable:

   sudo chmod +x /opt/zymtrace/zymtrace-profiler

4. Create a systemd service file:

   sudo vi /etc/systemd/system/zymtrace.service

5. Copy and paste the following configuration into the file:

   [Unit]
   Description=zymtrace profiler service
   After=network.target
   
   [Service]
   Type=simple
   ExecStart=/opt/zymtrace/zymtrace-profiler --collection-agent localhost:8080 --disable-tls
   Restart=always
   RestartSec=10
   WorkingDirectory=/opt/zymtrace
   
   [Install]
   WantedBy=multi-user.target

GPU Metrics Collection

GPU metrics collection is enabled by default and includes power usage, memory utilization, temperature, and performance metrics. For more details, see the GPU Metrics documentation.

Note: For NVIDIA GPU metrics, add --nvml-auto-scan to the ExecStart line to detect the NVML library automatically, or --nvml-path=/path/to/libnvidia-ml.so if you know the exact path.

tip

Be sure to update the --collection-agent value in the systemd service file to point to your zymtrace gateway service.

6. Enable the service

   # Reload daemon
   sudo systemctl daemon-reload
   
   # Start agent
   sudo systemctl start Zymtrace
   
   # Enable agent start on boot
   sudo systemctl enable Zymtrace
   
   # Check agent status
   sudo systemctl status Zymtrace

Management commands

Once the systemd service is set up, use these commands to manage the profiler:

# Check systemd service status
sudo systemctl status Zymtrace

# View service logs
sudo journalctl -u Zymtrace -f

# Restart service
sudo systemctl restart Zymtrace

# Stop service
sudo systemctl stop Zymtrace