Install zymtrace profiler

The zymtrace profiler is a lightweight, OpenTelemetry compliant agent that collects performance profiles from your applications and systems with minimal overhead (<1% CPU and ~256MB RAM ). It can be deployed in various ways to suit your infrastructure needs. This guide covers installation using Kubernetes manifests, Helm charts, Docker containers, or direct binary installation.

tip

Please review the prerequisites before beginning, particularly if you are working in an airgapped environment.

Installation methods

Choose the installation method that best suits your environment. Each method provides the same functionality with different deployment characteristics.

The Add Data page provides a guided approach to generating the deployment commands based on your environment.

Kubectl

Install with Kubectl

The profiler agent is deployed as a DaemonSet.

1. Create a namespace

   kubectl create namespace zymtrace

2. Deploy

   kubectl apply -n zymtrace -f https://helm.zystem.io/k8s-manifests/profiler/zymtrace-profiler.yaml

tip

Important: If you need GPU profiling capabilities, we recommend using the Helm installation method instead, which provides more configuration options for enabling GPU profiling.

tip

Collection Agent Configuration

By default, the collection agent is set to `zymtrace-ui.zymtrace.svc.cluster.local:80.

Remember that the collection agent should point to your UI/gateway service. If you're installing the profiling agent on a different cluster than the one hosting the backend services, you'll need to modify this setting. In this case, we recommend downloading the configuration file first.

curl -O https://helm.zystem.io/k8s-manifests/profiler/zymtrace-profiler.yaml

Helm

Install with Helm

# Add the zystem repository
helm repo add zymtrace https://helm.zystem.io

# List available charts and versions
helm search repo zymtrace --versions

# Install
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-ui.zymtrace.svc.cluster.local:80" \
 --set profiler.args[1]="--disable-tls" \
 --set profiler.args[2]="--project=colossus" \
 --set profiler.args[3]="--tags=prod;us-east" \
 --set "profiler.env.HTTPS_PROXY=http://username:password@proxy:port"

Enabling GPU Profiling

To enable GPU profiling capabilities with Helm, add the cudaProfiler.enabled setting:

# Install with GPU profiling enabled
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-ui.zymtrace.svc.cluster.local:80" \
 --set profiler.args[1]="--disable-tls" \
 --set profiler.cudaProfiler.enabled=true

This will automatically:

• Deploy the necessary GPU profiling libraries
• Configure the volume mounts for sharing libraries between containers
• Extract the profiling libraries to the shared host path
• Make GPU profiling available to all containers that mount the shared volume

important

GPU profiling is only available on AMD64/x86_64 architecture.

Using custom-values.yaml

Using custom values

You can create a custom values file with your configurations:

profiler:
 args:
   - "--collection-agent=zymtrace-ui.zymtrace.svc.cluster.local:80"  # Point to your UI/gateway service
   - "--disable-tls"
   - "--project=my-project-123"
   - "--tags=prod;ha100;fp16"
   
 # Enable GPU profiling
 cudaProfiler:
   enabled: true
   hostMountPath: "/var/lib/zymtrace/profiler"  # Default path
   
 env:
   HTTPS_PROXY: "http://user:[email protected]:8080"
   ZYM: "TRACE"

   fieldRefs:
     NODE_NAME: true
     KUBERNETES_NODE_NAME: true
     POD_NAMESPACE: metadata.namespace

Then install using:

helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 -f custom-values.yaml

For more details on using the GPU profiler, see the GPU Profiler documentation.

Docker

Install with Docker

docker run --pid=host --privileged --net=host \
-v /etc/machine-id:/etc/machine-id:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
--rm -d --name zymtrace-profiler ghcr.io/zystem-io/zymtrace-pub-profiler:25.6.6 \
--disable-tls --collection-agent <host>:8375

tip

Be sure to update the collection agent value to point to your UI/gateway service.

Enabling GPU Profiling

To enable GPU profiling capabilities with Docker (AMD64/x86_64 only), mount an additional volume:

docker run --pid=host --privileged --net=host \
-v /etc/machine-id:/etc/machine-id:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/lib/zymtrace/profiler:/opt/zymtrace-cuda-profiler \
--rm -d --name zymtrace-profiler ghcr.io/zystem-io/zymtrace-pub-profiler:25.6.6 \
--disable-tls --collection-agent <host>:8375

When the profiler starts with this volume mount, it will automatically extract the necessary CUDA profiler libraries to the host path (/var/lib/zymtrace/profiler), making them available to your GPU workloads.

To verify the profiler libraries were extracted correctly:

# Check the contents of the shared volume directory
ls -la /var/lib/zymtrace/profiler

# You should see
# libzymtracecudaprofiler.so

For more details on using the GPU profiler, see the GPU Profiler documentation.

Binary

Install with Binary

x86_64

x86_64 Installation

# Download the binaries
curl -LO https://dl.zystem.io/zymtrace/25.6.6/amd64/zymtrace-profiler.tar.gz

# Extract them
sudo tar -xzf zymtrace-profiler.tar.gz -C /

# Run the agent
sudo /opt/zymtrace/profiler/zymtrace-profiler --collection-agent localhost:80 --disable-tls

ARM64

ARM64 Installation

# Download the binaries
curl -LO https://dl.zystem.io/zymtrace/25.6.6/arm64/zymtrace-profiler.tar.gz

# Extract them
sudo tar -xzf zymtrace-profiler.tar.gz -C /

# Run the agent
sudo /opt/zymtrace/profiler/zymtrace-profiler --collection-agent localhost:80 --disable-tls

systemctl

systemctl Service

This guide explains how to set up and run the zymtrace profiler as a system service using systemctl.

1. Create the zymtrace directory:

   sudo mkdir -p /opt/zymtrace

2. Move the downloaded agent binary to the installation directory:

   sudo mv zymtrace-profiler /opt/zymtrace/

3. Make the binary executable:

   sudo chmod +x /opt/zymtrace/zymtrace-profiler

4. Create a systemd service file:

   sudo vi /etc/systemd/system/zymtrace.service

5. Copy and paste the following configuration into the file:

   [Unit]
   Description=zymtrace profiler service
   After=network.target
   
   [Service]
   Type=simple
   ExecStart=/opt/zymtrace/zymtrace-profiler --collection-agent localhost:8375 --disable-tls
   Restart=always
   RestartSec=10
   WorkingDirectory=/opt/zymtrace
   
   [Install]
   WantedBy=multi-user.target
   

6. Enable the service

    sudo systemctl daemon-reload
   
    sudo systemctl start zymtrace
   
    sudo systemctl enable zymtrace
   
    sudo systemctl status zymtrace

Management

These commands help you monitor and maintain your zymtrace profiler installation. Use them to check the agent's status and logs.

Kubectl

Kubectl Management

# Check agent status
kubectl get pods -n zymtrace -l app=zymtrace,component=profiler

# View agent logs
 kubectl logs -f -n zymtrace -l app=zymtrace,component=profiler

Helm

Helm Management

# Check profiler status (not agent)
helm status profiler -n zymtrace

# Upgrade charts, force a new image pull
helm upgrade profiler zymtrace/profiler --n zymtrace  --set global.imagePullPolicy=Always --reuse-values

# Get pod status
kubectl get pods -n zymtrace -l app=zymtrace,component=profiler

# Upgrade profiler (not agent)
helm upgrade profiler zymtrace/profiler -n zymtrace

# View profiler values (not agent)
helm get values profiler -n zymtrace

# Uninstall profiler (not agent)
helm uninstall profiler -n zymtrace

Docker

Docker Management

# View running agents
docker ps | grep zymtrace/profiler

# View agent logs
docker logs <container-id>

# Stop agent
docker stop <container-id>

# Remove agent container
docker rm <container-id>

Binary

Binary Management

  # reladed daemon
  sudo systemctl daemon-reload

  # start agent
  sudo systemctl start zymtrace
  
  # Enable agent on boot
  sudo systemctl enable zymtrace
  
  # Check agent status
  sudo systemctl status zymtrace