Install zymtrace profiler

The zymtrace profiler is a lightweight, OpenTelemetry compliant agent that collects performance profiles from your applications and systems with minimal overhead (<1% CPU and ~256MB RAM ). It can be deployed in various ways to suit your infrastructure needs. This guide covers installation using Kubernetes manifests, Helm charts, Docker containers, or direct binary installation.

tip

Please review the prerequisites before beginning, particularly if you are working in an airgapped environment.

Installation methods

Choose the installation method that best suits your environment. Each method provides the same functionality with different deployment characteristics.

Kubectl

Install with Kubectl

The profiler agent is deployed as a DaemonSet.

1. Create a namespace

   kubectl create namespace zymtrace

2. Deploy

   kubectl apply -n zymtrace -f https://helm.zystem.io/k8s-manifests/profiler/zymtrace-profiler.yaml

tip

Important: If you need GPU profiling capabilities, we recommend using the Helm installation method instead, which provides more configuration options for enabling GPU profiling.

tip

Collection Agent Configuration

By default, the collection agent is set to zymtrace-gateway.zymtrace.svc.cluster.local:80.

Remember that the collection agent should point to your gateway service. If you're installing the profiling agent on a different cluster than the one hosting the backend services, you'll need to modify this setting. In this case, we recommend downloading the configuration file first.

curl -O https://helm.zystem.io/k8s-manifests/profiler/zymtrace-profiler.yaml

Helm

Install with Helm

# Add the zystem repository
helm repo add zymtrace https://helm.zystem.io

# List available charts and versions
helm search repo zymtrace --versions

# Install
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80" \
 --set profiler.args[1]="--disable-tls" \
 --set profiler.args[2]="--project=colossus" \
 --set profiler.args[3]="--tags=prod;us-east" \
 --set "profiler.env.HTTPS_PROXY=http://username:password@proxy:port"

Enabling GPU Profiling

To enable GPU profiling capabilities with Helm, add the cudaProfiler.enabled setting:

# Install with GPU profiling enabled
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 --set profiler.args[0]="--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80" \
 --set profiler.args[1]="--disable-tls" \
 --set profiler.cudaProfiler.enabled=true

This will automatically:

• Deploy the necessary GPU profiling libraries
• Configure the volume mounts for sharing libraries between containers
• Extract the profiling libraries to the shared host path
• Make GPU profiling available to all containers that mount the shared volume

important

GPU profiling is only available on AMD64/x86_64 architecture.

Using custom-values.yaml

Using custom values

You can create a custom values file with your configurations:

profiler:
 args:
   - "--collection-agent=zymtrace-gateway.zymtrace.svc.cluster.local:80"  # Point to your gateway service
   - "--disable-tls"
   - "--project=my-project-123"
   - "--tags=prod;ha100;fp16"

 # Enable GPU profiling
 cudaProfiler:
   enabled: true
   hostMountPath: "/var/lib/zymtrace/profiler"  # Default path

 env:
   HTTPS_PROXY: "http://user:[email protected]:8080"
   ZYM: "TRACE"

   fieldRefs:
     NODE_NAME: true
     KUBERNETES_NODE_NAME: true
     POD_NAMESPACE: metadata.namespace

Then install using:

helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 -f custom-values.yaml

For more details on using the GPU profiler, see the GPU Profiler documentation.

Docker

Install with Docker

docker run --cgroupns=host --pid=host --privileged --net=host \
-v /etc/machine-id:/etc/machine-id:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
--rm -d --name zymtrace-profiler ghcr.io/zystem-io/zymtrace-pub-profiler:25.8.2 \
--disable-tls --collection-agent <host>:8080

tip

Be sure to update the collection agent value to point to your gateway service.

Enabling GPU Profiling

To enable GPU profiling capabilities with Docker (AMD64/x86_64 only), mount an additional volume:

docker run --cgroupns=host --pid=host --privileged --net=host \
-v /etc/machine-id:/etc/machine-id:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/lib/zymtrace/profiler:/opt/zymtrace-cuda-profiler \
--rm -d --name zymtrace-profiler ghcr.io/zystem-io/zymtrace-pub-profiler:25.8.2 \
--disable-tls --collection-agent <host>:8080

When the profiler starts with this volume mount, it will automatically extract the necessary CUDA profiler libraries to the host path (/var/lib/zymtrace/profiler), making them available to your GPU workloads.

To verify the profiler libraries were extracted correctly:

# Check the contents of the shared volume directory
ls -la /var/lib/zymtrace/profiler

# You should see
# libzymtracecudaprofiler.so

For more details on using the GPU profiler, see the GPU Profiler documentation.

Binary

Install with Binary

x86_64

x86_64 Installation

# Download the binaries
curl -LO https://dl.zystem.io/zymtrace/25.8.2/amd64/zymtrace-profiler.tar.gz

# Extract them
sudo tar -xzf zymtrace-profiler.tar.gz -C /

# Run the agent
sudo /opt/zymtrace/profiler/zymtrace-profiler --collection-agent localhost:8080 --disable-tls

ARM64

ARM64 Installation

# Download the binaries
curl -LO https://dl.zystem.io/zymtrace/25.8.2/arm64/zymtrace-profiler.tar.gz

# Extract them
sudo tar -xzf zymtrace-profiler.tar.gz -C /

# Run the agent
sudo /opt/zymtrace/profiler/zymtrace-profiler --collection-agent localhost:8080 --disable-tls

tip

Be sure to update the --collection-agent value to point to your zymtrace gateway service.

Management

These commands help you monitor and maintain your zymtrace profiler installation. Use them to check the agent's status and logs.

Kubectl

Kubectl Management

# Check agent status
kubectl get pods -n zymtrace -l app=zymtrace,component=profiler

# View agent logs
kubectl logs -f -n zymtrace -l app=zymtrace,component=profiler

Helm

Helm Management

# Check profiler status (not agent)
helm status profiler -n zymtrace

# Upgrade charts, force a new image pull
helm upgrade profiler zymtrace/profiler --n zymtrace  --set global.imagePullPolicy=Always --reuse-values

# Get pod status
kubectl get pods -n zymtrace -l app=zymtrace,component=profiler

# Upgrade profiler (not agent)
helm upgrade profiler zymtrace/profiler -n zymtrace

# View profiler values (not agent)
helm get values profiler -n zymtrace

# Uninstall profiler (not agent)
helm uninstall profiler -n zymtrace

Docker

Docker Management

# View running agents
docker ps | grep zymtrace/profiler

# View agent logs
docker logs <container-id>

# Stop agent
docker stop <container-id>

# Remove agent container
docker rm <container-id>

Systemd

Setting up systemd service

To manage the profiler as a systemd service:

1. Create the zymtrace directory:

   sudo mkdir -p /opt/zymtrace

2. Move the downloaded agent binary to the installation directory:

   sudo mv zymtrace-profiler /opt/zymtrace/

3. Make the binary executable:

   sudo chmod +x /opt/zymtrace/zymtrace-profiler

4. Create a systemd service file:

   sudo vi /etc/systemd/system/zymtrace.service

5. Copy and paste the following configuration into the file:

   [Unit]
   Description=zymtrace profiler service
   After=network.target
   
   [Service]
   Type=simple
   ExecStart=/opt/zymtrace/zymtrace-profiler --collection-agent localhost:8080 --disable-tls
   Restart=always
   RestartSec=10
   WorkingDirectory=/opt/zymtrace
   
   [Install]
   WantedBy=multi-user.target

tip

Be sure to update the --collection-agent value in the systemd service file to point to your zymtrace gateway service.

6. Enable the service

   # Reload daemon
   sudo systemctl daemon-reload
   
   # Start agent
   sudo systemctl start zymtrace
   
   # Enable agent start on boot
   sudo systemctl enable zymtrace
   
   # Check agent status
   sudo systemctl status zymtrace

Management commands

Once the systemd service is set up, use these commands to manage the profiler:

# Check systemd service status
sudo systemctl status zymtrace

# View service logs
sudo journalctl -u zymtrace -f

# Restart service
sudo systemctl restart zymtrace

# Stop service
sudo systemctl stop zymtrace