Install zymtrace profiler

The zymtrace profiler is a lightweight, OpenTelemetry compliant agent that collects performance profiles from your applications and systems with minimal overhead (<1% CPU and ~256MB RAM ). It can be deployed in various ways to suit your infrastructure needs. This guide covers installation using Kubernetes manifests, Helm charts, Docker containers, or direct binary installation.

As a reminder, the profiling agent runs only on Linux machines and it requires minimum kernel version 4.19 for x86_64, and 5.5 for ARM64

tip

Please review the prerequisites before beginning, particularly if you are working in an airgapped environment.

Installation methods

Choose the installation method that best suits your environment. Each method provides the same functionality with different deployment characteristics.

The Add Data page provides a guided approach to generating the deployment commands based on your environment.

Kubectl

The profiler agent is deployed as a DaemonSet.

1. Create a namespace

   kubectl create namespace zymtrace

2. Create a secret using the credentials we shared

   kubectl create secret docker-registry zymtrace-registry-cred-profiler \
   --namespace=zymtrace \
   --docker-server=ghcr.io/zystem-io \
   --docker-username=<your-username> \
   --docker-password=<your-password>
   

3. Deploy

   kubectl apply -n zymtrace -f https://helm.zystem.io/k8s-manifests/profiler/zymtrace-profiler.yaml

tip

Collection Agent Configuration

By default, the collection agent is set to zymtrace-ingest.zymtrace.svc.cluster.local:8375.

If you're installing the profiling agent on a different cluster than the one hosting the backend services, you'll need to modify this setting. In this case, we recommend downloading the configuration file first.

curl -O https://helm.zystem.io/k8s-manifests/profiler/zymtrace-profiler.yaml

Helm

# Add the zystem repository
helm repo add zymtrace https://helm.zystem.io

# List available charts and versions
helm search repo zymtrace --versions

# Install
helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace-local \
 --set profiler.args.disable-tls=true \
 --set profiler.args.project=colossus \
 --set "profiler.args.tags={prod;us-east}" \
 --set global.registry.username=<username> \
 --set global.registry.password=<password> \
 --set "profiler.env.HTTPS_PROXY=http://username:password@proxy:port" \
 --set profiler.args.collection-agent=zymtrace-ingest.zymtrace.svc.cluster.local:8375

Using custom-values.yaml

Using custom values

You can create a custom values file with your configurations:

profiler:
 args:
   disable-tls: true
   collection-agent: "custom-ingest.internal:8375"
   tags: ["prod;ha100;fp16"]
   project: "my-project-123" 
   
 env:
   HTTPS_PROXY: "http://user:[email protected]:8080"
   ZYM: "TRACE"

   fieldRefs:
     NODE_NAME: true
     KUBERNETES_NODE_NAME: true
     POD_NAMESPACE: metadata.namespace

Then install using:

helm install profiler zymtrace/profiler \
 --create-namespace \
 --namespace zymtrace \
 -f custom-values.yaml

Docker

# Login with the username and password you've been given
docker login ghcr.io/zystem-io -u <username>

# Run
docker run --pid=host --privileged --net=host \
  -v /etc/machine-id:/etc/machine-id:ro \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /sys/kernel/debug:/sys/kernel/debug:ro \
  --rm -ti ghcr.io/zystem-io/zymtrace-profiler:latest \
  --disable-tls --collection-agent <localhost>:8375

tip

Be sure to update the collection agent value, as necessary.

Binary

x86_64

# Download the binary
curl -LO https://dl.zystem.io/zymtrace/latest/amd64/zymtrace-profiler

# Make it executable
chmod +x zymtrace-profiler

# Run the agent
sudo ./zymtrace-profiler --collection-agent localhost:8375 --disable-tls

ARM64

# Download the binary
curl -LO https://dl.zystem.io/zymtrace/latest/arm64/zymtrace-profiler

# Make it executable
chmod +x zymtrace-profiler

# Run the agent
sudo ./zymtrace-profiler --collection-agent localhost:8375 --disable-tls

systemctl

This guide explains how to set up and run the zymtrace profiler as a system service using systemctl.

1. Create the zymtrace directory:

   sudo mkdir -p /opt/zymtrace

2. Move the downloaded agent binary to the installation directory:

   sudo mv zymtrace-profiler /opt/zymtrace/

3. Make the binary executable:

   sudo chmod +x /opt/zymtrace/zymtrace-profiler

4. Create a systemd service file:

   sudo vi /etc/systemd/system/zymtrace.service

5. Copy and paste the following configuration into the file:

   [Unit]
   Description=zymtrace profiler service
   After=network.target
   
   [Service]
   Type=simple
   ExecStart=/opt/zymtrace/zymtrace-profiler --collection-agent localhost:8375 --disable-tls
   Restart=always
   RestartSec=10
   WorkingDirectory=/opt/zymtrace
   
   [Install]
   WantedBy=multi-user.target
   

6. Enable the service

    sudo systemctl daemon-reload
   
    sudo systemctl start zymtrace
   
    sudo systemctl enable zymtrace
   
    sudo systemctl status zymtrace

Management

These commands help you monitor and maintain your zymtrace profiler installation. Use them to check the agent's status and logs.

Kubectl

# Check agent status
kubectl get pods -n zymtrace -l app=zymtrace,component=profiler

# View agent logs
 kubectl logs -f -n zymtrace -l app=zymtrace,component=profiler

Helm

# Check profiler status (not agent)
helm status profiler -n zymtrace

# Upgrade charts, force a new image pull
helm upgrade profiler zymtrace/profiler --n zymtrace  --set global.imagePullPolicy=Always --reuse-values

# Get pod status
kubectl get pods -n zymtrace -l app=zymtrace,component=profiler

# Upgrade profiler (not agent)
helm upgrade profiler zymtrace/profiler -n zymtrace

# View profiler values (not agent)
helm get values profiler -n zymtrace

# Uninstall profiler (not agent)
helm uninstall profiler -n zymtrace

Docker

# View running agents
docker ps | grep zymtrace/profiler

# View agent logs
docker logs <container-id>

# Stop agent
docker stop <container-id>

# Remove agent container
docker rm <container-id>

Binary

  # reladed daemon
  sudo systemctl daemon-reload

  # start agent
  sudo systemctl start zymtrace
  
  # Enable agent on boot
  sudo systemctl enable zymtrace
  
  # Check agent status
  sudo systemctl status zymtrace