Skip to main content

Profiling GPU Workloads

The zymtrace GPU profiler extends the zymtrace eBPF profiler to provide deep visibility into CUDA-enabled applications. It traces GPU performance issues—such as kernel stalls and memory bottlenecks—directly to the PyTorch code, CUDA kernels, native code, or scheduler threads that launched them.

It requires the zymtrace profiler to be installed and running first.

important

The GPU profiler is supported on AMD64/x86_64 and ARM64 architectures and supports CUDA 12.x and higher.

Configuring GPU Profiler Injection​

The GPU profiler can be injected into matching processes automatically through the zymtrace UI, or manually via environment variables.

Automatic Injection via the UI​

The recommended approach is to configure injection directly from the zymtrace UI. This eliminates the need to modify container manifests or set environment variables on each workload.

In the agent configuration settings, two options control GPU profiler injection:

  • Inject CUDA Profiler — enables automatic injection of the CUDA profiler library into matching processes.
  • GPU Profiler Config — configures the profiler settings (equivalent to the ZYMTRACE_CUDAPROFILER__* environment variables).

Both settings accept a process query as a condition — the same query syntax used in the Zymtrace query bar. Only processes matching the query will be injected into. This means you can target specific applications, containers, namespaces, or any combination of process or host attributes without touching deployment manifests.

tip

As with all agent configuration settings, injection rules and profiler configuration can be updated on the fly through the UI. No restart of the zymtrace profiler or target processes is required for changes to take effect.

Configuring injection settings for the GPU profiler in the UI
warning

NVTX is not supported in this mode, please use the manual injection mode if you need NVTX support.

warning

ARM64 is not supported yet in this mode, please use the manual injection mode if you need ARM support.

Manual Injection via Environment Variables​

The traditional approach uses CUDA_INJECTION64_PATH (and optionally NVTX_INJECTION64_PATH) to preload the profiler library into each process. This method still works and is useful when UI-based injection is not available.

Choose the configuration method that matches your deployment:

Kubernetes GPU Configuration​

The zymtrace approach to GPU profiling is similar to OpenTelemetry's automatic instrumentation,but with a key difference: instead of using Kubernetes Custom Resource Definitions (CRDs) to mutate pods, zymtrace delivers GPU profiler libraries via a DaemonSet, extracting them to a shared host path. Workloads access these libraries by mounting the shared volume.

important

ensure you have the zymtrace profiler installed and running before enabling the GPU profiler.

After installing the zymtrace profiler with GPU support via Helm, you can configure your application containers to use the GPU profiler:

  1. Mount the GPU profiler library into your container:
volumes:
  - name: zymtrace-gpu-profiler
    hostPath:
      path: /var/lib/zymtrace/profiler
      type: Directory
containers:
  - name: your-container
    volumeMounts:
      - name: zymtrace-gpu-profiler
        mountPath: /opt/zymtrace/profiler
        readOnly: true
  1. Set the required environment variables:
env:
  - name: CUDA_INJECTION64_PATH
    value: "/opt/zymtrace/profiler/libzymtracecudaprofiler.so"
  - name: NVTX_INJECTION64_PATH
    value: "/opt/zymtrace/profiler/libzymtracecudaprofiler.so"

Example Kubernetes Job​

Below is an example of a Kubernetes Job that uses the GPU profiler with a PyTorch application:

apiVersion: batch/v1
kind: Job
metadata:
  name: pytorch-cifar-training
  labels:
    app: pytorch-cifar
spec:
  parallelism: 1
  backoffLimit: 5
  template:
    metadata:
      labels:
        app: pytorch-cifar
    spec:
      restartPolicy: OnFailure
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      volumes:
        - name: zymtrace-gpu-profiler
          hostPath:
            path: /var/lib/zymtrace/profiler
            type: Directory
      containers:
        - name: pytorch-cifar
          image: pytorch/pytorch:latest
          volumeMounts:
            - name: zymtrace-gpu-profiler
              mountPath: /opt/zymtrace/profiler
              readOnly: true
          env:
            - name: RUST_LOG
              value: "zymtracecudaprofiler=info"
            - name: ZYMTRACE_CUDAPROFILER__PRINT_STATS
              value: "true"
            - name: ZYMTRACE_CUDAPROFILER__QUIET
              value: "false"
            - name: CUDA_INJECTION64_PATH
              value: "/opt/zymtrace/profiler/libzymtracecudaprofiler.so"
            - name: NVTX_INJECTION64_PATH
              value: "/opt/zymtrace/profiler/libzymtracecudaprofiler.so"
          command:
            - "python"
            - "-c"
            - |
              # PyTorch CIFAR training code
              import torch
              # ... rest of training code ...
          resources:
            limits:
              nvidia.com/gpu: 1
            requests:
              memory: "4Gi"
              cpu: "2"

Verification​

You can verify that the GPU profiler is working by checking for specific log entries:

Kubernetes Verification​

# Check the application logs for GPU profiler status
kubectl logs -n your-namespace <pod-name>

You should also see periodic statistics like this in the logs:

2025-04-28T15:27:46.401606Z  INFO zymtracecudaprofiler: buffers_processed: 0.0/s (total: 0) | bytes_sent: 0.0/s (total: 0) | cubins_loaded: 0.0/s (total: 19) | cubins_size: 0.0/s (total: 44567632) | io_errors: 0.0/s (total: 0) | lru_misses: 15856.5/s (total: 261865856) |

Troubleshooting​

If you encounter issues with the GPU profiler, check the following:

  1. Ensure the zymtrace profiler is running and has successfully extracted libraries to the shared volume
  2. Verify the volume mount paths are correct
  3. Check that the CUDA_INJECTION64_PATH points to the correct library path
  4. Make sure your application is running on NVIDIA GPUs
  5. Check the contents of /var/lib/zymtrace/profiler to verify the libraries are present
  6. Review the application logs for any errors or warnings from the GPU profiler