Profiling Inference Engines

Zymtrace profiles inference engines such as vLLM, SGLang, NVIDIA Dynamo-Triton, and TensorRT-LLM without code changes. The GPU profiler library is loaded into the engine process when it starts, and the kernels it samples are attributed to the Python and native code that launched them:

• NVIDIA GPUs: set CUDA_INJECTION64_PATH to the Zymtrace CUDA profiler.
• AMD GPUs: set ROCP_TOOL_LIBRARIES and ROCPROFILER_REGISTER_LIBRARY to the Zymtrace ROCm profiler.

For vLLM, SGLang, and Dynamo-Triton, the profiler agent also collects the engine's own metrics as inference server metrics. For training workloads or custom inference code, see Profiling GPU Workloads.

Engines

Each engine has its own page with step-by-step instructions:

Engine	NVIDIA	AMD	Example model
vLLM	✓	✓	zai-org/GLM-4-9B-0414
SGLang	✓	✓	zai-org/GLM-4-9B-0414
NVIDIA Dynamo-Triton	✓	—	zai-org/GLM-4-9B-0414 (vLLM backend)
TensorRT-LLM	✓	—	Qwen/Qwen3-8B

CUDA and HIP graphs

Zymtrace supports CUDA graphs on NVIDIA and HIP graphs on AMD, which these engines use by default.

On NVIDIA, graph launches are sampled, and a sampled launch profiles every kernel in the graph:

Environment variable	Description
ZYMTRACE_CUDAPROFILER__GRAPH_SAMPLING__ENABLED	Sample graph launches rather than profiling every launch. Default is true.
ZYMTRACE_CUDAPROFILER__GRAPH_SAMPLING__TARGET	Graph launches to profile, per OS thread, per second. Default is 20.

On AMD, a HIP graph is sampled as a unit through the ZYMTRACE_ROCMPROFILER__STACK_TRACE_SAMPLING__* settings. See GPU Profiler Variables.

Prerequisites

• The Zymtrace profiler running with GPU profiling enabled. See Install Zymtrace Profiler for Helm, Docker, and binary installs.
• A GPU with at least 40 GB of memory (see supported GPUs):
  • NVIDIA: driver 580 or newer. The pinned images are built against CUDA 13.0; newer engine releases may require a newer driver.
  • AMD: ROCm 7.x or above on an x86_64 host with the amdgpu kernel driver.

tip

On NVIDIA, if you can't (or don't want to) set environment variables on the engine, use automatic injection via the UI instead. AMD workloads must be started with the environment variables.

Sending requests

Kernels are only profiled while the engine is doing work. Each engine's page has an example request.

PC sampling

PC sampling is available on NVIDIA GPUs. To capture SASS instructions and stall reasons, enable it on the engine. NVIDIA requires privileged access for it (see PC Sampling):

• Kubernetes: add ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING=true to the container's env and set securityContext.privileged: true.
• Docker: add -e ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING="true" and --privileged to docker run.
• Binary: start the engine with sudo, keeping the virtual environment and model cache of your user. For example, with vLLM:

  sudo env "PATH=$PATH" HF_HOME="$HOME/.cache/huggingface" \
      ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING="true" \
      CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
      vllm serve zai-org/GLM-4-9B-0414 --port 8000

Verification

To log the profiler's startup, add these variables to the engine's environment:

• NVIDIA: RUST_LOG="zymtracecudaprofiler=info" and ZYMTRACE_CUDAPROFILER__QUIET="false". The engine output then includes:

  INFO zymtracecudaprofiler: loaded zymtrace CUDA profiler version ...
• AMD: RUST_LOG="zymtracerocmprofiler=info" and ZYMTRACE_ROCMPROFILER__QUIET="false". The engine output then includes zymtracerocmprofiler log lines.

In Zymtrace, filter on the engine's container, pod, or process to see its GPU flamegraph. For per-method checks and troubleshooting, see Verification in Profiling GPU Workloads.