Profiling TensorRT-LLM

Run TensorRT-LLM with the Zymtrace CUDA profiler injected. TensorRT-LLM runs on NVIDIA GPUs only. The examples serve Qwen/Qwen3-8B (~16 GB of BF16 weights) with trtllm-serve, with CUDA graphs enabled, which is TensorRT-LLM's default. Check the prerequisites first.

Kubernetes (Helm)

Install the Zymtrace profiler first

Install the Zymtrace profiler on your cluster with Helm and enable the CUDA profiler (--set profiler.cudaProfiler.enabled=true). It extracts the CUDA profiler library to /var/lib/zymtrace/profiler on every GPU node. See Install with Helm.

The manifest mounts /var/lib/zymtrace/profiler from the node and sets CUDA_INJECTION64_PATH, as described in Kubernetes GPU configuration. These are the fields that enable profiling:

spec:
  template:
    spec:
      hostIPC: true
      containers:
        - name: trtllm
          image: nvcr.io/nvidia/tensorrt-llm/release:1.1.0
          args: ["trtllm-serve", "Qwen/Qwen3-8B", "--host", "0.0.0.0", "--port", "8000"]
          env:
            - name: CUDA_INJECTION64_PATH
              value: /opt/zymtrace/profiler/libzymtracecudaprofiler.so
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: zymtrace-profiler
              mountPath: /opt/zymtrace/profiler
              readOnly: true
      volumes:
        - name: zymtrace-profiler
          hostPath:
            path: /var/lib/zymtrace/profiler
            type: Directory

Apply the full manifest:

kubectl apply -f https://gist.githubusercontent.com/iogbole/2cba56573253f0f9e1caf86d1953929e/raw/tensorrt-llm.yaml

# Reach the OpenAI-compatible API on localhost:8000
kubectl -n trtllm port-forward svc/trtllm 8000:8000

Docker

Start the Zymtrace profiler first

Run the Zymtrace profiler container on the host. It extracts the CUDA profiler library to /var/lib/zymtrace/profiler. See Install with Docker.

Mount /var/lib/zymtrace/profiler from the host and set CUDA_INJECTION64_PATH, as described in Docker GPU configuration. The --ulimit settings are NVIDIA's recommendation for the TensorRT-LLM container:

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /var/lib/zymtrace/profiler:/opt/zymtrace/profiler:ro \
  -e CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
  -p 8000:8000 \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --rm -d \
  --name trtllm-server \
  nvcr.io/nvidia/tensorrt-llm/release:1.1.0 \
  trtllm-serve Qwen/Qwen3-8B --host 0.0.0.0 --port 8000

Binary

Install the Zymtrace profiler first

Install and start the Zymtrace profiler binary on the host. It ships the CUDA profiler library at /opt/zymtrace/profiler/libzymtracecudaprofiler.so. See Install with Binary.

TensorRT-LLM 1.1.0 needs the CUDA 13.0 toolkit (with CUDA_HOME set), OpenMPI, and PyTorch built for CUDA 13.0. See NVIDIA's Linux installation guide for details.

sudo apt-get -y install libopenmpi-dev

python3 -m venv ~/trtllm-env
source ~/trtllm-env/bin/activate
pip install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu130

# Stop pip from replacing the CUDA 13.0 build of PyTorch
python3 -c "import torch; print(f'torch=={torch.__version__}')" > /tmp/torch-constraint.txt
pip install tensorrt_llm==1.1.0 -c /tmp/torch-constraint.txt

env CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
    trtllm-serve Qwen/Qwen3-8B --host 0.0.0.0 --port 8000

Send requests

trtllm-serve exposes an OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "How does machine learning work?"}], "max_tokens": 256}'

Metrics

For TensorRT-LLM, Zymtrace collects GPU profiles and GPU metrics.

For PC sampling and verification, see the overview.