Profiling NVIDIA Dynamo-Triton

Run NVIDIA Dynamo-Triton with the Zymtrace CUDA profiler injected. Dynamo-Triton runs on NVIDIA GPUs only. The examples serve zai-org/GLM-4-9B-0414 through Triton's vLLM backend, with CUDA graphs enabled. Triton is distributed as a container image, so there are no binary install instructions. Check the prerequisites first.

Kubernetes (Helm)

Install the Zymtrace profiler first

Install the Zymtrace profiler on your cluster with Helm and enable the CUDA profiler (--set profiler.cudaProfiler.enabled=true). It extracts the CUDA profiler library to /var/lib/zymtrace/profiler on every GPU node. See Install with Helm.

The manifest keeps the model repository in a ConfigMap, mounts /var/lib/zymtrace/profiler from the node, and sets CUDA_INJECTION64_PATH, as described in Kubernetes GPU configuration. These are the fields that define the model and enable profiling:

apiVersion: v1
kind: ConfigMap
metadata:
  name: triton-model-repository
data:
  config.pbtxt: |
    backend: "vllm"
    instance_group [{ count: 1, kind: KIND_MODEL }]
  model.json: |
    {"model": "zai-org/GLM-4-9B-0414", "gpu_memory_utilization": 0.85, "max_model_len": 8192}
---
# Deployment
spec:
  template:
    spec:
      hostIPC: true
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:25.11-vllm-python-py3
          command: ["tritonserver"]
          args: ["--model-repository=/models"]
          env:
            - name: CUDA_INJECTION64_PATH
              value: /opt/zymtrace/profiler/libzymtracecudaprofiler.so
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: zymtrace-profiler
              mountPath: /opt/zymtrace/profiler
              readOnly: true
            - name: model-repository
              mountPath: /models/glm4
              readOnly: true
      volumes:
        - name: zymtrace-profiler
          hostPath:
            path: /var/lib/zymtrace/profiler
            type: Directory
        - name: model-repository
          configMap:
            name: triton-model-repository
            items:
              - key: config.pbtxt
                path: config.pbtxt
              - key: model.json
                path: 1/model.json

Apply the full manifest:

kubectl apply -f https://gist.githubusercontent.com/iogbole/2fda4ed7e53f70beb2df949d3eb320ef/raw/triton.yaml

# Reach the generate endpoint on localhost:8000 and metrics on localhost:8002
kubectl -n triton port-forward svc/triton 8000:8000 8002:8002

Docker

Start the Zymtrace profiler first

Run the Zymtrace profiler container on the host. It extracts the CUDA profiler library to /var/lib/zymtrace/profiler. See Install with Docker.

Create a model repository with one model, glm4, served by the vLLM backend:

mkdir -p triton-models/glm4/1

cat > triton-models/glm4/config.pbtxt <<'EOF'
backend: "vllm"
instance_group [{ count: 1, kind: KIND_MODEL }]
EOF

cat > triton-models/glm4/1/model.json <<'EOF'
{"model": "zai-org/GLM-4-9B-0414", "gpu_memory_utilization": 0.85, "max_model_len": 8192}
EOF

Then start Triton with the repository mounted, /var/lib/zymtrace/profiler mounted from the host, and CUDA_INJECTION64_PATH set, as described in Docker GPU configuration. Port 8000 serves inference and port 8002 serves metrics:

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$PWD/triton-models:/models:ro" \
  -v /var/lib/zymtrace/profiler:/opt/zymtrace/profiler:ro \
  -e CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
  -p 8000:8000 \
  -p 8002:8002 \
  --ipc=host \
  --rm -d \
  --name triton-server \
  nvcr.io/nvidia/tritonserver:25.11-vllm-python-py3 \
  tritonserver --model-repository=/models

Send requests

Triton exposes a generate endpoint per model:

curl http://localhost:8000/v2/models/glm4/generate \
  -H "Content-Type: application/json" \
  -d '{"text_input": "How does machine learning work?", "parameters": {"stream": false, "max_tokens": 256}}'

Metrics

Triton exposes its Prometheus metrics on port 8002 by default, and Zymtrace collects them alongside the profiles. See Inference Server Metrics.

For PC sampling and verification, see the overview.