Profiling vLLM

Run vLLM with the Zymtrace GPU profiler injected, on NVIDIA or AMD GPUs. The examples serve zai-org/GLM-4-9B-0414 (~19 GB of BF16 weights) with CUDA or HIP graphs enabled, which is vLLM's default. Check the prerequisites first.

NVIDIA (CUDA)

Kubernetes (Helm)

Install the Zymtrace profiler first

Install the Zymtrace profiler on your cluster with Helm and enable the CUDA profiler (--set profiler.cudaProfiler.enabled=true). It extracts the CUDA profiler library to /var/lib/zymtrace/profiler on every GPU node. See Install with Helm.

The manifest mounts /var/lib/zymtrace/profiler from the node and sets CUDA_INJECTION64_PATH, as described in Kubernetes GPU configuration. These are the fields that enable profiling:

spec:
  template:
    spec:
      hostIPC: true
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.32.0
          args: ["--model", "zai-org/GLM-4-9B-0414", "--host", "0.0.0.0", "--port", "8000"]
          env:
            - name: CUDA_INJECTION64_PATH
              value: /opt/zymtrace/profiler/libzymtracecudaprofiler.so
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: zymtrace-profiler
              mountPath: /opt/zymtrace/profiler
              readOnly: true
      volumes:
        - name: zymtrace-profiler
          hostPath:
            path: /var/lib/zymtrace/profiler
            type: Directory

Apply the full manifest:

kubectl apply -f https://gist.githubusercontent.com/iogbole/0060415b87d6d8ecff8cdc4b5a3143e4/raw/vllm.yaml

# Reach the OpenAI-compatible API on localhost:8000
kubectl -n vllm port-forward svc/vllm 8000:8000

Docker

Start the Zymtrace profiler first

Run the Zymtrace profiler container on the host. It extracts the CUDA profiler library to /var/lib/zymtrace/profiler. See Install with Docker.

Mount /var/lib/zymtrace/profiler from the host and set CUDA_INJECTION64_PATH, as described in Docker GPU configuration:

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /var/lib/zymtrace/profiler:/opt/zymtrace/profiler:ro \
  -e CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
  -p 8000:8000 \
  --ipc=host \
  --rm -d \
  --name vllm-server \
  vllm/vllm-openai:v0.32.0 \
  --model zai-org/GLM-4-9B-0414

Binary

Install the Zymtrace profiler first

Install and start the Zymtrace profiler binary on the host. It ships the CUDA profiler library at /opt/zymtrace/profiler/libzymtracecudaprofiler.so. See Install with Binary.

Install vLLM into a virtual environment (Python 3.11 or newer) and start it with CUDA_INJECTION64_PATH set:

python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install vllm==0.32.0

env CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so" \
    vllm serve zai-org/GLM-4-9B-0414 --port 8000

AMD (ROCm)

Kubernetes (Helm)

Install the Zymtrace profiler first

Install the Zymtrace profiler on your cluster with Helm and enable GPU profiling (--set profiler.cudaProfiler.enabled=true). It extracts the ROCm profiler libraries to /var/lib/zymtrace/profiler on every GPU node. See Install with Helm.

Run vLLM's ROCm image with /var/lib/zymtrace/profiler mounted from the node and the ROCm profiling variables set. The AMD GPU device plugin exposes GPUs as amd.com/gpu. Add these fields to your vLLM Deployment:

spec:
  template:
    spec:
      hostIPC: true
      containers:
        - name: vllm
          image: vllm/vllm-openai-rocm:v0.32.0
          args: ["--model", "zai-org/GLM-4-9B-0414", "--host", "0.0.0.0", "--port", "8000"]
          env:
            - name: ROCP_TOOL_LIBRARIES
              value: /opt/zymtrace/profiler/libzymtracerocmprofiler.so
            - name: ROCPROFILER_REGISTER_LIBRARY
              value: /opt/zymtrace/profiler/librocprofiler-sdk.so.1
          resources:
            limits:
              amd.com/gpu: 1
          volumeMounts:
            - name: zymtrace-profiler
              mountPath: /opt/zymtrace/profiler
              readOnly: true
      volumes:
        - name: zymtrace-profiler
          hostPath:
            path: /var/lib/zymtrace/profiler
            type: Directory

Docker

Start the Zymtrace profiler first

Run the Zymtrace profiler container on the host. It extracts the ROCm profiler libraries to /var/lib/zymtrace/profiler. See Install with Docker.

Pass the AMD GPU devices through, mount /var/lib/zymtrace/profiler from the host, and set the ROCm profiling variables:

docker run \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add video \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /var/lib/zymtrace/profiler:/opt/zymtrace/profiler:ro \
  -e ROCP_TOOL_LIBRARIES="/opt/zymtrace/profiler/libzymtracerocmprofiler.so" \
  -e ROCPROFILER_REGISTER_LIBRARY="/opt/zymtrace/profiler/librocprofiler-sdk.so.1" \
  -p 8000:8000 \
  --ipc=host \
  --rm -d \
  --name vllm-server \
  vllm/vllm-openai-rocm:v0.32.0 \
  --model zai-org/GLM-4-9B-0414

Binary

Install the Zymtrace profiler first

Install and start the Zymtrace profiler binary on the host. It ships the ROCm profiler libraries in /opt/zymtrace/profiler. See Install with Binary.

Start vLLM from a ROCm-enabled environment with the ROCm profiling variables set:

env ROCP_TOOL_LIBRARIES="/opt/zymtrace/profiler/libzymtracerocmprofiler.so" \
    ROCPROFILER_REGISTER_LIBRARY="/opt/zymtrace/profiler/librocprofiler-sdk.so.1" \
    vllm serve zai-org/GLM-4-9B-0414 --port 8000

See vLLM's ROCm installation guide to install vLLM for ROCm.

Send requests

vLLM exposes an OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "zai-org/GLM-4-9B-0414", "messages": [{"role": "user", "content": "How does machine learning work?"}], "max_tokens": 256}'

Metrics

vLLM exposes its Prometheus metrics by default, and Zymtrace collects them alongside the profiles. See Inference Server Metrics.

For PC sampling (NVIDIA) and verification, see the overview.