Skip to main content

Profiling SGLang

Run SGLang with the Zymtrace GPU profiler injected, on NVIDIA or AMD GPUs. The examples serve zai-org/GLM-4-9B-0414 (~19 GB of BF16 weights) with CUDA or HIP graphs enabled, which is SGLang's default. Check the prerequisites first.

Install the Zymtrace profiler first

Install the Zymtrace profiler on your cluster with Helm and enable the CUDA profiler (--set profiler.cudaProfiler.enabled=true). It extracts the CUDA profiler library to /var/lib/zymtrace/profiler on every GPU node. See Install with Helm.

The manifest mounts /var/lib/zymtrace/profiler from the node and sets CUDA_INJECTION64_PATH, as described in Kubernetes GPU configuration. These are the fields that enable profiling:

spec:
  template:
    spec:
      hostIPC: true
      containers:
        - name: sglang
          image: lmsysorg/sglang:v0.5.22
          command: ["python3", "-m", "sglang.launch_server"]
          args: ["--model-path", "zai-org/GLM-4-9B-0414", "--host", "0.0.0.0", "--port", "30000", "--enable-metrics"]
          env:
            - name: CUDA_INJECTION64_PATH
              value: /opt/zymtrace/profiler/libzymtracecudaprofiler.so
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: zymtrace-profiler
              mountPath: /opt/zymtrace/profiler
              readOnly: true
      volumes:
        - name: zymtrace-profiler
          hostPath:
            path: /var/lib/zymtrace/profiler
            type: Directory

Apply the full manifest:

kubectl apply -f https://gist.githubusercontent.com/iogbole/16c39f0915cb4998a7a201c8c0864ba8/raw/sglang.yaml

# Reach the OpenAI-compatible API on localhost:30000
kubectl -n sglang port-forward svc/sglang 30000:30000

Send requests​

SGLang exposes an OpenAI-compatible API:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "zai-org/GLM-4-9B-0414", "messages": [{"role": "user", "content": "How does machine learning work?"}], "max_tokens": 256}'

Metrics​

SGLang only exposes its Prometheus metrics when started with --enable-metrics, which every example above sets. Zymtrace then collects them alongside the profiles. See Inference Server Metrics.

For PC sampling (NVIDIA) and verification, see the overview.