Metrics Reference

Zymtrace collects metrics across five layers: the host, NVIDIA GPUs (including MIG partitions), AMD GPUs, AWS Neuron accelerators, and inference servers (vLLM, NVIDIA Dynamo-Triton, and SGLang). This page lists every metric by category: its key, type, the aggregation applied, the dimensions it can be grouped by, and the chart it drives.

To enable collection, see NVIDIA GPU Metrics & Profiles, AMD GPU Metrics & Profiles, and Inference Server Metrics. The scrape frequency is controlled by -collect-metrics-interval; see Profiler ENV & CLI Args.

Conventions

• Metric key — the metric identifier queried by the dashboard.
• Temporal — aggregation over time: avg, max, rate, increase.
• Spatial — aggregation across series/hosts: avg, sum, max.
• Group-by — the dimensions a chart breaks down by. by process / script / container means the metric is charted per main executable, per script (Python), and per container.
• Latency histograms report the p50, p95, and p99 percentiles.

Dashboards

Dashboard	Route
Host	/dashboards/host
NVIDIA GPU (incl. MIG)	/dashboards/gpu
AMD GPU (ROCm)	/dashboards/rocm
Neuron	/dashboards/neuron
vLLM	/dashboards/vllm
Triton	/dashboards/triton
SGLang	/dashboards/sglang

---

Host

The memory-source toggle selects working-set (WSS) vs resident-set (RSS) memory.

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
process.utime	gauge	avg	sum	—	CPU Utilization	User-mode CPU.
process.stime	gauge	avg	sum	—	CPU Utilization	System/kernel-mode CPU.
process.utime	gauge	avg	sum	process / script / container	CPU Consumer (User)	User-mode CPU by consumer.
process.stime	gauge	avg	sum	process / script / container	CPU Consumer (System)	System-mode CPU by consumer.
process.wss	gauge	avg	sum	pod / container / deployment / namespace	Memory Consumer (cgroup)	Working-set memory by cgroup entity.
process.rss	gauge	avg	sum	process / script / container	Memory Consumer (non-cgroup)	Resident-set memory by consumer.
process.wss	gauge	max	sum	—	Memory Utilization (cgroup)	Total WSS across containers.
process.rss	gauge	max	sum	—	Memory Utilization (non-cgroup)	Total RSS across processes.
io.read_throughput	gauge	avg	sum	process / script / container	Disk Bytes Read	Bytes/s read per consumer.
io.write_throughput	gauge	avg	sum	process / script / container	Disk Bytes Written	Bytes/s written per consumer.
io.network.read_throughput	gauge	avg	sum	process / script / container	Network Bytes Received	Bytes/s received per consumer.
io.network.write_throughput	gauge	avg	sum	process / script / container	Network Bytes Sent	Bytes/s sent per consumer.
io.read_latency	histogram	increase	max	—	Disk Latency	Read op latency (p50/p95/p99).
io.write_latency	histogram	increase	max	—	Disk Latency	Write op latency (p50/p95/p99).
io.discard_latency	histogram	increase	max	—	Disk Latency	Discard/TRIM op latency (p50/p95/p99).
io.flush_latency	histogram	increase	max	—	Disk Latency	Flush op latency (p50/p95/p99).

---

NVIDIA GPU

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
hw.gpu.utilization	gauge	avg	avg	—	GPU Utilization	% time GPU actively processing.
hw.gpu.process.utilization	gauge	avg	avg	process / script / container	GPU Consumer	GPU utilization by process.
hw.gpu.memory.utilization	gauge	max	sum	—	GPU Memory Utilization	% VRAM in use.
hw.gpu.process.memory.utilization	gauge	max	sum	process / script / container	GPU Memory Consumer	GPU memory by process.
hw.gpu.power	gauge	avg	sum	—	GPU Power Usage	Current power draw (W).
hw.gpu.temp	gauge	avg	max	—	GPU Temperature	Current GPU temp (°C).
hw.gpu.clock.sm	gauge	avg	avg	—	Clock Frequencies	SM clock (MHz).
hw.gpu.clock.mem	gauge	avg	avg	—	Clock Frequencies	Memory clock (MHz).
hw.gpu.clocks_throttle	gauge	avg	max	gpu.throttle_reason	Clocks Throttle Reasons	Fraction of time each throttle reason active.
hw.gpu.sm.efficiency	gauge	avg	avg	—	SM Efficiency	SM efficiency (%).
hw.gpu.sm.occupancy	gauge	avg	avg	—	SM Occupancy	SM occupancy (%).
hw.gpu.tensor_util	gauge	avg	avg	—	Tensor Core Utilization	Tensor Core utilization (%).
hw.gpu.pcie.rx	gauge	avg	sum	—	PCIe Transmission Throughput	PCIe received.
hw.gpu.pcie.tx	gauge	avg	sum	—	PCIe Transmission Throughput	PCIe transmitted.
hw.gpu.nvlink.rx	gauge	avg	sum	—	NVLink Transmission Throughput	NVLink received.
hw.gpu.nvlink.tx	gauge	avg	sum	—	NVLink Transmission Throughput	NVLink transmitted.
hw.gpu.tensor_util.dfma	gauge	avg	avg	—	Tensor Core Breakdown	DFMA (FP64).
hw.gpu.tensor_util.hmma	gauge	avg	avg	—	Tensor Core Breakdown	HMMA (FP16).
hw.gpu.tensor_util.dmma	gauge	avg	avg	—	Tensor Core Breakdown	DMMA (INT8 sparse).
hw.gpu.tensor_util.imma	gauge	avg	avg	—	Tensor Core Breakdown	IMMA (integer).
hw.gpu.fp64_util	gauge	avg	avg	—	FP & Integer Utilization	FP64.
hw.gpu.fp32_util	gauge	avg	avg	—	FP & Integer Utilization	FP32.
hw.gpu.fp16_util	gauge	avg	avg	—	FP & Integer Utilization	FP16.
hw.gpu.integer_util	gauge	avg	avg	—	FP & Integer Utilization	Integer.
hw.gpu.dram_bw_util	gauge	avg	avg	—	DRAM Bandwidth Utilization	% DRAM bandwidth vs max.
hw.gpu.hostmem_cache_hit	gauge	avg	avg	—	L2 Cache Hit Rates	Host memory (PCIe); Grace-coupled only.
hw.gpu.peermem_cache_hit	gauge	avg	avg	—	L2 Cache Hit Rates	Peer GPU memory (NVLink).
hw.gpu.dram_cache_hit	gauge	avg	avg	—	L2 Cache Hit Rates	CPU DRAM (C2C).
hw.gpu.hostmem_cache_miss	gauge	avg	avg	—	L2 Cache Miss Rates	Host memory (PCIe); Grace-coupled only.
hw.gpu.peermem_cache_miss	gauge	avg	avg	—	L2 Cache Miss Rates	Peer GPU memory (NVLink).
hw.gpu.dram_cache_miss	gauge	avg	avg	—	L2 Cache Miss Rates	CPU DRAM (C2C).
hw.gpu.c2c.data_rx	gauge	avg	sum	—	C2C Bandwidth	Chip-to-chip received; Grace/NVL only.
hw.gpu.c2c.data_tx	gauge	avg	sum	—	C2C Bandwidth	Chip-to-chip transmitted.

NVIDIA GPU — MIG partitions

Scoped to MIG devices.

Metric key	Type	Temporal	Spatial	Group-by	Chart / usage	Description
hw.gpu.memory.utilization	gauge	max	sum	—	Combined MIG — GPU Memory Utilization	VRAM % for selected/all MIG devices.
hw.gpu.utilization	gauge	avg	avg	—	Combined MIG — GPU Utilization	Active-processing % for MIG devices.
hw.gpu.tensor_util	gauge	avg	avg	—	Combined MIG — Tensor Core Utilization	Tensor Core % for MIG devices.
hw.gpu.utilization	gauge	avg	avg	gpu.uuid, gpu.name, host.name	MIG devices table	"Utilization" column + device discovery.
hw.gpu.memory.utilization	gauge	max	sum	gpu.uuid, gpu.name, host.name	MIG devices table	"Memory" column per MIG device.
hw.gpu.tensor_util	gauge	avg	avg	gpu.uuid, gpu.name, host.name	MIG devices table	"Tensor Util" column per MIG device.

---

AMD GPU

AMD device metrics are reported per device and are not available per process. Metric availability depends on the GPU and AMD SMI support for that ASIC; without libdrm, GPU names and memory metrics are unavailable.

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
hw.amd_gpu.utilization	gauge	avg	avg	—	GPU Utilization	% time GPU actively processing.
hw.amd_gpu.memory.activity	gauge	avg	avg	—	GPU Memory Activity	How actively the memory controllers service requests (%).
hw.amd_gpu.multimedia.utilization	gauge	avg	avg	—	Multimedia Engine Utilization	Multimedia encode/decode engine utilization (%).
hw.amd_gpu.memory.utilization	gauge	max	sum	—	GPU Memory Utilization	GPU memory currently in use (bytes); reported per device, fleet total is the spatial sum.
hw.amd_gpu.power	gauge	avg	sum	—	GPU Power	Current power usage (mW).
hw.amd_gpu.power_limit	gauge	max	sum	—	GPU Power	Configured power limit (mW).
hw.amd_gpu.temp	gauge	avg	max	—	GPU Temperature	GPU edge temperature (°C), where the ASIC exposes an edge sensor.
hw.amd_gpu.clock.graphics	gauge	avg	avg	—	Clock Frequencies	Current graphics clock (MHz).
hw.amd_gpu.clock.mem	gauge	avg	avg	—	Clock Frequencies	Current memory clock (MHz).

---

Neuron

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
hw.process.neuron.memory.utilization	gauge	avg	sum	process / script / container	Neuron Runtime Memory Usage by process	Runtime memory by process.
hw.process.neuron.inference_count	gauge	avg	sum	process / script / container	Neuron Inferences by process	Successful inferences by process.
hw.process.neuron.model_load_count	gauge	avg	sum	process / script / container	Neuron Model Loads by process	Model loads by process.
hw.process.neuron.flop_count	gauge	avg	sum	process / script / container	Neuron Device FLOP/s by process	Device FLOP/s by process.
hw.process.neuron.memory.utilization	gauge	max	sum	—	Neuron Device Memory Usage	Total device memory usage.
hw.neuron.power.utilization	gauge	max	avg	—	Neuron Device Peak Power Utilization	Peak device power.
hw.neuron.core.inference_count	gauge	avg	sum	—	Neuron Inferences	Total successful inferences.
hw.neuron.core.model_load_count	gauge	avg	sum	—	Neuron Model Loads	Total model loads.
hw.process.neuron.flop_count	gauge	avg	sum	—	Neuron Device FLOP/s	Total device FLOP/s.

---

vLLM

Optionally scoped to a selected model.

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
vllm:generation_tokens_total	gauge	rate / avg	sum	process / script / container	Generated Tokens	Output tokens produced.
vllm:prompt_tokens_total	gauge	rate / avg	sum	process / script / container	Prompt Tokens	Input tokens processed.
vllm:time_per_output_token_seconds	histogram	increase	max	process / script / container	Time per output Token	Per-output-token latency (p50/p95/p99).
vllm:time_to_first_token_seconds	histogram	increase	max	process / script / container	Time to first token	TTFT latency (p50/p95/p99).
vllm:e2e_request_latency_seconds	histogram	increase	max	process / script / container	End to end request latency	E2E latency (p50/p95/p99).
vllm:num_requests_running	gauge	avg	sum	process / script / container	Running Requests	In-flight requests.
vllm:kv_cache_usage_perc	gauge	avg	avg	process / script / container	KV Cache Usage	KV-cache memory utilization %.

---

Triton

Optionally scoped to a selected model.

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
nv_inference_count	gauge	avg	sum	process / script / container	Inferences	Inferences performed (excl. cached).
nv_inference_request_success	gauge	avg	sum	process / script / container	Successful Inferences	Successful requests, all batch sizes.
nv_inference_request_failure	gauge	avg	sum	process / script / container	Failed Inferences	Failed requests, all batch sizes.
nv_inference_exec_count	gauge	avg	sum	process / script / container	Model Executions	Model executions (excl. cached).
nv_inference_request_duration_us	gauge	avg	sum	process / script / container	Request Duration	Cumulative request duration (incl. cached).
nv_inference_queue_duration_us	gauge	avg	sum	process / script / container	Queue Duration	Cumulative queuing duration (incl. cached).
nv_inference_compute_input_duration_us	gauge	avg	sum	process / script / container	Compute Input Duration	Cumulative compute-input duration (excl. cached).
nv_inference_compute_infer_duration_us	gauge	avg	sum	process / script / container	Compute Inference Duration	Cumulative compute-infer duration (excl. cached).
nv_inference_compute_output_duration_us	gauge	avg	sum	process / script / container	Compute Output Duration	Cumulative compute-output duration (excl. cached).
nv_inference_pending_request_count	gauge	avg	sum	process / script / container	Pending Requests	Pending requests awaiting execution per model.
nv_model_load_duration_secs	gauge	avg	sum	process / script / container	Model Load Duration	Model load duration.
nv_pinned_memory_pool_used_bytes	gauge	avg	sum	process / script / container	Pinned Memory Pool Used	Pinned memory pool used.
nv_pinned_memory_pool_total_bytes	gauge	avg	sum	process / script / container	Pinned Memory Pool Total	Total pinned memory pool size.

---

SGLang

Optionally scoped to a selected model.

Metric key	Type	Temporal	Spatial	Group-by	Chart	Description
sglang:generation_tokens_total	gauge	avg	sum	process / script / container	Generated Tokens	Output tokens produced.
sglang:num_requests_total	gauge	avg	sum	process / script / container	Requests Processed	Requests processed.
sglang:prompt_tokens_total	gauge	avg	sum	process / script / container	Prompt Tokens	Input tokens processed.
sglang:gen_throughput	gauge	avg	sum	process / script / container	Generation Throughput	Tokens/s generation throughput.
sglang:time_to_first_token_seconds	histogram	increase	max	process / script / container	Time to first token	TTFT latency (p50/p95/p99).
sglang:e2e_request_latency_seconds	histogram	increase	max	process / script / container	End to end request latency	E2E latency (p50/p95/p99).
sglang:cache_hit_rate	gauge	avg	avg	process / script / container	Prefix Cache Hit Rate	Prefix cache hit rate.
sglang:num_running_reqs	gauge	avg	sum	process / script / container	Running Requests	In-flight requests.
sglang:num_queue_reqs	gauge	avg	sum	process / script / container	Queued Requests	Requests in waiting queue.

---

Host summary cards

Metrics behind the host-stats summary cards. Spatial aggregation is max (or sum depending on the view).

Metric key	Type	Temporal	Spatial	Group-by	Purpose
host.core_count	gauge	max	max|sum	—	CPU cores card.
host.memory	gauge	max	max|sum	—	Memory card.
gpu.memory	gauge	max	max|sum	—	GPU memory card.
gpu.count	gauge	max	max|sum	—	GPU count card.
amd_gpu.memory	gauge	max	max|sum	—	AMD GPU memory card.
amd_gpu.count	gauge	max	max|sum	—	AMD GPU count card.
neuron.memory	gauge	max	max|sum	—	Neuron memory card.
neuron.device.count	gauge	max	max|sum	—	Neuron device count card.
neuron.core.count	gauge	max	max|sum	—	Neuron core count card.