What is Zymtrace?
AI training and inference fleets lose throughput and leave expensive accelerators idle because bottlenecks can hide across application code, CPU execution, GPU kernels, runtimes, interconnects, and hardware topology. Finding and fixing them across a heterogeneous cluster requires correlating metrics and profiles across layers, hosts, and accelerators.
Zymtrace is the autonomous optimization layer for heterogeneous, multi-silicon AI fleets. Its always-on, zero-friction, cluster-wide introspection captures how workloads execute from application code down to GPU instructions. Hardware-aware analysis identifies the kernels, code paths, and topology issues wasting compute. Profile-guided recommendations and coding agents turn those findings into fixes.
Squeeze more FLOPs from the hardware you already own, with no code changes, no recompilation, and no system restarts.
Read our GPU profiling launch blog post to see how Zymtrace helps organizations unlock efficient AI by identifying performance bottlenecks in CUDA kernels, optimizing inference batch sizes, and eliminating idle GPU cycles, all with zero friction.
Unified optimization layer for multi-silicon fleets​
Silicon-aware and always-on, whether your fleet is all NVIDIA or a mix: Zymtrace supports NVIDIA and AMD GPUs and AWS Neuron accelerators, all reporting into one self-hosted backend and one set of views. Support for Google TPUs and Cerebras is coming soon.
GPU topology health​
When a performance problem isn't in your code, it's in your hardware. Zymtrace surfaces GPU health, topology, and interconnect issues that waste the compute you paid for:
- NVLink health: CRC, replay, and recovery per link
- ECC and XID: memory and driver errors, fleet-wide
- NUMA affinity: catch non-affine CPU placement
- Actionable recommendations: not just an alert
Inference, training, and everything underneath​
Use Zymtrace to debug and optimize inference and training on the engines you already use, from vLLM and SGLang to distributed training, so you maximize throughput per GPU, per watt.
Disaggregated inference, end to end​
Metrics and profiles collected from the inference engines themselves, across prefill and decode, on heterogeneous accelerators. Drill from a dip in token throughput straight into the stall behind it. See Inference Server Metrics.
Training and distributed runs​
One execution flow across the CPU/GPU boundary, from Python down to the exact GPU instruction. Attribute everything to owners: SLURM jobs, MIG instances, Kubernetes namespaces, containers, and custom tags, across multi-GPU, multi-host fleets.
Whole-system profiling, every language​
Continuous profiles of every process on the box, compiled and interpreted alike, with automatic symbolization of native and GPU frames. See Supported Languages.
Recommendations built on top​
Everything above feeds one ranked list: what to change, why it matters, and where to apply it.
Tracing starts where execution starts: the CPU​
Most GPU clusters run under 40% utilization, and the bottleneck is rarely the hardware. Zymtrace picks up execution on the CPU with eBPF and traces it down to the accelerator. An MRI for your stack: it exposes the bottlenecks hiding between the layers, with minimal overhead.
Refer to the architecture page for the system design, and the GPU Profiler Architecture for how GPU profiling works across silicon vendors.
Profile-guided agentic optimization​
Zymtrace enriches coding agents with performance data over MCP, so they find the bottleneck and open a pull request:
- MCP server: flamegraphs, top functions, and GPU metrics exposed as MCP tools. Any MCP-capable agent can query production directly.
- Agent skills: ready-made skills teach your coding agent the whole optimization workflow: rank consumers, read flamegraphs, interpret stall reasons, apply the fix.
- AI assistant: the built-in assistant reads flamegraphs and produces ranked hotspots with concrete fixes.
Deploys into your existing stack, unchanged​
No re-architecting. No code changes. The agent drops into whatever you already run.
- Acceleration: NVIDIA CUDA 12.x and above, AMD ROCm 7.x and above, and AWS Neuron. Google TPUs and Cerebras coming soon.
- ML Frameworks: PyTorch, JAX
- CPU Profiling: Python, C/C++, Java, Go, Rust, Node.js, Ruby, PHP, .NET, Perl
- Environments: Cloud, containers, Kubernetes, SLURM, on-premises
Open standards, no lock-in​
Zymtrace is self-hosted. Your profiling data lives in your infrastructure, in databases you control. You can bring existing instances for all three storage backends:
| Storage | Purpose | |
|---|---|---|
| ClickHouse | Profiling events and analytics | Guide |
| PostgreSQL | User data, metadata, project configuration | Guide |
| S3 / MinIO | Native debug symbols | Guide |
Because the data is in your ClickHouse, you can query it with SQL at any time, independently of the Zymtrace UI.
Every piece of data visible in the UI is also accessible via a REST API. Click API Explorer inside your Zymtrace instance to browse and test every endpoint interactively.
OpenTelemetry: Zymtrace is built by the team that pioneered, open-sourced, and donated the eBPF profiler to OpenTelemetry. The founders were core contributors to the OTel Profiles specification (recently reached alpha) and helped define it inside the OpenTelemetry project. Zymtrace is fully OTel-compliant for resource attributes and metadata. By default the profiler uses a custom wire protocol that is 6x more compute-efficient than the OTel Profiles proto; you can switch between the Zymtrace and OTel Profiles protocol at any time without losing data. See Profiler ENV & CLI Args.
Get started​
- Quick Start: backend plus profiler in three steps with Helm or Docker Compose
- GPU Profiler Quick Start: profile your first GPU workload in minutes
- Install with an AI assistant: let Claude Code, Codex, or Cursor run the install for you