Skip to main content

Platform Architecture

The zymtrace platform is only available for on-premises/self-hosted installation, allowing you to host and manage zymtrace entirely within your infrastructure for full control over your data and setup. The platform consists of two main components:

  1. zymtrace profiler: This is the eBPF-based agent that needs to be installed on the machines running your applications. It collects performance profiles from both GPU & CPU-bound workloads.
  2. Backend services: The backend services store, process, and analyze performance profiles. All our core backend services are written in Rust 🦀. The front-end is a combination of ReactJS and WASM.

The diagram below depicts a high-level architecture of the components:

Architecture Architecture
info

Need a hosted zymtrace backend?

We can provision a dedicated cloud deployment for you. Email us at [email protected] or request access here

Components overview​

zymtrace profiler​

The zymtrace profiler runs on each node, deployed either as a Kubernetes DaemonSet or as a standalone binary on a standard VM. It collects performance profiles of resource-intensive processes on the node, aggregates and compresses them, and sends them to the backend via gRPC. TLS is supported by default, with an option to disable it if needed. Here's a more detailed description of how it works:

Agent architecture Agent architecture
  1. Unwinder eBPF programs are loaded into the kernel.
  2. The kernel verifies the safety of the BPF program. If approved, the program is attached to probes and triggered upon specific events.
  3. The eBPF programs collect data and pass it to userspace via maps.
  4. The agent retrieves the collected data from maps. This data includes process-specific and interpreter-specific meta-information, which helps the eBPF unwinder programs perform mixed-stack unwinding across different languages (e.g., Python calling into C libraries).
  5. The agent pushes stack traces, metrics, and metadata to the zymtrace gateway service for routing to the appropriate backend services for analysis.
  6. Easily identify & optimize the most inefficient functions across your entire infrastructure.

zymtrace GPU profiler​

GPU utilization, temperature, power—those metrics are easy to collect, but they don't explain the inner workings of the GPU or why it is idling. That's why we didn't start with NVML or DCGM metrics. We started deeper, closer to the hardware. That's where real insight lives, not just in metrics.

zymtrace profiles down to CUDA kernels, disassembles CUDA SASS mnemonics, exposes GPU stall reasons, and correlates it all back to the CPU traces that launched the kernels, without requiring recompilation.

Learn more: GPU Profiler deep dive →

zymtrace backend services​

The zymtrace backend is designed to store, process and visualize profiling data efficiently. Below is an overview of the key backend services and their roles:

gateway service​

The gateway service acts as the main entry point for all user requests, using Envoy Proxy for request routing and load balancing. It routes requests to the appropriate backend services (UI, ingest, symDB) and handles mTLS when configured.

UI service​

The UI service serves the web-based user interface for zymtrace, built with ReactJS and WASM. Users access the UI through the gateway service, which routes web requests to this service.

ingest service​

The ingest service receives profiling data from the zymtrace profiler via the gateway service. The gateway routes profiling data requests to the ingest service based on the request path. The service is also responsible for storing profiling events in ClickHouse, a high-performance database designed for ultra-fast querying and analysis.

symDB service​

The symDB service handles symbol resolution upon request. It retrieves native symbols stored in S3/Minio or fetches them from the global symbolization service. This service is critical for converting raw profiling data into meaningful stack traces by resolving both native and interpreted symbols.

Learn more: Symbolization deep dive →

identity service​

zymtrace provides the granularity to segregate your profiling data into different projects within an organization. The identity service currently manages these projects. It associates incoming profiling data from the ingest service with the correct project, laying the foundation for future user authentication and role-based access control.

Storage​

  • ClickHouse: Stores all profiling events and interpreted symbols.
  • Postgres: Stores user data, meta-data and project information.
  • S3/Minio: Stores native symbols.

Get started​

zymtrace backend (On-Premises)​

Refer to our on-premises installation guide for detailed instructions.

zymtrace profiler​

Refer to the profiler host agent installation guide for more details.