Skip to main content

Unit Reference

This page explains the key metrics and units used throughout zymtrace.

CPU % Mode: Absolute vs Relative​

The CPU % Mode toggle allows you to switch between two different ways of calculating CPU percentages:

  • Abs (Absolute) – Shows real CPU usage percentages—just like what you'd see in tools such as htop or btop. For example, if your cluster has 2 machines with 8 cores each (16 cores total), and your workloads are consuming 4 cores on each machine (8 cores total), zymtrace will display 50% CPU usage for that service.

  • Rel (Relative) – Presents CPU utilization in relation to other workloads you're profiling, helping you see which services dominate your compute resources. This shows each entity's percentage of the total consumed compute across all profiled workloads.

Use Absolute mode to understand real infrastructure utilization. Use Relative mode to understand the distribution of CPU consumption among actively profiled workloads.

Core Size Reference​

Core usage values are displayed with the following units:

  • mc - millicore (1/1000 core)
  • c - core
  • kc - kilocore (1000 cores)
  • Mc - megacore (1,000,000 cores)

Self vs Total Time​

Understanding the difference between self and total time is crucial for identifying optimization opportunities:

Self Time​

Time spent executing the function's own code only, excluding any functions it calls.

  • Indicates if the function itself is expensive
  • High self time = optimize the function's algorithm
  • Measures the direct cost of the function's instructions

Total Time​

Time including all called functions (the function plus everything it calls).

  • Shows the full cost of calling this function
  • High total time with low self time = optimize what it calls
  • Measures the complete execution path

Example​

Function: processData()
Self CPU: 2%    (function is fast)
Total CPU: 45%  (but calls expensive operations)
→ Optimize what it calls, not the function itself

In this example, processData() spends very little time in its own code (2%), but the functions it calls consume 43% of CPU time. The optimization opportunity is in the called functions, not processData() itself.

Unit Definitions​

zymtrace uses specific units and metrics to measure and display resource consumption across your infrastructure. Understanding these units is essential to fully utilize the platform. The following sections detail the units used for GPU and CPU profiling.

GPU Units​

UnitDescription
Self TimeFor CUDA frames, Self Time reflects the time spent executing this particular function on the GPU, excluding any functions it called. For CPython and Native frames this is always zero because they are not actually executed on the GPU.
Total TimeFor CPython and native frames, Total Time indicates the GPU time consumed by CUDA kernels launched by the function (including its callees); for CUDA frames, it reflects time spent executing the function's body and any functions it called on the GPU.
Self GPU %For CUDA frames, Self GPU reflects the percentage of time spent executing this particular function on the GPU, excluding any functions it called. For CPython and Native frames this is always zero because they are not actually executed on the GPU.
Total GPU %For CPython and native frames, Total GPU indicates the percentage of GPU time consumed by CUDA kernels launched by the function (including its callees); for CUDA frames, it reflects percentage of time spent executing the function's body and any functions it called on the GPU.
Frame Types

The data above varies based on the frame type:

  • CUDA frames: Represent actual GPU kernel execution
  • CPython/Native frames: Represent CPU code that launches GPU kernels

When to Use Self vs Total (GPU)​

Use Self GPU Time to identify expensive GPU kernels, find kernels that need optimization, or look for inefficient CUDA implementations.

Use Total GPU Time to understand the full GPU cost of a function, find which high-level functions trigger expensive GPU operations, or decide what GPU workloads to optimize or replace.