GPU Timeline

The timeline view plots what each GPU was doing, moment to moment, over a short window: the CUDA kernels launched on every stream and how much GPU memory each process held. Use it to line up a memory spike with the work running at that moment, or to see how work is distributed across streams and GPUs.

Unlike the GPU metrics dashboard, which trends utilization and memory over long ranges, the timeline is a high-resolution, per-kernel view meant for zooming into a few seconds of activity.

Beta feature, requires high-resolution profiling

The timeline is in Beta. It is populated only while high-resolution profiling is enabled, which has high overhead — turn it on for short windows and scope it to the specific hosts or pods you are investigating. See Enabling high-resolution profiling below.

Enabling high-resolution profiling

High-resolution profiling is off by default. When enabled, it collects per-kernel-launch CUDA events with nanosecond precision instead of aggregating them, and samples GPU memory far more often than the default -collect-metrics-interval. The denser memory sampling increases the volume of memory series sent over the wire and stored.

Enable it from a rule under Settings → Agent Config:

1. Open Settings → Agent Config and click + Add Rule.
2. Scope the rule with a CEL expression to the process, container, or host you want to inspect.
3. Turn on the High-Resolution Profiling setting.
4. Set an expiry so profiling stops on its own. High-resolution rules require one.

Keep the window short

Per-kernel collection is expensive. High-resolution profiling records every kernel launch instead of sampling them, so its cost scales with your launch rate. On a high-throughput workload it takes roughly 400mc of extra CPU for the profiled scope, on top of the baseline cost in the profiler resource guide. Enable it just long enough to capture the behavior you are chasing, on as narrow a scope as possible, then let the rule expire.

Opening the timeline

The timeline lives behind an overview chart showing high-resolution event density over your selected range. Brush a sub-window on that chart to load its detail. The profiler builds the timeline for that slice and renders it.

If the brushed window holds too many events, the view reports "Too many events to load" instead of rendering. Select a shorter interval, or narrow the query, and try again.

If a range shows "No high-resolution data in this range", high-resolution profiling was not active for those hosts during that time. Enable it (see above) and capture a fresh window.

Reading the timeline

Tracks are grouped top-down by process, then by GPU, then by the streams on that GPU:

my-training-job (pid 4812)      ← process
  GPU 0                         ← GPU on that process
    Stream 7                    ← CUDA stream: kernel launches
    Stream 13
    GPU Memory                  ← memory the process held on that GPU
  GPU 1
    Stream 7
    GPU Memory

• Process groups every GPU a single process touched, labeled with the executable name and pid.
• GPU groups the lanes for one physical GPU (or MIG instance). Turn on Show GPU UUID to display the device UUID beneath the GPU name.
• Stream lanes show the CUDA kernel launches on that stream, laid out on the shared time axis. Pan and zoom to move through the window.
• GPU Memory lanes show how much memory the process held on that GPU over time.

The GPU memory graph

Each GPU Memory lane draws the sampled memory series as a filled area under a line. Under high-resolution profiling the series is sampled many times per second, so short-lived allocation spikes are visible rather than averaged away. Hover any point to read its timestamp and byte value.

By default the lane's vertical scale spans the minimum and maximum of the whole captured series, so its height is comparable as you pan. Enable Scale GPU memory to current view to rescale the graph to only what is visible, which exaggerates small variations within the current window.

Timeline options

Open the Options popover (gear icon) above the timeline to toggle:

Option	Effect
Collapse to CUDA-only frames	Hide non-CUDA frames in collapsed stream lanes, leaving only the CUDA call stacks.
Show relative offset	Label the time axis as an offset from the start of the window instead of absolute wall-clock time.
Scale GPU memory to current view	Rescale every GPU memory graph to the min/max of the visible range, updating live as you pan and zoom. When off, the scale spans the whole series.
Show GPU UUID	Show each GPU's device UUID as a detail line under its group header.

Reset restores the defaults.

Reordering lanes

Stream and GPU memory lanes carry a drag grip (the dotted handle on the left of each lane's label). Drag a lane up or down to reorder it within its GPU group — for example, to place the GPU Memory lane directly beside the stream you are comparing it against. Neighboring lanes shift aside as the dragged lane's edge passes them, and drop into place when you release.

Lanes can only be reordered among their siblings under the same GPU; you cannot move a lane into a different process or GPU.

Collapsing lanes and groups

• Click a process or GPU header to collapse its subtree and hide everything beneath it.
• Click a stream lane's caret (on the right) to collapse it into a compact band, freeing vertical space while keeping the lane on screen.