Skip to main content

Tutorial Overview

Welcome to zymtrace! This guide covers all the features and capabilities that help you analyze performance, identify bottlenecks, and optimize your applications using continuous profiling data.

Core Analysis Views​

zymtrace provides five main views for analyzing your application performance:

Efficiency IQ - Your Performance Dashboard​

Your main entry point showing:

  • Overall compute utilization across your entire fleet
  • Top 5 most expensive functions consuming CPU/GPU resources
  • Top 5 entities (hosts, deployments, containers) by resource consumption
  • Absolute vs relative CPU usage modes for infrastructure and workload analysis

Perfect for: Getting a quick system overview and identifying what needs attention first.

Top Functions - Function-Level Analysis​

Identify the most resource-intensive functions across your applications:

  • CPU View: Find functions consuming the most CPU time
  • GPU View: Discover GPU kernels and CPU-side GPU operations
  • Sandwich View: See callers and callees for any function to understand execution context
  • Filter by language, service, or infrastructure component

Perfect for: Code-level optimization and understanding execution patterns.

GPU & CPU Flamegraphs - Interactive Call Stack Visualization​

Explore your application's execution flow with interactive flamegraphs:

  • Mixed-Stack Visualization: See call stacks across multiple runtimes in a single view
  • Interactive Controls: Pan, zoom, search, and focus on specific functions
  • CPU & GPU Profiles: Analyze both CPU performance and GPU utilization patterns
  • Advanced Filtering: Group by executable, script, host, or thread for targeted analysis

Perfect for: Understanding execution flow, identifying bottlenecks, and visualizing performance patterns across your entire stack.

Top Entities - Infrastructure Analysis​

Strategic macro-level view of resource consumption:

  • Hosts: See which machines are consuming the most resources
  • Namespaces/Pods: Understand Kubernetes resource distribution
  • Containers: Identify resource-heavy containers
  • Main Executables: Track consumption by process or runtime

Perfect for: Infrastructure planning, resource allocation decisions, and finding system bottlenecks.

Diff Functions - Performance Comparison​

Compare performance between different states:

  • Release Validation: Before vs after deployment comparison
  • A/B Testing: Compare feature variants or configurations
  • Blue/Green Deployments: Validate performance during traffic migration
  • Optimization Tracking: Measure the impact of code changes

Perfect for: Detecting regressions, validating optimizations, and measuring change impact.

Essential Tools​

Filtering Data - Focus Your Analysis​

Master the filtering system to slice and dice your data:

  • Click & Apply: Quick filtering from dropdown menus
  • Query Mode: Advanced filtering using CEL syntax and OTEL attributes
  • Global Filters: Apply filters across all views consistently

Unit Reference - Understanding Metrics​

Essential guide to zymtrace measurements:

  • CPU Modes: Absolute vs relative percentage calculations
  • Time Metrics: Self time vs total time explained
  • Core Units: Millicores, cores, kilocores, and megacores
  • GPU Metrics: Understanding GPU profiling data

Here's how to effectively use zymtrace for performance analysis:

1. Start with Efficiency IQ​

  • Get an overview of your system's resource consumption
  • Identify the top resource-consuming functions and entities
  • Toggle between absolute and relative CPU modes based on your analysis goal

2. Apply Filtering​

  • Use entity filters to focus on specific services, environments, or time periods
  • Combine multiple filters for precise analysis (e.g., specific pods + Python functions)
  • Switch between click-and-apply and query mode based on complexity

3. Deep Dive Analysis​

  • For code optimization: Use Flamegraphs, Top Functions → Sandwich View
  • For infrastructure decisions: Use Top Entities → drill Flamegraphs or Top functions
  • For release validation: Use Diff to compare before/after states
Need Help?

If you have questions or need assistance, reach out to us at [email protected] or join our community discussions.