GPU Profiler Environment Variables

NVIDIA (CUDA)

Zymtrace GPU profiler

The following environment variables enable and control the behavior of the Zymtrace GPU profiler. You are required to set them as part of your CUDA application startup arguments. Refer to the docker and K8s manifest examples.

Environment Variable	Description
CUDA_INJECTION64_PATH	Enable CUDA profiler by setting env path to libzymtracecudaprofiler, Example: `CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so".
NVTX_INJECTION64_PATH	Enable NVTX support in the profiler by setting env path to libzymtracecudaprofiler, Example: `NVTX_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so".
ZYMTRACE_CUDAPROFILER__COLLECT_PER_GPU	Whether kernels should be aggregated per GPU or globally. Example: ZYMTRACE_CUDAPROFILER__COLLECT_PER_GPU=false. Default is true.
ZYMTRACE_CUDAPROFILER__ENABLE_NVTX	Turn NVTX support on or off. Note: requires the NVTX_INJECTION64_PATH path variable to be set as well. Example: ZYMTRACE_CUDAPROFILER__ENABLE_NVTX=false. Default is true.
ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING	Override for enabling PC sampling (takes precedence over auto-detection). Example: ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING=false. Default is false.
ZYMTRACE_CUDAPROFILER__NOOP	Whether to do nothing at all and exit immediately without profiling. Example: ZYMTRACE_CUDAPROFILER__NOOP=true. Default is false.
ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD	Whether to print overhead statistics (CPU, memory usage) to the console. Example: ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD=true. Default is false.
ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD_INTERVAL	Time between printing overhead metrics, in seconds (fractions allowed). Example: ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD_INTERVAL=10. Default is 5.
ZYMTRACE_CUDAPROFILER__PRINT_STATS	Whether to periodically dump profiling stats to console. Example: ZYMTRACE_CUDAPROFILER__PRINT_STATS=true. Default is false.
ZYMTRACE_CUDAPROFILER__PRINT_STATS_INTERVAL	Time between printing stats, in seconds (fractions allowed). Example: ZYMTRACE_CUDAPROFILER__PRINT_STATS_INTERVAL=10. Default is 5.
ZYMTRACE_CUDAPROFILER__QUIET	If set to false, enable output to the console. Example: ZYMTRACE_CUDAPROFILER__QUIET=false. Default is true (silent operation). None of the PRINT_* settings have any effect unless this is false.
ZYMTRACE_CUDAPROFILER__STACK_TRACE_SAMPLING__ENABLED	Whether to sample kernel launches, rather than profiling all launches. Default is true.
ZYMTRACE_CUDAPROFILER__STACK_TRACE_SAMPLING__TARGET	How many kernels to profile, per (OS) thread, per WINDOW_SIZE. Default is 200.
ZYMTRACE_CUDAPROFILER__STACK_TRACE_SAMPLING__WINDOW_SIZE	The duration, in seconds (fractions allowed), over which to reach the above defined TARGET sample rate. For example, if this is 1, that means profile (up to) $TARGET kernels / second. Default is 1.

important

PC sampling requires appropriate system permissions. Either CAP_SYS_ADMIN privileges or RmProfilingAdminOnly=0 in NVIDIA driver setting.

Check this by running cat /proc/driver/nvidia/params | grep RmProfilingAdminOnly

AMD (ROCm)

Zymtrace ROCm profiler

The following environment variables enable the Zymtrace ROCm profiler. They must be set as part of your ROCm application startup: the ROCm profiler cannot attach to an already-running GPU process. Refer to the AMD GPU Metrics & Profiles page for Docker and Kubernetes examples.

Environment Variable	Description
ROCP_TOOL_LIBRARIES	Enables the ROCm profiler by pointing ROCprofiler at the Zymtrace tool library. Example: ROCP_TOOL_LIBRARIES="/opt/zymtrace/profiler/libzymtracerocmprofiler.so".
ROCPROFILER_REGISTER_LIBRARY	Path to the rocprofiler-sdk runtime that ships with the Zymtrace profiler, so the workload loads Zymtrace's copy rather than the host ROCm installation's. Example: ROCPROFILER_REGISTER_LIBRARY="/opt/zymtrace/profiler/librocprofiler-sdk.so.1".

The following variables control the behavior of the ROCm profiler once it is loaded. All are optional.

Environment Variable	Description
ZYMTRACE_ROCMPROFILER__COLLECT_PER_GPU	Whether kernels should be aggregated per GPU or globally. Example: ZYMTRACE_ROCMPROFILER__COLLECT_PER_GPU=false. Default is true.
ZYMTRACE_ROCMPROFILER__ENABLE_ROCTX	Turn ROCTX integration on or off. Example: ZYMTRACE_ROCMPROFILER__ENABLE_ROCTX=false. Default is true.
ZYMTRACE_ROCMPROFILER__PROFILE_MEMORY	Whether to profile memory-related operations (memcpy, memset). Example: ZYMTRACE_ROCMPROFILER__PROFILE_MEMORY=false. Default is true.
ZYMTRACE_ROCMPROFILER__PROFILE_SYNC	Whether to profile synchronization-related operations. Example: ZYMTRACE_ROCMPROFILER__PROFILE_SYNC=false. Default is true.
ZYMTRACE_ROCMPROFILER__NOOP	Whether to do nothing at all and exit immediately without profiling. Example: ZYMTRACE_ROCMPROFILER__NOOP=true. Default is false.
ZYMTRACE_ROCMPROFILER__PRINT_STATS	Whether to periodically dump profiling stats to the console. Example: ZYMTRACE_ROCMPROFILER__PRINT_STATS=true. Default is false.
ZYMTRACE_ROCMPROFILER__PRINT_STATS_INTERVAL	Time between printing stats, in seconds (fractions allowed). Example: ZYMTRACE_ROCMPROFILER__PRINT_STATS_INTERVAL=10. Default is 5.
ZYMTRACE_ROCMPROFILER__QUIET	If set to false, enable output to the console. Example: ZYMTRACE_ROCMPROFILER__QUIET=false. Default is true (silent operation). PRINT_STATS has no effect unless this is false.
ZYMTRACE_ROCMPROFILER__STACK_TRACE_SAMPLING__ENABLED	Whether to sample kernel launches, rather than profiling all launches. Note: a HIP graph is sampled as a unit, so skipping one skips the full graph. Default is true.
ZYMTRACE_ROCMPROFILER__STACK_TRACE_SAMPLING__TARGET	How many kernels to profile, per (OS) thread, per WINDOW_SIZE. Default is 200.
ZYMTRACE_ROCMPROFILER__STACK_TRACE_SAMPLING__WINDOW_SIZE	The duration, in seconds (fractions allowed), over which to reach the above defined TARGET sample rate. For example, if this is 1, that means profile (up to) $TARGET kernels / second. Default is 1.
ZYMTRACE_ROCMPROFILER__MEMORY_SAMPLING__ENABLED	Whether to sample memory operations, rather than profiling all of them. Default is true.
ZYMTRACE_ROCMPROFILER__MEMORY_SAMPLING__TARGET	How many memory operations to profile, per (OS) thread, per WINDOW_SIZE. Default is 500.
ZYMTRACE_ROCMPROFILER__MEMORY_SAMPLING__WINDOW_SIZE	The duration, in seconds (fractions allowed), over which to reach the memory sampling TARGET. Default is 1.
ZYMTRACE_ROCMPROFILER__SYNC_SAMPLING__ENABLED	Whether to sample synchronization operations, rather than profiling all of them. Default is true.
ZYMTRACE_ROCMPROFILER__SYNC_SAMPLING__TARGET	How many synchronization operations to profile, per (OS) thread, per WINDOW_SIZE. Default is 500.
ZYMTRACE_ROCMPROFILER__SYNC_SAMPLING__WINDOW_SIZE	The duration, in seconds (fractions allowed), over which to reach the sync sampling TARGET. Default is 1.

Console output

To see profiler log output, set ZYMTRACE_ROCMPROFILER__QUIET=false and use RUST_LOG to select the level, e.g. RUST_LOG=zymtracerocmprofiler=info.

important

The ROCm profiler is available on x86_64 hosts only. For upcoming AMD capabilities, see the AMD roadmap.