GPU Profiler environment variables
zymtrace GPU profiler​
The following environment variables enable and control the behavior of the zymtrace GPU profiler. You are required to set them as part of your CUDA application startup arguments. Refer to the docker and K8s manifest examples.
| Environment Variable | Description |
|---|---|
CUDA_INJECTION64_PATH | Enable CUDA profiler by setting env path to libzymtracecudaprofiler, Example: `CUDA_INJECTION64_PATH="/opt/zymtrace/profiler/libzymtracecudaprofiler.so". |
ZYMTRACE_CUDAPROFILER__COLLECT_PER_GPU | Whether kernels should be aggregated per GPU or globally. Example: ZYMTRACE_CUDAPROFILER__COLLECT_PER_GPU=false. Default is true. |
ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING | Override for enabling PC sampling (takes precedence over auto-detection). Example: ZYMTRACE_CUDAPROFILER__ENABLE_PC_SAMPLING=false. Default is false. |
ZYMTRACE_CUDAPROFILER__NOOP | Whether to do nothing at all and exit immediately without profiling. Example: ZYMTRACE_CUDAPROFILER__NOOP=true. Default is false. |
ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD | Whether to print overhead statistics (CPU, memory usage) to the console. Example: ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD=true. Default is false. |
ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD_INTERVAL | Time between printing overhead metrics. Example: ZYMTRACE_CUDAPROFILER__PRINT_OVERHEAD_INTERVAL=10s. Default is 5s. |
ZYMTRACE_CUDAPROFILER__PRINT_STATS | Whether to periodically dump profiling stats to console. Example: ZYMTRACE_CUDAPROFILER__PRINT_STATS=true. Default is false. |
ZYMTRACE_CUDAPROFILER__PRINT_STATS_INTERVAL | Time between printing stats. Example: ZYMTRACE_CUDAPROFILER__PRINT_STATS_INTERVAL=10s. Default is 5s. |
ZYMTRACE_CUDAPROFILER__QUIET | If set to false, enable output to the console. Example: ZYMTRACE_CUDAPROFILER__QUIET=false. Default is true (silent operation). None of the PRINT_* settings have any effect unless this is false. |
important
PC sampling requires appropriate system permissions. Either CAP_SYS_ADMIN privileges or RmProfilingAdminOnly=0 in NVIDIA driver setting.
Check this by running cat /proc/driver/nvidia/params | grep RmProfilingAdminOnly