Skip to main content

Upload native symbols

zymtrace requires debug symbols to display function names and line numbers in traces from applications written in languages that compile to native code (C, C++, Rust, Go).

  • C/C++ applications must be built with debug symbols (-g) for symbolization to work.
  • Rust applications must be built with debug = "line-tables-only" (or a higher debug level that also includes line tables).
  • Go binaries do not require any special compiler flags and include debug information even if the executable is fully stripped.

The zymtrace-cli upload-symbols subcommand allows you to upload debug symbols from both stripped and unstripped executables.

Download zymtrace-cli​

First, download the zymtrace-cli tool.

curl -L https://dl.zystem.io/zymtrace/latest/arm64/zymtrace-cli -o zymtrace-cli
chmod +x zymtrace-cli
info

zymtrace-cli is only compatible with Linux machines.

Usage​

Use the help command to get started.

./zymtrace-cli upload-symbols -h

Usage: zymtrace-cli upload-symbols --project-id <PROJECT_ID> --endpoint <ENDPOINT> <EXECUTABLE> [DEBUG_OBJECT]

Arguments:
  <EXECUTABLE>    Path to the executable
  [DEBUG_OBJECT]  Extract symbols from a split debug object instead

Options:
  -p, --project-id <PROJECT_ID>  Project ID to upload to
  -e, --endpoint <ENDPOINT>      Destination for symbol uploads
  -h, --help                     Display help information
  • The project-id can be found in the zymtrace UI, within the URL. The default project ID is 00000000-0000-0000-0000-000000000000.
  • The endpoint refers to the zymtrace UI. If you are using an ingress controller, this is the UI ingress.
tip

If you're using an ingress controller, ensure you configure it by setting nginx.ingress.kubernetes.io/proxy-body-size: "0" to allow unlimited file size uploads, or specify a hard limit, e.g., 5GB. This example assumes that you are using Nginx. See the ingress docs for details.

Depending on how you are deploying your executables to production with or without symbols, we need to use one of two different approaches.

Executables are deployed with symbols (unstripped)​

This is the simplest case. All that you need to do it to run:

./zymtrace-cli upload-symbols \
  --project-id 00000000-0000-0000-0000-000000000000 \
  --endpoint http://zymtrace.internal:80 fincore_llm

This command uploads the debug symbols for the fincore_llm binary to the zymtrace UI at http://zymtrace.internal:80

Executables are stripped in production​

zymtrace uses a hash-based identifier to map executables to their symbols. When the executable is stripped, the hash changes and symbols uploaded from the unstripped executable can no longer be associated with the profiles collected for the stripped variant.

To handle this case, the upload-symbols command has a second mode of operation. If two files are passed, the first first one is used to compute the hash (the stripped executable) while symbols are extracted from the second one (the unstripped executable).

./zymtrace-cli upload-symbols \
  --project-id 00000000-0000-0000-0000-000000000000 \
  --endpoint http://zymtrace.internal:80 fincore_llm_stripped fincore_llm_unstripped

To illustrate this further, if you want to strip your executables but still upload symbols for them, you'd go about that like so:

# Build your applications with symbols, then:
cp ./fincore_llm ./fincore_llm_stripped
strip ./fincore_llm_stripped
./zymtrace-cli upload-symbols \
  --project-id 00000000-0000-0000-0000-000000000000 \
  --endpoint http://zymtrace.internal:80 fincore_llm fincore_llm_unstripped 
rm ./fincore_llm
tip

This variant is also applicable for builds that produce separate debug objects (e.g. split DWARF).