Monitoring with Prometheus

The Zymtrace backend services collect internal metrics such as request rates and latencies, ingestion and drop counters, LLM token usage, and allocator statistics. Every service exposes them at GET /metrics in the Prometheus text exposition format, for Prometheus or any other compatible scraper.

Endpoints

Metrics are served on each service's internal HTTP listener:

Service	Port	Listener
identity	7831	health-check
ingest	8377	health-check
symdb	6109	health-check
web	9933	REST API

All metrics are prefixed with zt_. The endpoints are enabled by default and are only reachable inside your cluster or Docker network; the Zymtrace gateway never routes /metrics, so they are not exposed publicly.

Scraping in Kubernetes

PodMonitor (prometheus-operator)

If your cluster runs the prometheus-operator (e.g. kube-prometheus-stack), enable the bundled PodMonitors in your custom-values.yaml:

metricReporter:
  podMonitor:
    enabled: true
    # Match your operator's podMonitorSelector, e.g. for kube-prometheus-stack:
    additionalLabels:
      release: kube-prometheus-stack
    # Optional: override the operator's default scrape interval
    interval: 30s

This creates one PodMonitor per backend service, scraping /metrics on every pod.

Pod annotations

For annotation-based service discovery, add the conventional Prometheus hints to the pods of each service:

services:
  ingest:
    podAnnotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8377"
      prometheus.io/path: "/metrics"
  identity:
    podAnnotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "7831"
      prometheus.io/path: "/metrics"
  symdb:
    podAnnotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "6109"
      prometheus.io/path: "/metrics"
  web:
    podAnnotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9933"
      prometheus.io/path: "/metrics"

Scraping with Docker Compose

The compose deployment publishes the metrics ports on the host loopback: ingest (8377), symdb (6109), and web (9933) are always available; identity (7831) only when the deployment runs in dev mode. A minimal scrape config for a Prometheus running on the same host:

scrape_configs:
  - job_name: zymtrace
    static_configs:
      - targets: ["127.0.0.1:8377"]
        labels: { service: ingest }
      - targets: ["127.0.0.1:6109"]
        labels: { service: symdb }
      - targets: ["127.0.0.1:9933"]
        labels: { service: web }

Alternatively, run Prometheus as a container attached to the compose network and target the service names directly (e.g. ingest:8377). This also reaches endpoints that are not published on the host, such as identity:7831.