Skip to main content

Configure mTLS (Mutual TLS)

Mutual TLS (mTLS) provides bidirectional authentication where both the client and server authenticate each other using certificates. This configuration sets up the zymtrace gateway as a TLS server that requires client certificate authentication, providing an additional layer of security for your deployment.

The zymtrace gateway uses Envoy Proxy as the underlying proxy server. When mTLS is enabled, Envoy terminates the TLS connection and handles the client certificate validation, ensuring secure communication between clients and the backend services.

Overview​

When mTLS is enabled:

  • The Envoy gateway requires clients to present valid certificates signed by your Certificate Authority (CA)
  • Envoy terminates the TLS connection and validates client certificates
  • The gateway presents its own certificate to clients for server authentication
  • All communication is encrypted and mutually authenticated
  • Only clients with valid certificates can access the gateway
  • Backend services receive the validated requests from Envoy over internal HTTP

Prerequisites​

Before configuring mTLS, ensure you have:

  • Certificate Authority (CA): A CA certificate and private key for signing client and server certificates
  • Server Certificate: A certificate for the gateway service signed by your CA
  • Server Private Key: The private key corresponding to the server certificate
  • Client Certificates: Certificates for clients that need to connect
  • Ingress Controller with SSL Passthrough: NGINX, Traefik, or compatible ingress controller

Certificate Requirements​

All certificates must be in PEM format. You'll need:

  1. CA Certificate (ca.crt): Used to validate client certificates
  2. Server Certificate (server.crt): Gateway's identity certificate
  3. Server Private Key (server.key): Private key for the server certificate
Certificate Format

Provide certificates in plain text PEM format. The zymtrace helm automatically converts PEM certificates to base64 encoding when creating Kubernetes secrets. Do not provide pre-encoded base64 certificates as this will cause double encoding issues.

Configuration​

1. Enable mTLS in Helm Values​

Create or update your custom-values.yaml file:

custom-values.yaml
services:
  gateway:
    mtls:
      enabled: true
      cert: |
        -----BEGIN CERTIFICATE-----
        # Your server certificate content here
        # Copy the entire content of server.crt
        -----END CERTIFICATE-----
      key: |
        -----BEGIN PRIVATE KEY-----
        # Your server private key content here
        # Copy the entire content of server.key
        -----END PRIVATE KEY-----
      ca: |
        -----BEGIN CERTIFICATE-----
        # Your CA certificate content here
        # Copy the entire content of ca.crt
        -----END CERTIFICATE-----

ingress:
  enabled: true
  className: "nginx"
  
  hosts:
    gateway:
      # Regular gateway ingress (non-mTLS)
      enabled: true
      host: "zymtrace.company.com"
      paths:
        - path: /
          pathType: Prefix
      
      # mTLS gateway ingress
      mtls:
        enabled: true
        host: "mtls.zymtrace.company.com"  # Different hostname for mTLS
        paths:
          - path: /
            pathType: Prefix
        annotations:
          nginx.ingress.kubernetes.io/ssl-passthrough: "true"
          nginx.ingress.kubernetes.io/proxy-body-size: "0"
          nginx.ingress.kubernetes.io/client-body-buffer-size: "1M"

  tls:
    - secretName: zymtrace-gateway-tls
      hosts:
        - "zymtrace.company.com"

2. Using --set-file for Certificate Files​

Alternatively, you can load certificates from files using --set-file:

helm upgrade --install backend zymtrace/backend \
  --namespace zymtrace \
  --set services.gateway.mtls.enabled=true \
  --set-file services.gateway.mtls.cert=server.crt \
  --set-file services.gateway.mtls.key=server.key \
  --set-file services.gateway.mtls.ca=ca.crt \
  --set ingress.enabled=true \
  --set ingress.hosts.gateway.mtls.enabled=true \
  --set ingress.hosts.gateway.mtls.host="mtls.zymtrace.company.com"

3. Deploy with mTLS Configuration​

helm upgrade --install backend zymtrace/backend \
  --namespace zymtrace \
  --create-namespace \
  -f custom-values.yaml

Ingress Controller Configuration​

Different ingress controllers require specific annotations for mTLS support:

NGINX Ingress Controller​

annotations:
  nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  nginx.ingress.kubernetes.io/proxy-body-size: "0"
  nginx.ingress.kubernetes.io/client-body-buffer-size: "1M"

Important: SSL passthrough must be enabled on your NGINX ingress controller for mTLS to work properly.

Verification​

1. Check Ingress Configuration​

Verify the mTLS ingress is properly configured:

kubectl get ingress -n zymtrace

Expected output showing both regular and mTLS ingresses:

NAME                            CLASS   HOSTS                      ADDRESS         PORTS     AGE
zymtrace-gateway-ingress        nginx  zymtrace.company.com        35.192.XXX.XX   80, 443   24h
zymtrace-gateway-ingress-mtls   nginx  mtls.zymtrace.company.com   35.192.XXX.XX   80        24h

2. Test mTLS Connection​

Test the mTLS endpoint with a valid client certificate:

# Test with valid client certificate - should succeed
curl --insecure 'https://mtls.zymtrace.company.com/project/00000000-0000-0000-0000-000000000000/dashboard?start_ts=-1h&end_ts=now&query_mode=basic' \
  --cacert certs/ca/ca.crt \
  --cert   certs/client/client.crt \
  --key    certs/client/client.key \
  --user zsystem:zsystem123

You can also describe the mTLS ingress for detailed configuration:

kubectl describe ingress zymtrace-gateway-ingress-mtls -n zymtrace

Troubleshooting​

Common Issues​

  1. SSL Passthrough Not Enabled

    # Enable SSL passthrough in NGINX ingress controller
    kubectl patch configmap nginx-configuration -n ingress-nginx \
      --patch '{"data":{"enable-ssl-passthrough":"true"}}'

  2. Certificate Format Issues

    • Ensure certificates are in PEM format
    • Check for correct BEGIN/END markers
    • Verify certificate chain order

  3. Client Certificate Verification Failures

    # Verify client certificate against CA
    openssl verify -CAfile ca.crt client.crt

  4. DNS Resolution

    • Ensure mTLS hostname resolves correctly
    • Update DNS records for the mTLS endpoint

Debug Commands​

# Check gateway pod logs
kubectl logs -n zymtrace -l app=zymtrace-gateway

# Verify certificate secrets
kubectl get secrets -n zymtrace | grep tls

# Test certificate chain
openssl s_client -connect mtls.zymtrace.company.com:443 \
  -cert client.crt -key client.key -CAfile ca.crt