Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler (HPA) automatically scales zymtrace services based on resource utilization metrics. This ensures optimal performance during varying workloads by:

Scaling up during high traffic periods to maintain performance
Scaling down during low usage to conserve resources
Balancing resource utilization for cost efficiency

The zymtrace backend chart supports autoscaling for all microservices including ingest, web, symdb, ui, and identity services.

Requirements

Before configuring HPA, ensure you have:

A Kubernetes cluster with Metrics Server installed and operational
- HPA requires Metrics Server to collect resource utilization data
- Verify with: kubectl get apiservice v1beta1.metrics.k8s.io
- If not installed, install with:
```
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```
Kubernetes version 1.23+ (for autoscaling/v2 API)
Resource requests/limits defined for your deployments (already included in default configuration)

1. Enabling HPA

HPA can be configured at two levels:

Global level - Apply the same settings to all services
Service-specific level - Override global settings for individual services

Update your custom-values.yaml to enable HPA:

custom-values.yaml

services:
  common:
    hpa:
      enabled: true  # Enable HPA globally for all services
      minReplicas: 1  # Minimum number of replicas
      maxReplicas: 10  # Maximum number of replicas
      targetCPUUtilizationPercentage: 80  # Scale when CPU reaches 80%
      targetMemoryUtilizationPercentage: 80  # Scale when memory reaches 80%
      # Optional advanced scaling behavior
      # behavior:
      #   scaleDown:
      #     stabilizationWindowSeconds: 300  # Wait 5 minutes before scaling down
      #     policies:
      #     - type: Percent
      #       value: 100
      #       periodSeconds: 15

Service-Specific Configuration

To customize HPA for individual services, add settings under the specific service section:

custom-values.yaml with service-specific settings

services:
  ingest:
    # Other ingest settings...
    hpa:
      enabled: true
      minReplicas: 1  # Start with 2 ingest pods minimum
      maxReplicas: 10  # Allow up to 15 replicas for high traffic
      targetCPUUtilizationPercentage: 70  # Scale earlier at 70% CPU
      targetMemoryUtilizationPercentage: 80
  
  web:
    # Other web settings...
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 5  # Fewer replicas needed for web service
      targetCPUUtilizationPercentage: 80
      
  # Other services can be similarly configured

Explanation of Configuration Parameters

enabled: Enables or disables HPA for services
minReplicas: Minimum number of pods, even during low usage periods
maxReplicas: Maximum number of pods allowed during peak demand
targetCPUUtilizationPercentage: CPU utilization threshold that triggers scaling
targetMemoryUtilizationPercentage: Memory utilization threshold that triggers scaling
behavior: Optional advanced scaling policies (stabilization windows, scale rates, etc.)

2. Deploying with HPA Enabled

After configuring your custom-values.yaml, deploy or upgrade the zymtrace backend:

helm upgrade --install backend zymtrace/backend -f custom-values.yaml

Alternatively, use --set in the command line:

 helm upgrade backend zymtrace/backend \
   --set services.common.hpa.enabled=true \
   --set services.common.hpa.targetCPUUtilizationPercentage=80 \
   --set services.common.hpa.targetMemoryUtilizationPercentage=80

Verify that HPAs have been created:

kubectl get hpa -n zymtrace

Expected output:

NAME               REFERENCE                      TARGETS                       MINPODS   MAXPODS   REPLICAS   AGE
zymtrace-ingest    Deployment/zymtrace-ingest     45%/80%, 30%/80%             2         15        3          10m
zymtrace-web       Deployment/zymtrace-web        15%/80%, 20%/80%             1         5         1          10m
zymtrace-symdb     Deployment/zymtrace-symdb      10%/80%, 15%/80%             1         10        1          10m
zymtrace-ui        Deployment/zymtrace-ui         5%/80%, 10%/80%              1         10        1          10m
zymtrace-identity  Deployment/zymtrace-identity   5%/80%, 10%/80%              1         10        1          10m

3. Monitoring HPA Behavior

Monitor how your HPAs are performing with:

kubectl describe hpa zymtrace-ingest -n zymtrace

This will show recent scaling events, current metrics, and other useful information:

Name:                     zymtrace-ingest
Namespace:                zymtrace
Reference:                Deployment/zymtrace-ingest
Metrics:                  ( current / target )
  resource cpu:           45% / 80%
  resource memory:        30% / 80%
Min replicas:             2
Max replicas:             15
Deployment pods:          3 current / 3 desired
...
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 3; reason: CPU utilization above target

4. Advanced Configuration

Fine-Tuning Scaling Behavior

For workloads with specific scaling needs, you can configure advanced behavior policies:

Advanced HPA behavior configuration

services:
  ingest:
    hpa:
      enabled: true
      minReplicas: 2
      maxReplicas: 15
      targetCPUUtilizationPercentage: 70
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0  # Scale up immediately
          policies:
          - type: Percent
            value: 100  # Double the pod count
            periodSeconds: 60  # Every minute if needed
          selectPolicy: Max  # Use the policy with highest change
        scaleDown:
          stabilizationWindowSeconds: 300  # Wait 5 minutes before scaling down
          policies:
          - type: Percent
            value: 20  # Only reduce by 20% at a time
            periodSeconds: 60  # Every minute
          selectPolicy: Min  # Use the policy with smallest change

Custom Metrics (Advanced)

While the default CPU and memory metrics are sufficient for most scenarios, you can also configure HPAs based on custom metrics if you have a metrics provider like Prometheus installed. This requires additional configuration and is beyond the scope of this document.

5. Troubleshooting

Common Issues

HPAs not being created:

kubectl logs -n zymtrace -l app.kubernetes.io/name=helm

HPAs created but not scaling:

kubectl get events -n zymtrace | grep HorizontalPodAutoscaler

Metrics not available:

kubectl describe apiservice v1beta1.metrics.k8s.io

Disabling HPA

If you encounter issues with HPA, you can disable it:

services:
  common:
    hpa:
      enabled: false

Then upgrade your deployment:

helm upgrade --install backend zymtrace/backend -f custom-values.yaml

References

For further information, refer to:

Requirements​

1. Enabling HPA​

Service-Specific Configuration​

Explanation of Configuration Parameters​

2. Deploying with HPA Enabled​

3. Monitoring HPA Behavior​

4. Advanced Configuration​

Fine-Tuning Scaling Behavior​

Custom Metrics (Advanced)​

5. Troubleshooting​

Common Issues​

Disabling HPA​

References​