Kubernetes autoscaling on queue depth with KEDA

CPU is a poor proxy for whether a worker pool is keeping up: workers can be blocked on I/O while a queue grows, or use plenty of CPU while the queue is already draining. Kubernetes autoscaling on queue depth lets you scale on the work waiting to be done, but you need limits and cooldowns to stop a sudden backlog from turning into a costly replica spike.

#How it works

KEDA reads a queue metric and exposes it to Kubernetes’ Horizontal Pod Autoscaler (HPA). The HPA adjusts the worker Deployment toward the configured target, while KEDA handles activation and, when configured, scaling the Deployment down to zero.

flowchart LR
  A["Producers"] --> B["RabbitMQ queue"]
  B --> C["KEDA RabbitMQ scaler"]
  C --> D["KEDA metrics adapter"]
  D --> E["Kubernetes HPA"]
  E --> F["Worker Deployment"]
  F --> B

The key difference from CPU scaling is that the signal represents pending work, not resource use. A worker blocked on a database call can be nearly idle on CPU and still be part of a pool that’s falling behind. Queue depth makes that backlog visible to the autoscaler.

For a RabbitMQ queue, KEDA can estimate how many workers are needed from a target queue length per replica. Say the target is 20 ready messages per replica. If the queue has 200 messages, the HPA will aim for roughly 10 replicas, subject to the minimum and maximum you set. That’s an estimate, not a drain-time guarantee: message processing time, worker concurrency, and downstream limits still matter.

Queue depth also has limits as a signal. A growing queue might mean more incoming work, slower workers, or a blocked dependency. If message age matters more than message count, monitor the age of the oldest message too. For Kafka, use consumer-group lag instead of treating the topic’s total message count as pending work.

#Setting it up

This example uses RabbitMQ and pins the KEDA Helm chart to version 2.16.1. Check the KEDA and Kubernetes compatibility matrix for your cluster before installing; don’t let a floating chart version change your autoscaling behavior during a routine deployment.

#1. Install KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

helm upgrade --install keda kedacore/keda \
  --namespace keda \
  --create-namespace \
  --version 2.16.1

Keep KEDA in a platform-managed namespace so application teams can own their ScaledObject resources without needing to manage the autoscaler itself.

#2. Deploy a worker pool

The worker needs access to RabbitMQ, and it should handle termination without abandoning work. Make sure it stops taking new messages and acknowledges or safely releases any message it already owns before exiting. The exact shutdown behavior depends on the worker library.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: email-worker
spec:
  replicas: 0
  selector:
    matchLabels:
      app: email-worker
  template:
    metadata:
      labels:
        app: email-worker
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: worker
          image: example.invalid/email-worker:replace-me
          env:
            - name: RABBITMQ_URL
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-connection
                  key: url
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi

Replace the example image with your worker image, preferably referenced by an immutable digest. Set requests and limits based on the worker’s real memory and CPU needs; KEDA decides how many pods to run, but Kubernetes still needs resource requests to schedule them sensibly.

Store the broker URL in a Secret managed by your usual secrets workflow. Avoid putting credentials directly in a ScaledObject or committing them in a manifest.

#3. Configure the queue scaler

The ScaledObject connects the worker Deployment to RabbitMQ. Its value is the target number of ready messages per replica. activationValue controls when KEDA considers the queue active; with this setting, a non-empty queue can wake a Deployment from zero.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: rabbitmq-auth
spec:
  secretTargetRef:
    - parameter: host
      name: rabbitmq-connection
      key: url
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: email-worker
spec:
  scaleTargetRef:
    name: email-worker
  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 0
  maxReplicaCount: 40
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Pods
              value: 4
              periodSeconds: 60
  triggers:
    - type: rabbitmq
      authenticationRef:
        name: rabbitmq-auth
      metadata:
        protocol: amqp
        queueName: email-jobs
        mode: QueueLength
        value: "20"
        activationValue: "1"

The example permits scale-up to 40 replicas, but limits scale-down to four pods per minute and adds a five-minute stabilization window. Those settings are deliberately cautious starting points, not universal production values. Tune them against your queue’s arrival rate and the time a worker needs to become useful.

Before applying this to production, verify that the scaler can authenticate and read the intended queue. Also confirm the queue name and RabbitMQ virtual host: a scaler looking at the wrong queue may produce valid-looking metrics that don’t describe the work your workers consume.

#Running it in production

#Watch whether the queue is draining

Queue depth is the scaling input, but it shouldn’t be the only operational signal. Track ready messages and unacknowledged messages separately if your broker exposes them, along with the age of the oldest message and worker completion rate. A queue that stays flat may be healthy if arrivals and completions are balanced; a queue that grows while workers are at their maximum is a capacity or processing problem.

For the autoscaling path, watch the current and desired replica counts, HPA events, KEDA scaler errors, and pod startup time. If KEDA can’t read the queue, the metric can’t guide scale changes. Alert on that failure rather than assuming the worker count is still responding to demand.

#Choose the target from worker behavior

The example’s target of 20 messages per replica is an illustrative assumption. A useful target depends on how quickly each worker can process messages and how much backlog your service can tolerate. If one message takes a long time to process, a target that looks small may still imply a long wait.

A simple way to reason about a starting value is to measure queue arrival and completion rates over normal and busy periods, then check whether the worker pool catches up at the target. Don’t tune only for a brief peak: workers take time to start, load dependencies, and begin consuming. If the backlog must clear within a particular window, calculate that requirement using observed processing rates and validate it with a load test.

#Keep scaling from amplifying an outage

A queue can grow because a downstream database or API is unhealthy. Adding workers in that situation may increase retries and connections without increasing useful throughput. Use bounded retries, a dead-letter queue, and sensible connection limits; set maxReplicaCount low enough that workers can’t overwhelm dependencies during a backlog.

Scale-to-zero saves resources when the queue is idle, but it adds startup delay to the first job after a quiet period. If your service has a strict latency requirement, set minReplicaCount above zero instead. The right choice is a trade-off between idle capacity and first-message latency, not a rule that every queue should scale to zero.

Cost control comes from a combination of a replica ceiling, realistic resource requests, and a scaling policy that doesn’t hold a large pool after a burst has passed. Estimate the maximum worker footprint before enabling autoscaling: multiply the per-pod requests by maxReplicaCount, then compare that with namespace quotas and available cluster capacity. This is a capacity check, not a cloud price estimate; actual cost depends on your cluster and billing model.

#Common pitfalls

• Scaling on queue depth without checking message age. A small number of slow jobs can be more urgent than a large number of quick ones. Monitor both backlog size and age.
• Setting the maximum too high. The HPA may respond correctly to a backlog and still cause a downstream incident. Choose the cap based on dependency capacity.
• Confusing queue length with throughput. A replica target is not a promise that work will finish within a specific time. Measure processing rates and test the drain behavior.
• Ignoring shutdown and in-flight work. Pods that exit without releasing or safely completing messages can cause duplicate processing or delayed retries.
• Treating a scaler error as a queue problem. Authentication mistakes, network restrictions, or broker issues can break metric collection. Alert on scaler health and test recovery.
• Removing all warm capacity without checking latency. Scale-to-zero has a cold-start cost. Keep a minimum replica if the first-job delay is unacceptable.

#What I’d do

I’d start with a conservative maximum, a queue target based on measured worker throughput, and gradual scale-down. Then I’d test three cases before relying on it: a normal burst, a downstream dependency slowdown, and a period where KEDA can’t reach the broker. The test should confirm that the queue drains when capacity is available and that the worker pool stays bounded when it isn’t.

This setup suits teams with asynchronous jobs and a queue metric that reflects actual pending work. The next step is to measure message age and worker completion rate alongside queue depth, then use those measurements to choose the target and replica cap.

#Key takeaways