Fix the worker before scaling the fleet

Queue backlogMeasure actual waiting work

Worker lifecycleBound and release child processes

KEDA scalingScale the corrected workload

KEDA was active in four environments. The 50-replica production ceiling is configuration, not demonstrated throughput capacity.

Contribution at a glance

My responsibility

Demand-signal analysis, KEDA configuration and worker-lifecycle diagnosis and repair.

Scope

A Pub/Sub-driven worker deployed across four environments.

Collaboration & boundaries

I worked across application process behavior and infrastructure scaling; the rollout exposed a worker defect that needed repair first.

Outcome

KEDA was active in four environments after containing stuck child processes. A 50-replica limit is configuration, not demonstrated capacity.

The misleading signal

The service captured livestreams using browser child processes. A pod could hold all its concurrency slots while using little CPU. Once full, it stopped taking messages, so additional demand accumulated in Pub/Sub rather than making the CPU-based autoscaler react.

My contribution

I evaluated the workload's actual demand signal, implemented a KEDA Pub/Sub trigger and investigated failures uncovered during rollout. That investigation changed the task: an active production incident needed a worker-lifecycle fix before scaling could help.

The application defect behind the backlog

Streams with no participant left child processes waiting indefinitely for a video element. Each child held a browser and a concurrency slot until pods ran out of memory. More replicas would only distribute those stuck processes across more machines.

I verified the behavior inside the pod's own browser and corrected the parent-side health-check lifecycle. An initial browser-side timeout hypothesis was rejected by the production evidence: the parent-side path was the one actually firing.

Choose a signal the workload really produces

I used backlog on the subscription consumed by this application. A sibling service's active-stream count represented a different capture flow, and its scaling metric could not safely be reused. A Prometheus trigger was also unsuitable because the required scrape infrastructure was absent from the development and staging clusters.

The previous HPA was removed as KEDA took ownership. Development, staging and sandbox used workload identity for the scaler; production used a separately verified credential path.

Results and measurement

The closeout recorded KEDA Ready in four environments and the related GitOps applications Synced/Healthy. The post-rollout snapshot reported zero restarts, compared with repeated OOM kills during the incident. This is a rollout observation, not a long-term availability claim.

The production maximum was configured to 50 replicas. That is a guardrail to monitor, not proof of sustained capacity at that size.

Limits and operational choices

Production promotion was an explicit incident exception before the usual QA validation. Follow-up remained around participant-less streams, the origin of their traffic spike and capacity pressure at higher replica counts. The change contained the resource-lifecycle failure; it did not establish an end-to-end capture success rate.

The lesson

Autoscaling controls how many workers exist. It cannot make each worker release resources correctly. Diagnose the saturation mechanism before turning up the fleet.

Decisions and next steps

A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.

Alternative and rationale

CPU and memory triggers miss a worker whose slots are occupied by stuck browser processes. Backlog better represents pending work.

Trade-off

Scaling a leaking worker multiplies the defect. Demand-based scaling still needs bounded child-process lifecycles and concurrency.

What I would improve next

Next improvement: load-test throughput and recovery with realistic stream failures before treating replica limits as capacity guarantees.