Autoscaling Self-Hosted CI Runners with KEDA

A fixed pool of ten self-hosted CI runners sits idle for eighteen hours a day, then chokes under a queue of ninety concurrent jobs during a release freeze. This is the default failure mode of static runner provisioning, and it is the single biggest source of wasted compute spend in mid-to-large engineering organisations running their own GitHub Actions or GitLab CI infrastructure. Provision for peak and you burn budget on idle EC2 instances or bare-metal nodes. Provision for average and every release day becomes a queueing disaster with jobs stacking for forty minutes before a runner picks them up. Neither static sizing model survives contact with real-world commit patterns, which are bursty, unpredictable, and heavily skewed towards business hours and release windows.

The fix is event-driven autoscaling for self-hosted CI runners, decoupled entirely from CPU or memory metrics and instead driven by the actual backlog of queued jobs. This article covers the architecture, implementation, and failure surface of using KEDA (Kubernetes Event-Driven Autoscaling) to scale self-hosted CI runners against a live job queue, using GitHub Actions as the reference platform, though the same pattern applies almost unchanged to GitLab CI and Buildkite.

#The Problem: Static Runner Pools Bleed Money and Time

Traditional self-hosted CI runner deployments fall into one of two categories: a fixed-size VM pool managed by Terraform, or a Kubernetes Deployment with a static replica count. Both suffer from the same architectural flaw — they scale on a schedule or a manual trigger, not on actual demand. The consequence is a bimodal cost curve: overspend on idle capacity outside peak hours, and underspend (in the form of engineer wait time) during bursts.

Horizontal Pod Autoscaler (HPA) is the obvious first attempt at a fix, but HPA scales on resource utilisation — CPU and memory — which is the wrong signal entirely for CI runners. A runner pod sits at near-zero CPU while polling for a job, then spikes during execution. By the time HPA reacts to the CPU spike, the job has often already finished. You need a queue-depth signal, not a resource-utilisation signal, and that is precisely the gap KEDA closes.

#Architectural Breakdown: Event-Driven Scaling for Self-Hosted CI Runners

#Why Kubernetes HPA Falls Short for Self-Hosted CI Runners

HPA’s control loop polls the metrics-server every fifteen seconds by default and reacts to a rolling average. Self-hosted CI runners, by contrast, generate a step-function demand pattern — zero to ninety concurrent jobs in the time it takes a monorepo’s CI graph to fan out. Metric lag under HPA means jobs queue for minutes while the autoscaler catches up to a signal that was never the right one in the first place.

#KEDA as the Control Plane

KEDA introduces a ScaledObject custom resource that translates an external event source — in this case, a GitHub Actions job queue exposed via webhook or the github-runner scaler — directly into replica count. Rather than measuring resource utilisation after the fact, KEDA measures the thing that actually matters: how many jobs are currently queued and unassigned. This is the correct architectural pattern for self-hosted CI runners because it treats the runner fleet as a consumer group against a work queue, structurally identical to autoscaling a Kafka consumer group against partition lag.

The full pipeline looks like this: GitHub fires a workflow_job webhook event on queue → an ingestion service (or the KEDA github-runner scaler polling the Actions API) records queued job count → KEDA’s metrics adapter exposes this as a custom metric → the ScaledObject’s HPA shim scales the runner Deployment or the actions-runner-controller RunnerSet accordingly. This decouples the scaling decision entirely from node-level telemetry, which is what makes it viable for self-hosted CI runners with highly variable job durations.

#Implementation Logic

The rollout sequence for a production-grade KEDA-driven runner fleet follows five stages:

• Deploy actions-runner-controller (ARC) into a dedicated arc-systems namespace, using webhook-based ephemeral runners rather than long-lived registered runners — ephemeral runners de-register automatically after one job, which avoids stale registration drift.
• Install KEDA via Helm into the cluster’s core namespace, ensuring the metrics-server API aggregation layer is already functioning correctly, since KEDA registers itself as an external metrics provider.
• Configure a GitHub App with actions:read and administration:write permissions, scoped to the specific organisation or repository set that ARC will service. Store the private key as a Kubernetes Secret, never as a plaintext ConfigMap.
• Define the ScaledObject targeting the RunnerDeployment, with minReplicaCount: 0 to enable scale-to-zero during idle periods, and a maxReplicaCount capped against your node pool’s autoscaling ceiling to prevent runaway billing.
• Attach a cluster-autoscaler or Karpenter provisioner to the underlying node group so pod-level scaling triggers node-level scaling in turn — KEDA scaling pods is meaningless if there is no compute capacity to schedule them onto.

#Code & Configurations

The ScaledObject below binds a RunnerDeployment to the GitHub Actions job queue using the native KEDA scaler, polling every fifteen seconds and scaling between zero and forty ephemeral runners:

1apiVersion: keda.sh/v1alpha1
2kind: ScaledObject
3metadata:
4 name: gha-runner-scaler
5 namespace: arc-runners
6spec:
7 scaleTargetRef:
8 apiVersion: actions.summerwind.dev/v1alpha1
9 kind: RunnerDeployment
10 name: default-runner-deployment
11 pollingInterval: 15
12 cooldownPeriod: 120
13 minReplicaCount: 0
14 maxReplicaCount: 40
15 triggers:
16 - type: github-runner
17 metadata:
18 githubApiURL: https://api.github.com
19 owner: kby-technologies
20 runnerScope: org
21 targetWorkflowQueueLength: "1"
22 authenticationRef:
23 name: gha-runner-auth

Authentication is handled via a TriggerAuthentication resource referencing the GitHub App credentials, kept out of the ScaledObject manifest entirely:

1apiVersion: keda.sh/v1alpha1
2kind: TriggerAuthentication
3metadata:
4 name: gha-runner-auth
5 namespace: arc-runners
6spec:
7 secretTargetRef:
8 - parameter: appID
9 name: gha-app-credentials
10 key: appId
11 - parameter: installationID
12 name: gha-app-credentials
13 key: installationId
14 - parameter: privateKey
15 name: gha-app-credentials
16 key: privateKey

Finally, the underlying RunnerDeployment should enforce ephemeral runner behaviour and pin resource requests tightly to avoid bin-packing fragmentation on the node pool:

1apiVersion: actions.summerwind.dev/v1alpha1
2kind: RunnerDeployment
3metadata:
4 name: default-runner-deployment
5 namespace: arc-runners
6spec:
7 replicas: 0
8 template:
9 spec:
10 ephemeral: true
11 organization: kby-technologies
12 labels:
13 - self-hosted
14 - linux
15 - x64
16 resources:
17 requests:
18 cpu: "1500m"
19 memory: "3Gi"
20 limits:
21 cpu: "2000m"
22 memory: "4Gi"
23 dockerEnabled: true

This is the exact pattern KBY Technologies applies when advising clients on architectural patterns for internal CI platforms — treat the runner fleet as a stateless, disposable consumer group, never as a pet infrastructure component.

#Failure Modes & Edge Cases

Self-hosted CI runners under event-driven autoscaling introduce a distinct failure surface compared to static pools, and each of the following needs explicit mitigation:

#Scale-to-zero cold start latency

When minReplicaCount is set to zero, the first job after an idle period pays the full cost of pod scheduling, image pull, and runner registration — typically 25 to 60 seconds depending on image size. For latency-sensitive pipelines (e.g. PR checks blocking merge), set minReplicaCount: 1 during business hours via a secondary ScaledObject with a cron trigger, and drop to zero only overnight.

#Registration token race conditions

GitHub Actions registration tokens are single-use and expire within one hour. If ARC’s controller experiences a reconciliation delay while KEDA is simultaneously scaling replicas up, you can end up with orphaned pods holding expired tokens, stuck in CrashLoopBackOff. Mitigate by setting a tight cooldownPeriod (120 seconds is a reasonable default) and monitoring the runner_registration_failures_total metric if you export ARC’s controller metrics to Prometheus.

#Webhook delivery gaps

If you drive scaling via GitHub webhooks rather than API polling, a missed webhook delivery — which does happen during GitHub API incidents — leaves KEDA blind to queued jobs. Always run the polling-based github-runner scaler as a fallback signal even if webhooks are your primary trigger, since polling degrades gracefully while webhooks fail silently.

#Node pool bin-packing fragmentation

Ephemeral self-hosted CI runners with tightly pinned resource requests can still fragment a node pool if job resource requirements vary wildly — a 1-CPU lint job and an 8-CPU integration test job scheduled onto the same node class produce poor bin-packing efficiency. Segment runner pools by workload class (e.g. runner-light vs runner-heavy labels) and attach separate ScaledObjects and node taints per class.

#Scaling & Security Trade-offs

Moving self-hosted CI runners to an event-driven model changes both the cost profile and the attack surface. The trade-offs worth quantifying before rollout:

• Cost efficiency vs cold-start latency — scale-to-zero can cut idle compute spend by 60-80% in typical business-hours-only usage patterns, at the cost of a 25-60 second cold start on the first job after idle.
• Ephemeral vs persistent runners — ephemeral self-hosted CI runners eliminate cross-job state leakage and reduce the blast radius of a compromised job, but increase registration API call volume against GitHub’s rate limits (5,000 requests/hour per App installation by default).
• Secret exposure window — persistent runners hold long-lived registration tokens and are a higher-value target for lateral movement; ephemeral runners rotate credentials per job, shrinking the exploitable window to a single job’s duration.
• Blast radius of a compromised runner image — a poisoned base image affects every job scheduled during its validity window; enforce image provenance via cosign signature verification in the RunnerDeployment’s init container to bound this risk.
• Autoscaler-to-autoscaler latency stacking — KEDA scaling pods and cluster-autoscaler or Karpenter scaling nodes are two independent control loops; under aggressive bursts, total time-to-capacity is the sum of both loops’ reaction times, often 90-150 seconds combined, which must be factored into SLA commitments for CI turnaround time.
• Multi-tenancy isolation — running self-hosted CI runners for multiple business units on a shared cluster requires network policies restricting pod-to-pod traffic and separate ScaledObjects per tenant to prevent one team’s burst from starving another’s runner budget.

Event-driven scaling turns self-hosted CI runners from a fixed capital cost into a variable, demand-proportional one, but it shifts operational complexity from capacity planning into control-loop tuning and credential lifecycle management. Get the cooldown periods and node provisioner sizing right, and the runner fleet becomes invisible infrastructure rather than a recurring capacity-planning meeting.