Building a Multi-Window SLO Burn-Rate Calculator

A static threshold alert — fire when error rate exceeds 1% — is functionally useless for any service with a meaningful SLO. It cannot distinguish between a brief blip that consumes 0.3% of a monthly error budget and a sustained regression that will exhaust the entire budget in four hours. The fix, formalised in Google’s SRE workbook, is a burn-rate calculator that converts raw error ratios into a normalised “budget consumption speed” and alerts on that speed across multiple time windows simultaneously. This article covers the architecture and implementation of a self-hosted burn-rate calculator suitable for wiring directly into Prometheus, Alertmanager, and a lightweight interactive front-end for on-call engineers.

#The Problem With Single-Window Error Rate Alerting

Consider a service with a 99.9% availability SLO over a rolling 30-day window. That gives an error budget of 43.2 minutes of full downtime equivalent per month, or 0.1% of total request volume. A naive alert rule such as error_rate > 0.001 has two failure modes baked in from day one:

• It fires on transient noise — a single bad deploy for 90 seconds trips the same alert as a six-hour regional outage.
• It says nothing about how fast the budget is depleting, so on-call engineers cannot triage severity without manually cross-referencing dashboards.

The multi-window multi-burn-rate model solves this by defining burn rate as a ratio: how many multiples of the “sustainable” error rate the service is currently consuming. A burn rate of 1 means the service will exhaust its entire 30-day budget in exactly 30 days. A burn rate of 14.4 against a short window means the budget disappears in roughly 2 hours if sustained — that is the threshold Google’s reference implementation uses for page-worthy fast burn.

#Architectural Breakdown of the Burn-Rate Calculator

The calculator itself is a pure function: given an SLO target, a lookback window, and observed good/total event counts, it returns a dimensionless burn-rate value. The architectural complexity lives around that function, not inside it. A production-grade burn-rate calculator needs three cooperating layers:

• Metric ingestion layer — Prometheus recording rules that pre-aggregate good_events and total_events at 1m resolution to avoid recomputing raw counters on every evaluation cycle.
• Calculation layer — the burn-rate formula itself, evaluated across at least two window pairs (short/long) to satisfy the multi-window requirement and suppress single-window flapping.
• Presentation layer — an interactive front-end (the actual “IT Toolkit” utility) that lets an engineer paste in an SLO target and current error counts to simulate remaining budget before deciding whether to roll back a deploy.

The core formula is straightforward:

1burn_rate = (1 - (good_events / total_events)) / (1 - slo_target)

If your SLO target is 0.999 and observed success ratio over the window is 0.995, burn rate is (1 – 0.995) / (1 – 0.999) = 0.005 / 0.001 = 5.0. At that rate the monthly budget is consumed in 30/5 = 6 days — well outside a page-worthy threshold but firmly inside a ticket-worthy one.

#Why Multi-Window Is Non-Negotiable

A single long window (say 6 hours) smooths out noise but delays detection — by the time it trips, budget is already gone. A single short window (5 minutes) detects fast but is dominated by sampling noise on low-traffic services. The standard mitigation is to require both a short window and a long window to independently exceed the burn-rate threshold before paging. This is the same coincidence-detection pattern used in architectural patterns for anomaly suppression elsewhere in observability pipelines — you are trading a small amount of detection latency for a large reduction in false positive rate.

#Implementation Logic

Building the calculator end-to-end follows this sequence:

• Define the SLI numerator/denominator query (e.g. http_requests_total{code!~"5.."} over http_requests_total).
• Precompute the ratio at multiple rolling windows via Prometheus recording rules — do not compute burn rate directly in the alert expression; pre-aggregation avoids repeated high-cardinality scans.
• Define burn-rate thresholds per severity tier, each requiring a short-window/long-window pair.
• Expose the raw ratios via an API endpoint so the interactive calculator front-end can run “what-if” simulations without querying Prometheus directly from the browser.
• Wire Alertmanager routing so fast-burn alerts page immediately and slow-burn alerts route to a ticket queue with a 24-hour SLA.

#Recording Rules

1groups:
2  - name: slo_burn_rate_recording
3    interval: 30s
4    rules:
5      - record: sli:requests:ratio_rate5m
6        expr: |
7          sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[5m]))
8          /
9          sum(rate(http_requests_total{job="checkout-api"}[5m]))
10      - record: sli:requests:ratio_rate1h
11        expr: |
12          sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[1h]))
13          /
14          sum(rate(http_requests_total{job="checkout-api"}[1h]))
15      - record: sli:requests:ratio_rate6h
16        expr: |
17          sum(rate(http_requests_total{job="checkout-api",code!~"5.."}[6h]))
18          /
19          sum(rate(http_requests_total{job="checkout-api"}[6h]))

#The Alerting Rule (Multi-Window, Multi-Burn-Rate)

1- alert: FastBurnSLOCheckoutAPI
2  expr: |
3    (1 - sli:requests:ratio_rate5m) > (14.4 * 0.001)
4    and
5    (1 - sli:requests:ratio_rate1h) > (14.4 * 0.001)
6  for: 2m
7  labels:
8    severity: page
9  annotations:
10    summary: "Checkout API burning error budget at 14.4x — budget exhausted in ~2h if sustained."
11
12- alert: SlowBurnSLOCheckoutAPI
13  expr: |
14    (1 - sli:requests:ratio_rate1h) > (3 * 0.001)
15    and
16    (1 - sli:requests:ratio_rate6h) > (3 * 0.001)
17  for: 15m
18  labels:
19    severity: ticket
20  annotations:
21    summary: "Checkout API burning error budget at 3x — budget exhausted in ~10 days if sustained."

The 0.001 constant here is 1 - slo_target for a 99.9% SLO. Hardcoding it per rule is brittle at scale; a mature implementation templates these rules via Jsonnet or a Kubernetes operator that reads SLO targets from a CRD and generates the recording/alerting rules automatically.

#The Interactive Calculator Function

The front-end “toolkit” component — the part engineers actually interact with during an incident — wraps the same formula in a client-side function so it can run offline simulations against hypothetical numbers before committing to a rollback decision:

1function calculateBurnRate(sloTarget, goodEvents, totalEvents) {
2  if (totalEvents === 0) return 0;
3  const errorBudget = 1 - sloTarget;
4  const observedErrorRatio = 1 - (goodEvents / totalEvents);
5  const burnRate = observedErrorRatio / errorBudget;
6  const daysToExhaust = burnRate > 0 ? (30 / burnRate) : Infinity;
7  return { burnRate: Number(burnRate.toFixed(2)), daysToExhaust: Number(daysToExhaust.toFixed(2)) };
8}
9
10// Example: 99.95% SLO, observed 9950/10000 successful requests
11calculateBurnRate(0.9995, 9950, 10000);
12// => { burnRate: 100, daysToExhaust: 0.3 }

That last example is deliberately alarming — a burn rate of 100 against a 99.95% SLO means the entire monthly budget is gone in roughly 7 hours. This is exactly the scenario the burn-rate calculator is designed to surface immediately, rather than leaving an engineer to mentally divide percentages under incident pressure.

#Failure Modes and Edge Cases

Multi-window burn-rate systems have their own failure surface, distinct from the naive threshold problems they solve.

• Low-traffic denominators — services below roughly 1 request/second produce wildly unstable ratios in 5-minute windows. A single failed request can spike the short window to a burn rate of 200+ purely from sample-size noise. Mitigate by enforcing a minimum event-count floor before evaluating the short window, falling back to the longer window’s verdict when volume is insufficient.
• Clock skew between recording rule evaluation and scrape intervals — if the Prometheus scrape interval and rule evaluation interval are misaligned, short windows can double-count or skip intervals, producing burn-rate values that oscillate independently of actual error rate.
• Budget reset ambiguity — calendar-month resets versus rolling 30-day windows produce different burn-rate curves near month boundaries. Rolling windows are strictly more accurate but require longer metric retention (30+ days at the recording-rule resolution), which has direct storage cost implications on the TSDB.
• Composite SLOs — when an SLO is derived from multiple upstream dependencies (e.g. checkout depends on payments and inventory), a naive burn-rate calculator will double-penalise the composite service for a burn event that is already being paged on independently in an upstream service’s own SLO.
• Alert flapping at threshold boundaries — a burn rate oscillating around exactly 14.4 will repeatedly fire and resolve. Adding a small hysteresis band (e.g. requiring burn rate to drop below 12 before resolving, not just below 14.4) prevents alert-storm behaviour during marginal incidents.

#Scaling and Security Trade-offs

Deploying a burn-rate calculator fleet-wide — across dozens or hundreds of services, each with distinct SLO targets — introduces trade-offs that do not appear in a single-service proof of concept.

• Client-side vs server-side computation: exposing raw good/total event counters to a browser-based calculator is fine for internal tooling but leaks request-volume and error-rate telemetry that may be commercially sensitive if the tool is ever exposed externally. Server-side computation behind an authenticated API is the safer default.
• Recording rule cardinality: templating burn-rate recording rules per service, per window, per severity tier multiplies quickly. At 50 services x 3 windows x 2 severity tiers, that is 300 recording rules minimum — manageable, but it warrants a dedicated Prometheus rule-file generator rather than hand-maintained YAML.
• Alertmanager routing fan-out: fast-burn alerts should page directly; slow-burn alerts should never share the same routing tree without severity-based inhibition rules, or a slow burn will eventually escalate into paging noise once it crosses into fast-burn territory — inhibit the slow-burn alert once the fast-burn alert for the same service is active.
• Data retention cost: rolling 30-day burn-rate accuracy requires 30 days of 1-minute-resolution recording rule data. On high-cardinality services this materially increases Prometheus/Thanos/Mimir storage costs; downsampling after 7 days is a common compromise, accepting slightly coarser long-window burn-rate precision beyond that point.
• Multi-tenant SLO ownership: if the calculator API is exposed to product teams for self-service SLO definition, enforce a schema validator that rejects impossible configurations (e.g. an SLO target of 100.0%, which produces a division-by-zero in the burn-rate formula) before they reach the recording rule generator.

The reference model for all of the thresholds above — 14.4x for fast burn, 3x for slow burn, and the specific window pairings — is documented in Google’s own SRE workbook chapter on alerting, and is worth treating as the canonical starting point rather than re-deriving thresholds from first principles.

Once the recording rules, alerting thresholds, and interactive front-end are wired together, the burn-rate calculator stops being a dashboard curiosity and becomes the primary triage instrument during an incident — it tells the on-call engineer not just that something is wrong, but precisely how much runway remains before the SLO is breached.