Flaky Test Detection via Statistical Failure Analysis

A single non-deterministic test in a monorepo with 40,000 test cases doesn’t just cost you one re-run. It costs you every engineer downstream who bisects a red build, every autoscaled CI worker that spins up to re-execute a suite that was never going to pass reliably, and eventually, the credibility of your entire pipeline. Once engineers stop trusting a red build to mean “something is broken,” they start merging on retry, and your flaky test quarantine problem stops being a testing issue and becomes a organisational risk. This article covers the architecture for detecting, scoring, and automatically quarantining non-deterministic tests at scale, without relying on manual triage tickets that nobody closes.

#The Bottleneck: Signal Decay in CI Trust

Most organisations handle flaky tests reactively. An engineer sees a failure, re-runs the job, it goes green, and the incident is forgotten until the next unlucky commit gets blamed for a regression it didn’t cause. At small scale this is an annoyance. At the scale of a multi-thousand-test monorepo running across sharded CI workers, it becomes a statistical certainty on every single build.

The core issue is that test failure signal and build health signal are conflated into a single binary: pass or fail. When a test has an intrinsic failure probability of even 0.5%, and your pipeline runs 20,000 tests per commit, you will see failures on effectively every build purely from noise, independent of code quality. Without a dedicated flaky test quarantine mechanism, engineers lose the ability to distinguish “this commit broke something” from “this test has bad state management.”

#Architectural Breakdown: Decoupling Quarantine from Execution

The correct architectural pattern separates three concerns that most teams bundle into the CI job itself: test execution, signal aggregation, and quarantine decisioning. Bundling these means every CI run has to make a real-time judgement call about flakiness with no historical context, which is why ad-hoc retry logic (retry: 3 in your YAML) is a band-aid, not a solution. It masks flakiness rather than isolating it, and it silently doubles or triples your compute cost on the worst-performing tests.

Instead, treat test results as a stream of structured telemetry, similar to how you’d treat application logs or traces. This is one of the architectural patterns borrowed directly from observability engineering: emit events, aggregate centrally, and make decisions asynchronously from a system of record rather than inline in the hot path.

#The Test Signal Pipeline

Every test execution, regardless of pass/fail, emits a structured event containing the test ID, commit SHA, shard ID, duration, exit status, and stack trace hash if failed. These events are streamed (Kafka, Kinesis, or even a simple append-only object store if you’re running lower volume) into a columnar store optimised for high-cardinality time-series queries. ClickHouse is the pragmatic choice here given the write throughput of large test suites.

1CREATE TABLE test_results (
2    test_id String,
3    suite String,
4    commit_sha String,
5    branch String,
6    shard_id UInt16,
7    status Enum8('pass' = 1, 'fail' = 2, 'error' = 3),
8    duration_ms UInt32,
9    stack_hash String,
10    executed_at DateTime
11) ENGINE = MergeTree()
12PARTITION BY toYYYYMM(executed_at)
13ORDER BY (test_id, executed_at);

The flaky test quarantine scoring job runs against this table on a rolling window, typically the trailing 200 executions per test, rather than a fixed calendar window. Fixed-time windows under-sample low-frequency tests and over-sample high-frequency ones, which skews the flakiness score. A rolling execution-count window normalises for test frequency across the suite.

#Flaky Test Quarantine State Machine

Quarantine decisions should never be a boolean toggle applied by a human clicking a button. Model it as a finite state machine with four states: HEALTHY, SUSPECT, QUARANTINED, and ESCALATED. A test transitions from HEALTHY to SUSPECT once its rolling flip-rate (pass-to-fail transitions independent of code diffs) exceeds a threshold, typically 3% over the last 200 runs. It transitions from SUSPECT to QUARANTINED once it fails without an associated source diff touching that test’s dependency graph, which rules out legitimate regressions. ESCALATED is reserved for tests that remain flaky for more than 14 days post-quarantine, which routes to a human owner rather than staying invisible forever.

1{
2  "test_id": "com.kby.billing.InvoiceReconciliationTest#testPartialRefund",
3  "state": "QUARANTINED",
4  "flip_rate_200": 0.061,
5  "first_flagged": "2024-02-11T09:32:00Z",
6  "quarantined_by": "auto-scoring-job-v3",
7  "owner_team": "payments-platform",
8  "expires_at": "2024-02-25T09:32:00Z",
9  "reason_code": "HIGH_FLIP_RATE_NO_DIFF_CORRELATION"
10}

#Implementation Logic

The end-to-end pipeline breaks down into five discrete steps:

• Instrumentation: wrap the test runner (JUnit, pytest, Bazel’s --test_output) so every execution emits a structured event regardless of outcome, including retries as separate events rather than overwriting the original result.
• Streaming ingestion: push events to Kafka with the commit SHA as the partition key, ensuring ordering guarantees per-commit for downstream correlation with diff metadata.
• Scoring: a scheduled job (every 15 minutes, not on every commit, to avoid recomputation storms) recalculates flip-rate and correlation-with-diff metrics per test and writes state transitions to the quarantine manifest.
• Manifest distribution: the quarantine manifest is a version-controlled artefact, not a live database query on the CI hot path, because a database outage should never block the entire build fleet from executing tests.
• Enforcement: the CI orchestrator reads the manifest at job start, excludes quarantined tests from blocking the build, but still executes them in a non-blocking “shadow” lane so the signal pipeline keeps receiving data for automatic un-quarantine.

That shadow-lane execution is the detail most implementations get wrong. If you stop running quarantined tests entirely, you lose the ability to detect when they’ve stabilised, and the quarantine becomes permanent by default. The flaky test quarantine list should be self-healing wherever possible, not a one-way ratchet.

#CI Orchestrator Enforcement

Enforcement logic in the pipeline definition should treat the manifest as a gate, not a filter baked into individual test files. Here’s a representative GitHub Actions step that consumes the manifest before invoking the test runner:

1jobs:
2  test:
3    runs-on: self-hosted
4    steps:
5      - uses: actions/checkout@v4
6      - name: Fetch quarantine manifest
7        run: |
8          curl -sf https://artifacts.kby.internal/quarantine/manifest.json 
9            -o quarantine.json
10      - name: Run test suite with quarantine gating
11        run: |
12          python ci/run_tests.py 
13            --quarantine-file quarantine.json 
14            --shadow-lane 
15            --fail-on-non-quarantined-only

The --fail-on-non-quarantined-only flag is the critical enforcement primitive: it changes the CI exit code logic so that failures in quarantined tests are logged and forwarded to the scoring pipeline but do not affect the build’s pass/fail status. Non-quarantined failures still block merges as normal.

#Scoring Algorithm Core

The scoring job itself is deliberately simple statistics rather than machine learning, because interpretability matters when a payments team is asking why their test got quarantined at 3am.

1def compute_flip_rate(executions):
2    """executions: list of ('pass'|'fail') ordered by time, last 200 runs"""
3    flips = sum(
4        1 for i in range(1, len(executions))
5        if executions[i] != executions[i - 1]
6    )
7    return flips / max(len(executions) - 1, 1)
8
9def correlates_with_diff(test_id, failing_commit, diff_paths):
10    dependency_graph = load_dependency_graph(test_id)
11    return any(path in dependency_graph for path in diff_paths)
12
13def should_quarantine(test_id, executions, failing_commit, diff_paths):
14    flip_rate = compute_flip_rate(executions)
15    if flip_rate < 0.03:
16        return False
17    return not correlates_with_diff(test_id, failing_commit, diff_paths)

Note the deliberate exclusion of any test whose failure correlates with an actual code change in its dependency graph, cross-referenced against the commit’s diff manifest. This is the mechanism that prevents genuine regressions from being silently swallowed by an overly aggressive quarantine policy, which is the single biggest risk of automating this process.

#Failure Modes and Edge Cases

The failure modes here are subtle because the system is designed to hide signal, and hidden signal is dangerous when the hiding logic itself has bugs.

Mass quarantine events. A shared test fixture, database migration, or CI runner image change can spike flip-rate across dozens of unrelated tests simultaneously. If the scoring job has no rate limiter, you can quarantine 200 tests in a single 15-minute cycle, effectively disabling large swathes of your regression coverage. Cap the number of quarantine transitions per scoring cycle (e.g. 10) and alert loudly when the cap is hit, because that’s almost always an infrastructure regression, not test flakiness.

Diff-correlation false negatives. The dependency graph used to correlate failures against diffs is only as accurate as your build system’s introspection. Dynamically loaded modules, reflection-based dependency injection, and feature-flag-gated code paths routinely evade static dependency graphs, meaning a genuine regression can get misclassified as flakiness and quarantined right when it matters most. Mitigate this by weighting recent quarantine candidates against a secondary signal: did the failure first appear on the same commit that introduced the flip-rate spike? If yes, treat it as SUSPECT rather than an immediate QUARANTINED transition, and require two independent scoring cycles to confirm.

Manifest staleness under network partition. If the artefact store hosting the quarantine manifest is unreachable, the fallback behaviour matters enormously. Failing closed (treat every test as non-quarantined) restores full blocking behaviour and is safe but noisy. Failing open (treat the last known manifest as authoritative indefinitely) risks running stale quarantine data for weeks if nobody notices the outage. The correct default is failing closed with a TTL-based staleness alert, never silent fail-open.

Owner attribution drift. Quarantined tests need an owning team baked into the manifest at quarantine time, derived from CODEOWNERS or a service catalogue, not assigned retroactively. Without this, ESCALATED tests accumulate in a queue nobody is accountable for, and your flaky test quarantine system becomes exactly the graveyard of ignored tickets it was built to prevent.

#Scaling and Security Trade-offs

Beyond a few thousand tests, the architecture needs to make explicit trade-offs between decisioning speed, blast radius, and auditability.

• Central DB gating vs Git-backed manifest: querying ClickHouse directly from every CI job gives real-time accuracy but couples build execution to database availability; a versioned, cached manifest artefact trades a few minutes of staleness for resilience against downstream outages.
• Scoring cadence vs compute cost: scoring on every commit gives the tightest feedback loop but multiplies aggregation query cost linearly with commit volume; a fixed 15-minute cadence decouples scoring cost from commit throughput at the expense of detection latency.
• RBAC on manifest writes: the quarantine manifest is a high-privilege artefact because it silently changes which failures block a merge. Writes must be restricted to the automated scoring service identity, with manual overrides requiring a signed commit and a two-person approval, otherwise it becomes a trivial vector for bypassing test coverage on a malicious or careless change.
• Audit trail granularity: every state transition needs an immutable log entry (test ID, previous state, new state, triggering metric, commit SHA) retained for at least one full release cycle; without this, post-incident reviews cannot distinguish “the test was legitimately flaky” from “the quarantine system suppressed a real regression.”
• Storage growth: at 20,000 tests per commit with 500 commits/day, the raw event table grows by roughly 10 million rows daily; partition by month and enforce a TTL (90 days is usually sufficient) on raw execution events while retaining aggregated flip-rate summaries indefinitely for trend analysis.

None of this replaces fixing the underlying non-determinism, whether that’s unbounded async waits, shared mutable test fixtures, or clock-dependent assertions. What automated flaky test quarantine buys you is a bounded, auditable holding pattern that keeps your CI signal trustworthy while the actual root causes get triaged on a schedule that doesn’t block every engineer’s merge queue. For further reading on the statistical basis of flip-rate scoring at scale, Google’s engineering team has published detailed findings on flaky test detection and remediation practices at testing.googleblog.com.