Mitigating Remote Cache Poisoning in CI/CD

A shared remote build cache is the single highest-leverage optimisation in a monorepo CI pipeline, collapsing multi-hour Bazel or Gradle builds into sub-minute incremental runs. It is also, by default, an unauthenticated write path into every trusted build that consumes it. When fork-based pull request runners and protected branch runners share the same action cache (AC) and content-addressable store (CAS), you have built a supply-chain vector disguised as a performance feature. This is remote cache poisoning: an attacker-controlled or non-hermetic build action produces an artifact that collides with a legitimate action key, and that poisoned artifact is silently served to your main branch’s release build.

This is not a theoretical concern. Bazel, Gradle, Turborepo and Pants all implement remote caching using a two-part lookup: an action key derived from the command line, inputs, and environment, mapped to a digest of the output blob. If the derivation of that action key omits sufficient context — toolchain version, absolute paths, environment variables, or trust tier — two semantically different actions can resolve to the same key. Whoever writes to the cache first, wins. In a distributed CI/CD fleet running hundreds of concurrent PR builds, that race is trivially exploitable.

#Architectural Breakdown: Why Shared Caches Fail Trust Boundaries

The core design flaw is treating the remote cache as a single trust domain when your CI topology has at least two: untrusted (fork PRs, external contributors, unreviewed branches) and trusted (protected branches, release tags, post-merge pipelines). Most default configurations — Bazel’s --remote_cache, Gradle’s build-cache block, Turborepo’s remote cache API — grant read and write access to every runner using the same bearer token or service account, regardless of which side of that boundary the job originated from.

Combine that with non-hermetic actions — genrules that shell out to date, embed absolute workspace paths, or read ambient environment variables not declared in the action’s input set — and you get a build that produces different outputs for the same nominal inputs. If a malicious PR runs a non-hermetic action that happens to key-collide with a legitimate action from main, and the CAS accepts the write, subsequent trusted builds will pull the attacker’s blob out of cache instead of rebuilding from source. No code review catches this because the malicious diff never lands in the trusted branch — it only ever touches the cache.

The correct architectural response is tiered cache segmentation combined with action hermeticity enforcement and cryptographic provenance verification on cache reads. This mirrors zero-trust architectural patterns applied at the network layer, but relocated to the build graph: never trust a cache write without verifying who produced it and under what conditions.

#Trust Tiers and Instance Namespacing

Bazel supports --remote_instance_name, which namespaces the AC and CAS server-side. This is the cheapest lever available: route untrusted PR runners to an isolated instance name with read-only credentials against the trusted instance, and write-only access to their own quarantined namespace. Trusted post-merge builds read exclusively from the trusted instance and populate it themselves — never from PR-tier output.

1# .bazelrc — trusted (protected branch) profile
2build:trusted --remote_cache=grpcs://cache.internal.kby:443
3build:trusted --remote_instance_name=main-trusted
4build:trusted --remote_upload_local_results=true
5build:trusted --experimental_remote_cache_async
6build:trusted --experimental_strict_action_env
7build:trusted --incompatible_strict_action_env
8
9# .bazelrc — untrusted (fork PR) profile
10build:pr --remote_cache=grpcs://cache.internal.kby:443
11build:pr --remote_instance_name=pr-quarantine
12build:pr --remote_upload_local_results=true
13build:pr --noremote_accept_cached
14build:pr --experimental_strict_action_env

Note --noremote_accept_cached on the PR profile. Untrusted runners are permitted to write to their quarantine namespace for their own incremental reuse, but they are never permitted to read cache entries seeded by trusted builds without going through provenance verification — this closes the exfiltration-via-cache-hit side channel where a malicious PR could confirm the contents of a private build output by observing cache hit/miss timing.

#Implementation Logic: Closing the Remote Cache Poisoning Window

Segmentation alone reduces blast radius but does not eliminate remote cache poisoning within a single trust tier — a compromised or misconfigured trusted runner can still poison the trusted namespace. The full mitigation stack requires five sequential controls:

• Hermeticity enforcement — sandbox every action (--sandbox_default_allow_network=false), strip ambient environment variables, and pin toolchains by digest, not by tag.
• Action key salting — inject a workspace status stamp and toolchain digest into every action’s input set so that environment drift cannot collide with a prior key.
• Credential scoping via OIDC — issue short-lived, branch-scoped tokens rather than long-lived static service account keys for cache access.
• Provenance attestation — sign cache writes with in-toto/SLSA metadata and verify signatures on read, not just on ingest.
• Reproducibility spot-checks — periodically rebuild a sample of cached actions from scratch and diff output hashes against the cached digest.

#Step 1 — OIDC-Scoped Cache Credentials in CI

Static bearer tokens for remote cache access are the single largest contributor to remote cache poisoning incidents, because a leaked token from a fork-executed workflow grants indefinite write access to the trusted namespace. Replace them with short-lived OIDC-issued tokens scoped per branch protection state.

1# .github/workflows/ci.yml
2jobs:
3  build:
4    runs-on: ubuntu-22.04
5    permissions:
6      id-token: write
7      contents: read
8    steps:
9      - uses: actions/checkout@v4
10      - name: Mint scoped cache token
11        id: token
12        run: |
13          if [ "${{ github.event.pull_request.head.repo.fork }}" = "true" ]; then
14            echo "instance=pr-quarantine" >> "$GITHUB_OUTPUT"
15            echo "role=cache-reader-pr" >> "$GITHUB_OUTPUT"
16          else
17            echo "instance=main-trusted" >> "$GITHUB_OUTPUT"
18            echo "role=cache-writer-trusted" >> "$GITHUB_OUTPUT"
19          fi
20      - name: Exchange OIDC for cache credential
21        run: |
22          TOKEN=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" 
23            "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=kby-remote-cache" | jq -r .value)
24          kby-cache-auth exchange --oidc-token "$TOKEN" 
25            --role "${{ steps.token.outputs.role }}" 
26            --instance "${{ steps.token.outputs.instance }}" 
27            --ttl 900 > cache-creds.json
28      - name: Bazel build
29        run: bazel --bazelrc=.bazelrc build //... --config=${{ steps.token.outputs.instance == 'main-trusted' && 'trusted' || 'pr' }}

The 900-second TTL is deliberate: it bounds the exploitation window if a token is exfiltrated mid-build to roughly the length of a single CI job, rather than the lifetime of a static secret rotated quarterly.

#Step 2 — Provenance Verification on Cache Read

Namespacing prevents cross-tier writes but does nothing against a compromised trusted runner poisoning its own tier. The remaining control is cryptographic: every cache write is accompanied by an in-toto attestation referencing the Bazel Remote Execution API action digest, signed by the CI runner’s workload identity. Reads verify the signature before the blob is trusted for reuse.

1#!/usr/bin/env bash
2# verify-cache-entry.sh — invoked by the remote cache proxy on read
3ACTION_DIGEST="$1"
4ATTESTATION=$(kby-cache-cli fetch-attestation --digest "$ACTION_DIGEST")
5
6cosign verify-blob 
7  --key /etc/kby/cache-signing.pub 
8  --signature <(echo "$ATTESTATION" | jq -r .signature) 
9  &2
10    exit 1
11  }
12
13echo "ACCEPT: verified provenance for digest ${ACTION_DIGEST}"

This adds roughly 20–40ms of latency per cache hit for the signature check, which is negligible against the multi-second cost of a cache miss rebuild, but it converts an implicit trust decision into an auditable one — every accepted cache entry now has a signed chain of custody back to a specific runner and workflow run ID.

#Failure Modes and Edge Cases

Segmentation and signing close the primary remote cache poisoning vector, but several edge cases remain and must be handled explicitly rather than assumed away:

• Weak hash collisions in legacy CAS backends. Some older remote cache proxies still default to SHA-1 for blob addressing. SHA-1 is not broken for this threat model in the classic collision-attack sense, but it materially lowers the cost of an intentional preimage search compared to SHA-256 or BLAKE3. Any cache backend accepting client-supplied digests without server-side recomputation is vulnerable regardless of hash strength.
• TOCTOU between AC lookup and CAS fetch. A race condition exists where the action cache lookup succeeds against a legitimate digest, but the subsequent CAS blob fetch is served from a different, since-overwritten object if the backend permits digest reuse after garbage collection. Mitigate by configuring the CAS as strictly immutable — write-once, no overwrite — and rejecting any write where the digest already exists but the byte content differs.
• Non-hermetic actions that pass CI but fail locally. Genrules invoking date, reading $HOSTNAME, or resolving DNS during the build will produce different outputs per invocation while presenting an identical declared input set, meaning the action key is stable but the output is not. This is the most common root cause of accidental (non-malicious) remote cache poisoning and should be caught by CI-level reproducibility checks, not code review.
• Federated caches across multiple CI providers. Organisations running GitHub Actions, self-hosted GitLab runners, and Jenkins agents against one shared cache backend multiply the number of distinct trust boundaries that must be independently namespaced and credentialed. A single shared static token across providers reintroduces the original flaw at a larger scale.
• Cache invalidation on toolchain rollback. Rolling back a compiler or SDK version without bumping the toolchain digest used in action key salting will cause the build to silently reuse artifacts compiled under the newer toolchain, producing a binary that does not match the declared build configuration — a correctness failure that masquerades as a successful, fast build.

#Scaling and Security Trade-offs

Every mitigation against remote cache poisoning imposes a measurable cost against raw cache hit-rate and build latency. These trade-offs should be tuned explicitly per environment rather than applied uniformly:

• Single shared cache — highest hit-rate (typically 85–95% on stable monorepos), lowest infrastructure cost, but zero isolation between trust tiers; unsuitable for any repository accepting external contributions.
• Tiered instance namespacing — hit-rate drops for PR builds (cold quarantine namespace, ~40–60% on first run per branch) but trusted-branch hit-rate is unaffected; infrastructure cost roughly doubles due to duplicate storage of common base-layer artifacts across namespaces unless a read-through shared base layer is implemented.
• Provenance signing on every write — adds 20–40ms per action server-side and requires key management infrastructure (HSM or KMS-backed signing keys), but is the only control that defends against a compromised trusted runner rather than just an untrusted one.
• SHA-256 vs BLAKE3 for content addressing — BLAKE3 offers roughly 3–5x the hashing throughput of SHA-256 on modern CPUs, materially reducing CAS ingest latency at scale, but has weaker adoption across existing remote execution proxies, forcing a compatibility shim layer.
• Aggressive TTL and eviction policies — shrinking the poisoning exploitation window by reducing entry lifetime (e.g., 24 hours instead of 30 days) directly reduces cache hit-rate and increases rebuild frequency, trading CI wall-clock time for a smaller attack surface.
• OIDC token exchange overhead — replacing static credentials adds approximately 50–150ms per job for the token exchange round trip, a cost that is trivial against build durations measured in minutes but becomes noticeable in high-frequency, short-lived CI matrix jobs.

None of these controls are optional in isolation for a monorepo accepting external contributions at scale — instance segmentation without provenance verification still permits a compromised trusted runner to poison its own tier indefinitely, and provenance verification without hermeticity enforcement merely signs non-reproducible garbage with a valid signature. The full stack — segmentation, hermeticity, scoped credentials, and signed provenance — is the minimum viable architecture for a remote build cache that cannot be turned into a supply-chain weapon against your own release pipeline.