Immutable CI/CD Backups via S3 Object Lock

Artifact registries are the single point of failure most CI/CD platform teams forget to threat-model. When a Nexus, Artifactory, or Docker Registry instance is compromised, ransomware doesn’t stop at the running service — it walks straight into the mutable backup

This article walks through building that architecture on Amazon S3 using Object Lock, cross-account replication, and IAM permission boundaries, with the artifact registry (Sonatype Nexus, in this case, though the pattern generalises to Artifactory and OCI registries) as the protected workload.

#The Blast Radius Problem: When Backups Share a Trust Domain

The default failure mode in most self-hosted CI/CD estates looks like this: a backup

An immutable backup architecture removes this dependency entirely. The retention guarantee has to hold even when the account that produced the backup is fully compromised, including its root credentials in the worst case. That requirement pushes the design towards WORM (Write Once Read Many) storage semantics enforced by the storage provider, not by application logic.

#Architectural Breakdown: Building an Immutable Backup Architecture

S3 Object Lock is the primitive that makes this practical without operating a dedicated tape library or hardware WORM appliance. It attaches a retention date to individual object versions, and while that retention is active, no principal — including the account owner — can delete or overwrite that specific version, provided the bucket is configured in Compliance mode.

#WORM Semantics and Retention Modes

Object Lock ships with two retention modes, and the distinction is the crux of the entire immutable backup architecture:

• Governance mode — retention can be overridden by principals holding s3:BypassGovernanceRetention. Useful for staging and cost control, but it reintroduces the exact trust dependency you’re trying to eliminate if that permission leaks.
• Compliance mode — retention cannot be shortened or bypassed by any principal, including the root user, until the retention period expires. This is the mode that satisfies a genuine immutable backup architecture requirement.

Compliance mode is irreversible for the duration of the retention window. There is no support ticket, no root override, no emergency exception. That is precisely the property you want for ransomware resilience, and precisely the property that will burn you if you misconfigure retention length on a bucket carrying multi-terabyte nightly dumps (see Failure Modes below).

#Isolating the Blast Radius with Cross-Account Replication

Object Lock alone protects against deletion and overwrite, but it does not protect against an attacker with valid write credentials in the source account simply never triggering a restore, or against an entire account-level compromise (e.g. a leaked AWS Organizations management credential). The architecture therefore needs a second boundary: cross-account replication into a dedicated backup account with no CI/CD workload identities provisioned in it at all.

The resulting flow looks like this: the CI/CD platform account writes backup archives to a local S3 bucket with Object Lock in Governance mode (for operational flexibility during the first replication window), S3 Cross-Region Replication (CRR) copies the object into a backup-only account where the destination bucket enforces Compliance mode, and the backup account’s IAM has no trust relationship back to the CI/CD account beyond the replication role itself, which is scoped to s3:ReplicateObject and nothing else.

#Implementation Logic: Step-by-Step Rollout

The rollout sequence matters — Object Lock can only be enabled on bucket creation (retroactively enabling it on an existing bucket requires opening a support case with AWS and is not guaranteed), so this has to be planned before the first backup job runs, not retrofitted after an incident.

1. Provision a dedicated backup AWS account under your Organization, with SCPs denying any principal from disabling CloudTrail or modifying Object Lock configuration on the destination bucket.
2. Create the destination bucket with Object Lock enabled at creation time and default retention set to Compliance mode.
3. Create the source bucket in the CI/CD account with versioning and Object Lock enabled in Governance mode, giving the platform team an emergency-only bypass path during the initial bake-in period.
4. Configure CRR with a dedicated replication IAM role, restricted to the replication action set only — no read, no delete, no list-bucket beyond what CRR itself requires.
5. Wire the artifact registry’s backup job (a scheduled pipeline stage, not an ad-hoc cron on the registry host) to write via a short-lived STS-assumed role scoped to PutObject on a single prefix.
6. Attach a KMS key with a key policy that denies key deletion for 30 days and restricts kms:Decrypt to the restore role only.
7. Run a scheduled, automated restore drill against the replicated copy — not the source — at least monthly. A backup you have not restored is a hypothesis, not a recovery point.

#Code & Configuration Reference

Terraform for the destination bucket, applied once and never mutated outside a change-controlled pipeline:

1resource "aws_s3_bucket" "artifact_backup_vault" {
2  bucket = "kby-registry-backups-vault"
3  object_lock_enabled = true
4}
5
6resource "aws_s3_bucket_versioning" "artifact_backup_vault" {
7  bucket = aws_s3_bucket.artifact_backup_vault.id
8  versioning_configuration {
9    status = "Enabled"
10  }
11}
12
13resource "aws_s3_bucket_object_lock_configuration" "artifact_backup_vault" {
14  bucket = aws_s3_bucket.artifact_backup_vault.id
15  rule {
16    default_retention {
17      mode = "COMPLIANCE"
18      days = 90
19    }
20  }
21}

The IAM permission boundary applied to every human and machine principal in the CI/CD account, denying the exact escape hatches an attacker would reach for:

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Sid": "DenyGovernanceBypass",
6      "Effect": "Deny",
7      "Action": "s3:BypassGovernanceRetention",
8      "Resource": "arn:aws:s3:::kby-registry-backups-*"
9    },
10    {
11      "Sid": "DenyObjectLockConfigChange",
12      "Effect": "Deny",
13      "Action": [
14        "s3:PutBucketObjectLockConfiguration",
15        "s3:PutObjectRetention",
16        "s3:PutObjectLegalHold"
17      ],
18      "Resource": "arn:aws:s3:::kby-registry-backups-vault/*"
19    }
20  ]
21}

The backup stage itself, wired into a GitLab CI scheduled pipeline rather than a host-level cron so that execution, exit codes, and duration are captured in the same observability plane as every other pipeline job:

1registry-backup:
2  stage: backup
3  image: amazon/aws-cli:2.15.30
4  rules:
5    - if: '$CI_PIPELINE_SOURCE == "schedule"'
6  script:
7    - export CREDS=$(aws sts assume-role --role-arn $BACKUP_ROLE_ARN --role-session-name backup-job --duration-seconds 900)
8    - export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r .Credentials.AccessKeyId)
9    - export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r .Credentials.SecretAccessKey)
10    - export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r .Credentials.SessionToken)
11    - pg_dump nexus_metadata | gzip > nexus-metadata-$(date +%F).sql.gz
12    - sha256sum nexus-metadata-$(date +%F).sql.gz > checksum.sha256
13    - aws s3api put-object --bucket kby-registry-backups-source --key nightly/$(date +%F)/nexus-metadata.sql.gz --body nexus-metadata-$(date +%F).sql.gz --object-lock-mode GOVERNANCE --object-lock-retain-until-date $(date -d "+30 days" --iso-8601=seconds)

Note the explicit --object-lock-mode and --object-lock-retain-until-date flags on the put-object call — Object Lock retention is set per-object-version, not inherited from bucket defaults unless the bucket’s default retention rule is configured, and relying purely on the default without an explicit override in the CLI call is a common source of unprotected “backups” that silently violate the immutable backup architecture requirement.

#Failure Modes & Edge Cases

Compliance mode’s irreversibility cuts both ways. Setting a 7-year retention on a bucket that later needs a schema migration of its metadata format means every historical object sits there, unmodifiable, for the full term — storage cost accrues regardless of relevance. Teams routinely discover this after the fact, once a 3.2 TB nightly dump has been locked at 90-day retention for six months and the storage bill has grown linearly with no way to prune early.

Lifecycle transitions to Glacier Instant Retrieval or Glacier Deep Archive interact badly with Object Lock if the transition rule fires before retention expiry combined with certain third-party backup orchestration tools that assume standard-tier objects are always synchronously readable — restore latency from Deep Archive can exceed 12 hours, which is unacceptable if the restore drill is gating an incident response SLA.

Clock skew on the principal issuing put-object calls matters more than most engineers assume: retention dates are calculated server-side against the S3 service clock, but a misconfigured NTP source on a self-hosted runner can cause retention windows to be requested with unintended offsets, occasionally producing retention periods a full day short or long of the intended value.

Cross-region replication lag is the final trap. CRR is asynchronous — under normal load it completes within minutes, but under sustained high-throughput backup windows (multiple registries backing up concurrently at midnight UTC) replication can queue, meaning your Recovery Point Objective (RPO) in the vault account is not identical to the timestamp on the source object. Monitor ReplicationLatency via CloudWatch metrics rather than assuming near-real-time parity; AWS documents the metric behaviour in detail in the official S3 Object Lock documentation.

#Scaling & Security Trade-offs

Rolling this out across a multi-team platform requires weighing retention rigidity against operational flexibility. Teams evaluating this alongside broader architectural patterns for platform resilience should treat the following as a decision matrix rather than a checklist:

• Governance vs Compliance mode — Governance gives operators a bypass path for legitimate cost or schema-migration needs but reintroduces a privileged-credential attack surface; Compliance removes that risk entirely at the cost of zero flexibility until expiry.
• Same-account vs cross-account replication — same-account Object Lock stops accidental deletion and most malware; cross-account isolation is required to survive a full account-level or root-credential compromise, at the cost of additional IAM and networking overhead.
• Storage class selection — Standard tier gives sub-second restore for drills and real incidents; Glacier Deep Archive cuts storage cost by roughly 80% but pushes RTO into the hours range, which may violate an incident response SLA for a registry backing production deploys.
• Retention window length — short windows (7–14 days) minimise storage cost but risk missing a slow-burn ransomware dwell period that exceeds detection time; 90-day windows are a reasonable default for most registries, extended to a year for compliance-regulated artifact provenance requirements.
• Replication throughput vs backup frequency — hourly incremental backups reduce RPO but multiply CRR queue depth under concurrent load; batching to a single nightly full dump with checksum-verified incrementals in between balances cost against recovery granularity.

None of these trade-offs are free, and the correct point on each axis depends on the blast radius the registry actually represents — a registry serving a single internal team tolerates a looser configuration than one gating production container images for a regulated workload.

Object Lock and cross-account replication are not a substitute for detection and response controls upstream of the registry; they are the last line of defence that guarantees a clean recovery point exists regardless of what happens further up the chain, which is the entire point of treating backup integrity as an architectural concern rather than an operational afterthought.