failurePolicy: Ignore Silently Skips Admission Checks
A validating webhook configured with failurePolicy: Ignore stops enforcing policy the moment its backend pod is unreachable, and the API server admits objects as if no controls exist. Kubernetes logs nothing that looks like an incident, so privileged workloads land in etcd during routine restarts, upgrades, or certificate rotations.
At a glance
- Unsafe setting
- ValidatingWebhookConfiguration webhooks[].failurePolicy set to Ignore on a security policy enforcement webhook.
- Failure trigger
- The webhook backend becomes unreachable due to a rolling restart, certificate expiry, scale-to-zero, or network disruption between the API server and the webhook service.
- Blast radius
- Objects that violate security policy are admitted and persisted to etcd during the outage window with no audit signal beyond a log line, and are never re-validated afterwards.
- Recommended control
- Set failurePolicy to Fail on security-critical webhooks, exclude kube-system via namespaceSelector, run multiple replicas behind a PodDisruptionBudget, and back it with a detective audit controller.
The Trap
failurePolicy: Ignore on a ValidatingWebhookConfiguration that enforces security policy, typically an OPA Gatekeeper or Kyverno deployment standing in for the retired PodSecurityPolicy admission chain.
The Default State
Helm charts for Gatekeeper and Kyverno, and most kubebuilder-scaffolded webhook operators, ship with failurePolicy set to Ignore out of the box. The rationale is bootstrap safety: if the webhook pod hasn’t started yet, cluster operators don’t want the API server refusing every kubectl apply. Teams install the chart, confirm policies block a test pod, and never revisit the field once the demo passes.
The Blast Radius
Ignore means any failure to reach the webhook, whether from a rolling update, a node drain, HPA scaling the deployment to zero replicas, a TLS certificate expiring on the webhook’s serving cert, or a CNI hiccup between the API server and the webhook’s ClusterIP, results in silent admission. The API server treats a connection timeout, a 5xx, or a refused connection identically: it lets the request through as though no ValidatingWebhookConfiguration existed. There is no admission-denied event, no annotation on the created object, nothing but a line in the audit log that most teams don’t query in real time. A privileged pod with hostPath /var/run/docker.sock, a container running as UID 0, or a Deployment missing resource limits can be created during a two-minute webhook restart window and will persist indefinitely once etcd has it, because nothing re-validates existing objects retroactively. On multi-tenant clusters this converts a transient outage into a permanent policy bypass for whatever landed during the gap.
The Lead Mechanic Fix
Set failurePolicy to Fail on every webhook that enforces a security boundary, then make the webhook itself resilient enough to justify that setting: kubectl patch validatingwebhookconfiguration gatekeeper-validating-webhook-configuration –type=’json’ -p='[{“op”:”replace”,”path”:”/webhooks/0/failurePolicy”,”value”:”Fail”}]’. Pair it with timeoutSeconds: 5, a namespaceSelector excluding kube-system so control-plane bootstrap can’t deadlock, at least two webhook replicas behind a PodDisruptionBudget with minAvailable: 1, and a readinessProbe that only reports ready once the policy cache is loaded, not just when the process starts. Run Gatekeeper’s audit controller in parallel as a compensating detective control, since it catches anything that slips through during a legitimate Fail-mode outage.