# Kyverno: admit images only with a fresh, passing VSA Source: https://www.testifysec.com/docs/cilock/guides/kyverno-admit-with-a-fresh-vsa/ A Kyverno ClusterPolicy that admits a pod only when its image digest carries a PASSED CI/lock VSA, signed by your VSA key and verified within a freshness window. Covers the Kyverno version floor, the policy settings, the re-verify job, and what fail-closed looks like in an outage. `cilock verify` writes a signed VSA (verification summary attestation) when evidence passes a policy. `cilock attach` puts the VSA on the image as an OCI 1.1 referrer, one Sigstore bundle v0.3 for each VSA. This guide gives a Kyverno `ClusterPolicy` that admits a pod only when all of these are true: - The image digest carries a VSA that your VSA key signed. Use a key that only the verify job holds, never the key that signs build records. - The VSA says `verificationResult: PASSED`. - The VSA `timeVerified` is inside a freshness window. Kyverno needs nothing else from the build. The pod is admitted by digest, and the VSA names that digest as its subject. The VSA comes from a job that runs `cilock verify` with `--vsa-outfile`, then `cilock attach` on the image digest. [Jenkins: record and verify a Pipeline build](https://www.testifysec.com/docs/cilock/guides/jenkins-record-and-verify-a-pipeline) shows a nightly job that does this. ## What we tested We tested this policy on Kyverno v1.19.2-rc.1 on kind (Kubernetes 1.36), with a registry that serves the OCI referrers API. We also ran the same policy shape on our own EKS cluster. The image names, namespace and secret names on this page are examples. Lines marked **adapted** differ from the tested text. ## Kyverno version: 1.19.2 or later, never 1.19.1 Use Kyverno 1.19.2 or later. Until 1.19.2 is released, use v1.19.2-rc.1. Do not use 1.19.1. It admits a pod that arrives with its own `kyverno.io/verify-images` annotation set to "verified", with no verification (fixed by kyverno pull request 17994). We saw this bypass on 1.19.1 in our test cluster. Anyone who can create a pod with annotations can use it. On v1.19.2-rc.1 the same forged pod was denied. Read the version that runs: ```bash kubectl -n kyverno get deploy kyverno-admission-controller -o jsonpath='{.spec.template.spec.containers[0].image}' ``` The result must be `v1.19.2-rc.1` or a later release, for example `reg.kyverno.io/kyverno/kyverno:v1.19.2-rc.1`. ## The ClusterPolicy ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: webapp-vsa spec: webhookTimeoutSeconds: 30 background: false rules: - name: vsa-passed-and-fresh match: any: - resources: kinds: [Pod] namespaces: [apps] # adapted: your namespace verifyImages: - imageReferences: ["*"] # adapted: we tested a registry pattern; see below type: SigstoreBundle failureAction: Enforce useCache: false mutateDigest: true verifyDigest: true required: true imageRegistryCredentials: secrets: ["registry-pull"] # adapted: a pull secret in the Kyverno namespace attestations: - type: https://slsa.dev/verification_summary/v1 attestors: - entries: - keys: publicKeys: |- rekor: ignoreTlog: true ctlog: ignoreSCT: true conditions: - all: - key: "{{ verificationResult }}" operator: Equals value: PASSED - key: "{{ time_since('', timeVerified, '') }}" operator: LessThan value: 24h ``` Each setting has a reason: | Setting | Why | | --- | --- | | `type: SigstoreBundle` | `cilock attach` writes Sigstore bundles as OCI referrers. The default type, `Cosign`, reads only the `sha256-.att` tag and does not see a referrer. | | `useCache: false` | The default cache keeps a pass for 60 minutes. Then a different image pushed under an old tag can pass. | | `webhookTimeoutSeconds: 30` | The verify reads the registry for each pod. 30 seconds is the value that we tested. | | `match` with `namespaces` | This is how to scope the policy. | | No `spec.webhookConfiguration.matchConditions` | With match conditions, Kyverno does not run `verifyImages`, and it denies every pod as not verified, also a pod with a valid VSA (kyverno issue 18042, open). | | `imageReferences: ["*"]` | `required: true` applies only to images that match `imageReferences`. With a registry pattern, a pod that names an image from another registry is not verified. With `*`, every image in the namespace needs a VSA, sidecars and init containers too. | | `mutateDigest: true` | A pod that names a tag gets the verified digest written into its spec. The VSA names that digest. | | `rekor` and `ctlog` ignored | The VSA is signed with a static key and carries an RFC 3161 timestamp. It has no transparency log entry. | Apply the policy, then wait for it. Right after a Kyverno install or restart, Kyverno's own policy webhook can time out for a short time. This loop tries again: ```bash for i in $(seq 1 30); do kubectl apply -f webapp-vsa.yaml >/dev/null 2>apply-policy.err && break [ "$i" = 30 ] && { cat apply-policy.err >&2; exit 1; }; sleep 5; done kubectl wait --for=condition=Ready cpol/webapp-vsa --timeout 120s >/dev/null ``` `kubectl get cpol webapp-vsa` shows `READY True`. ## Deploy, and what Kyverno denies Deploy the promoted image by its tag: ```bash kubectl -n apps run webapp --image=registry.example.com/apps/stable/webapp:3 kubectl -n apps get pod webapp -o jsonpath='{.spec.containers[0].image}' ``` The pod starts, and its image is `registry.example.com/apps/stable/webapp:3@sha256:...`, the digest that Kyverno verified. These are the results in our test: | Image | Result | | --- | --- | | Promoted image, fresh PASSED VSA | Admitted | | A drill image that the verify refused (no VSA) | Denied: `no matching signatures found` | | A VSA signed by a different key | Denied: `no matching signatures found` | | A FAILED VSA | Denied | | A PASSED and a FAILED VSA on one digest | Denied: each VSA that the key verifies must pass | | The good image, VSA older than the window | Denied: `attestation checks failed ... predicate https://slsa.dev/verification_summary/v1` | | The same image after a new verify with `attach --replace` | Admitted, 1 referrer | | A pod with a forged `kyverno.io/verify-images` annotation | Denied | | A tag pushed again to a different image | Denied (cache off) | A denial looks like this: ```text Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: resource Pod/apps/drill-sunspot was blocked due to the following policies webapp-vsa: vsa-passed-and-fresh: 'image attestations verification failed, verifiedCount: 0, requiredCount: 1, error: sigstore bundle verification failed: no matching signatures found' ``` ## The freshness window and the re-verify job The window is the `LessThan` value on `timeVerified`. Pick it with care: - The important freshness check is in the job that verifies just before it promotes. That job refuses a bad build. - The window at admission applies to every pod start, also a restart. With a short window, Kyverno denies each pod that starts again after the window, for example after a node drain or an autoscale event. This causes an outage. - Keep the admission window long (24 hours), or add a re-verify job on a schedule shorter than the window. The re-verify job runs `cilock verify` again on the records of the digest that serves, then attaches the new VSA with `--replace`: ```bash set -euo pipefail # adapted: a failed verify stops the script before the attach rm -f evidence/vsa.json # adapted: never attach a VSA from an earlier run "$CILOCK" verify --offline -p "$POLICY" -k "$POLICY_PUB" \ -a evidence/checkout.json -a evidence/build.json -a evidence/push.json \ --subjects "$DIGEST" $SIGN $VSA_TSA --vsa-outfile evidence/vsa.json "$CILOCK" attach "$STABLE_IMAGE@$DIGEST" --envelope evidence/vsa.json --replace ``` `cilock verify` exits non-zero when the evidence fails the policy. `set -e` then stops the script, so the attach does not run. `--replace` deletes only the VSAs whose signed `timeVerified` is older than the new one. Kyverno requires each attached VSA to pass, so an old VSA next to a new one gets the pod denied. Kyverno also fails an image with more than 50 referrers. With `--replace` the image keeps one VSA. On our own cluster we used a 30-minute window and ran the re-verify every 10 minutes. That gives three chances in each window, so one late or failed run does not deny pods. Make the job fail closed: if it cannot read the serving digest, finds more than one digest, cannot read the records, or the verify does not give PASSED for exactly that digest, it attaches nothing. ## The platform verdict endpoint On the TestifySec Platform, Kyverno can also ask the platform for a verdict on each image digest, and use the attached VSA when the platform cannot answer. The [Kyverno admission integration](https://pushgate.dev/integrations/kyverno-admission) generates that policy. We measured its failure behavior on Kyverno v1.19.2-rc.1: | Verdict endpoint | Image with a PASSED VSA | Image with no VSA | | --- | --- | --- | | DNS failure, connection refused, HTTP 503, 401, 403 or 429 | Admitted (the VSA rule decides) | Denied | | 200 with verdict PASSED | Admitted | Denied | | 200 with verdict FAILED or REFUSED | Denied | Denied | Any non-2xx answer counts as "unreachable", and the attached VSA decides. A revoked admission credential (401, 403) therefore does not show as a credential error. The cluster falls back to the VSA alone. ## What fail-closed looks like in an outage The policy does not set `failurePolicy`, so the webhook fails closed: the denial above came from `mutate.kyverno.svc-fail`. Plan for these results: - **Kyverno is down** (not measured). The Kubernetes API server cannot call the webhook, so it refuses to create pods in each namespace that Kyverno's webhook selects. Pods that already run are not affected. Admission happens only at create time. - **The registry is down, or the pull secret is wrong** (not measured). Kyverno cannot read the referrers, so it denies the pod. - **The VSA is too old.** Each new or restarted pod is denied until a re-verify attaches a fresh VSA. The outage reaches more than the namespaces in the policy. Kyverno cannot narrow the webhook of a `verifyImages` policy (kyverno issue 18042). Its failure policy applies to each namespace that the Kyverno chart's webhook `namespaceSelector` selects. On a Kyverno that serves only this policy, set that selector to the governed namespaces. On a shared Kyverno that change turns off the other Kyverno policies outside those namespaces. There, accept the outage risk or choose fail-open. Run the admission controller with 2 replicas and a PodDisruptionBudget. To remove Kyverno from admission in an emergency, uninstall it and delete its webhooks. The controller creates the webhooks, not the chart, so they stay after a crashed controller: ```bash helm uninstall kyverno -n kyverno --wait && \ kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \ -l webhook.kyverno.io/managed-by=kyverno --ignore-not-found kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration | grep -i kyverno # nothing ``` ## Troubleshooting | You see | Cause | Do this | | --- | --- | --- | | Every pod is denied as not verified, also with a valid VSA | The policy has `spec.webhookConfiguration.matchConditions` (kyverno issue 18042). | Remove `matchConditions`. Scope with `match` and `namespaces`. | | `no matching signatures found` | The image has no VSA, or the VSA was signed by a different key. Kyverno gives the same message for both. | Download the VSA with `cilock download`. No VSA means the verify or the attach did not run for this digest. A VSA that is there means a key mismatch. | | `attestation checks failed ... predicate https://slsa.dev/verification_summary/v1` | The VSA is FAILED, or older than the window. | Run the re-verify job. If the VSA is FAILED, the build did not pass. Do not deploy it. | | A pod with a valid new VSA is denied | An older VSA is still attached. Each VSA must pass. | Attach with `cilock attach --replace`. | | The VSA is on one repository path and the pod uses another | The registry keeps attestations per repository path. | Attach to the image path that the pod uses, after the promotion. | | `type: SigstoreBundle` finds nothing on a registry with no referrers API | The registry cannot serve referrers. | See [`cilock attach`](https://www.testifysec.com/docs/cilock/reference/cli#cilock-attach-reposha256hex) for the `cosign-att` route and `type: Cosign`. | | A forged pod is admitted | Kyverno 1.19.1 or earlier. | Upgrade to 1.19.2 or later. | To read the VSA on an image: ```bash mkdir -p vsa-dl && cilock download "$STABLE_IMAGE@$DIGEST" --predicate-type https://slsa.dev/verification_summary/v1 --outdir vsa-dl ``` [`cilock kyverno explain`](https://www.testifysec.com/docs/cilock/reference/cli#cilock-kyverno-explain), not in a release yet, says which of these causes applies to an image. ## Next - [Jenkins: record and verify a Pipeline build](https://www.testifysec.com/docs/cilock/guides/jenkins-record-and-verify-a-pipeline) to produce the VSA. - [Verify in a release gate](https://www.testifysec.com/docs/cilock/guides/verify-in-a-release-gate) for other places to put the verify. Reference generated from the product documentation. Match commands and support details to your installed release.