Kyverno: admit images only with a fresh, passing VSA

cilock verify writes a signed VSA (verification summary attestation) when evidence passes a policy. cilock attach puts the VSA on the image as an OCI 1.1 referrer, one Sigstore bundle v0.3 for each VSA. This guide gives a Kyverno ClusterPolicy that admits a pod only when all of these are true:

• The image digest carries a VSA that your VSA key signed. Use a key that only the verify job holds, never the key that signs build records.
• The VSA says verificationResult: PASSED.
• The VSA timeVerified is inside a freshness window.

Kyverno needs nothing else from the build. The pod is admitted by digest, and the VSA names that digest as its subject.

The VSA comes from a job that runs cilock verify with --vsa-outfile, then cilock attach on the image digest. Jenkins: record and verify a Pipeline build shows a nightly job that does this.

What we tested

We tested this policy on Kyverno v1.19.2-rc.1 on kind (Kubernetes 1.36), with a registry that serves the OCI referrers API. We also ran the same policy shape on our own EKS cluster. The image names, namespace and secret names on this page are examples. Lines marked adapted differ from the tested text.

Kyverno version: 1.19.2 or later, never 1.19.1

Use Kyverno 1.19.2 or later. Until 1.19.2 is released, use v1.19.2-rc.1.

Do not use 1.19.1. It admits a pod that arrives with its own kyverno.io/verify-images annotation set to "verified", with no verification (fixed by kyverno pull request 17994). We saw this bypass on 1.19.1 in our test cluster. Anyone who can create a pod with annotations can use it. On v1.19.2-rc.1 the same forged pod was denied.

Read the version that runs:

kubectl -n kyverno get deploy kyverno-admission-controller -o jsonpath='{.spec.template.spec.containers[0].image}'

The result must be v1.19.2-rc.1 or a later release, for example reg.kyverno.io/kyverno/kyverno:v1.19.2-rc.1.

The ClusterPolicy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: webapp-vsa
spec:
  webhookTimeoutSeconds: 30
  background: false
  rules:
    - name: vsa-passed-and-fresh
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [apps]                           # adapted: your namespace
      verifyImages:
        - imageReferences: ["*"]                           # adapted: we tested a registry pattern; see below
          type: SigstoreBundle
          failureAction: Enforce
          useCache: false
          mutateDigest: true
          verifyDigest: true
          required: true
          imageRegistryCredentials:
            secrets: ["registry-pull"]                     # adapted: a pull secret in the Kyverno namespace
          attestations:
            - type: https://slsa.dev/verification_summary/v1
              attestors:
                - entries:
                    - keys:
                        publicKeys: |-
                          <contents of keys/vsa.pub>
                        rekor:
                          ignoreTlog: true
                        ctlog:
                          ignoreSCT: true
              conditions:
                - all:
                    - key: "{{ verificationResult }}"
                      operator: Equals
                      value: PASSED
                    - key: "{{ time_since('', timeVerified, '') }}"
                      operator: LessThan
                      value: 24h

Each setting has a reason:

Apply the policy, then wait for it. Right after a Kyverno install or restart, Kyverno's own policy webhook can time out for a short time. This loop tries again:

for i in $(seq 1 30); do kubectl apply -f webapp-vsa.yaml >/dev/null 2>apply-policy.err && break
  [ "$i" = 30 ] && { cat apply-policy.err >&2; exit 1; }; sleep 5; done
kubectl wait --for=condition=Ready cpol/webapp-vsa --timeout 120s >/dev/null

kubectl get cpol webapp-vsa shows READY True.

Deploy, and what Kyverno denies

Deploy the promoted image by its tag:

kubectl -n apps run webapp --image=registry.example.com/apps/stable/webapp:3
kubectl -n apps get pod webapp -o jsonpath='{.spec.containers[0].image}'

The pod starts, and its image is registry.example.com/apps/stable/webapp:3@sha256:..., the digest that Kyverno verified.

These are the results in our test:

A denial looks like this:

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Pod/apps/drill-sunspot was blocked due to the following policies

webapp-vsa:
 vsa-passed-and-fresh: 'image attestations verification failed, verifiedCount: 0, requiredCount: 1, error: sigstore bundle verification failed: no matching signatures found'

The freshness window and the re-verify job

The window is the LessThan value on timeVerified. Pick it with care:

• The important freshness check is in the job that verifies just before it promotes. That job refuses a bad build.
• The window at admission applies to every pod start, also a restart. With a short window, Kyverno denies each pod that starts again after the window, for example after a node drain or an autoscale event. This causes an outage.
• Keep the admission window long (24 hours), or add a re-verify job on a schedule shorter than the window.

The re-verify job runs cilock verify again on the records of the digest that serves, then attaches the new VSA with --replace:

set -euo pipefail                # adapted: a failed verify stops the script before the attach
rm -f evidence/vsa.json          # adapted: never attach a VSA from an earlier run
"$CILOCK" verify --offline -p "$POLICY" -k "$POLICY_PUB" \
  -a evidence/checkout.json -a evidence/build.json -a evidence/push.json \
  --subjects "$DIGEST" $SIGN $VSA_TSA --vsa-outfile evidence/vsa.json
"$CILOCK" attach "$STABLE_IMAGE@$DIGEST" --envelope evidence/vsa.json --replace

cilock verify exits non-zero when the evidence fails the policy. set -e then stops the script, so the attach does not run.

--replace deletes only the VSAs whose signed timeVerified is older than the new one. Kyverno requires each attached VSA to pass, so an old VSA next to a new one gets the pod denied. Kyverno also fails an image with more than 50 referrers. With --replace the image keeps one VSA.

On our own cluster we used a 30-minute window and ran the re-verify every 10 minutes. That gives three chances in each window, so one late or failed run does not deny pods. Make the job fail closed: if it cannot read the serving digest, finds more than one digest, cannot read the records, or the verify does not give PASSED for exactly that digest, it attaches nothing.

The platform verdict endpoint

On the TestifySec Platform, Kyverno can also ask the platform for a verdict on each image digest, and use the attached VSA when the platform cannot answer. The Kyverno admission integration generates that policy.

We measured its failure behavior on Kyverno v1.19.2-rc.1:

Any non-2xx answer counts as "unreachable", and the attached VSA decides. A revoked admission credential (401, 403) therefore does not show as a credential error. The cluster falls back to the VSA alone.

What fail-closed looks like in an outage

The policy does not set failurePolicy, so the webhook fails closed: the denial above came from mutate.kyverno.svc-fail. Plan for these results:

• Kyverno is down (not measured). The Kubernetes API server cannot call the webhook, so it refuses to create pods in each namespace that Kyverno's webhook selects. Pods that already run are not affected. Admission happens only at create time.
• The registry is down, or the pull secret is wrong (not measured). Kyverno cannot read the referrers, so it denies the pod.
• The VSA is too old. Each new or restarted pod is denied until a re-verify attaches a fresh VSA.

The outage reaches more than the namespaces in the policy. Kyverno cannot narrow the webhook of a verifyImages policy (kyverno issue 18042). Its failure policy applies to each namespace that the Kyverno chart's webhook namespaceSelector selects. On a Kyverno that serves only this policy, set that selector to the governed namespaces. On a shared Kyverno that change turns off the other Kyverno policies outside those namespaces. There, accept the outage risk or choose fail-open. Run the admission controller with 2 replicas and a PodDisruptionBudget.

To remove Kyverno from admission in an emergency, uninstall it and delete its webhooks. The controller creates the webhooks, not the chart, so they stay after a crashed controller:

helm uninstall kyverno -n kyverno --wait && \
kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \
  -l webhook.kyverno.io/managed-by=kyverno --ignore-not-found
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration | grep -i kyverno   # nothing

Troubleshooting

To read the VSA on an image:

mkdir -p vsa-dl && cilock download "$STABLE_IMAGE@$DIGEST" --predicate-type https://slsa.dev/verification_summary/v1 --outdir vsa-dl

cilock kyverno explain, not in a release yet, says which of these causes applies to an image.

Next

• Jenkins: record and verify a Pipeline build to produce the VSA.
• Verify in a release gate for other places to put the verify.