# Jenkins: record and verify a Pipeline build with CI/lock

Source: https://www.testifysec.com/docs/cilock/guides/jenkins-record-and-verify-a-pipeline/

Run the checkout, build and push of a Java/Maven web app under CI/lock in a Jenkins Pipeline from SCM, verify the signed records each night by the image digest, attach a VSA, and promote only an image that passes.

note

Requires CI/lock 4.7.0-rc.2 (release candidate). We will update this guide when 4.7.0 is released.

 

This guide puts a Java/Maven web app on two Jenkins Pipeline jobs:

 

- A **compile job** runs `git clone`, `mvn` and `docker push`, each under `cilock run`. Each step writes a signed record.
 - A **nightly job** verifies the three records against a signed policy, from the image digest alone. If the records pass, it promotes the image and attaches the signed result, a VSA (verification summary attestation), to the image.

 

The compile job only records. It does not fail when a build is tampered with. The nightly job decides, and a failed verify stops it before the promotion.

 

```text
compile:  checkout -> build -> push     nightly:  verify -> promote -> attach     Kyverno:  admit by digest
```

 

The last part, admission in Kubernetes, is in [Kyverno: admit images only with a fresh, passing VSA](https://www.testifysec.com/docs/cilock/guides/kyverno-admit-with-a-fresh-vsa).

 

Other Jenkins pages:

 

- [Jenkins pipelines without changing the build script](https://www.testifysec.com/docs/cilock/tutorials/jenkins-pipeline) is a short tutorial: one traced step around a build script, and a release check. Start there if you do not use Maven or a container image.
 - [Sign Jenkins builds as the job](https://www.testifysec.com/docs/cilock/guides/jenkins-job-identity) replaces the file key in this guide with a short-lived certificate for the job, on the TestifySec Platform.
 - The blog post [Prove How a Jenkins Build Ran Without Changing the Build Script](https://www.testifysec.com/blog/jenkins-provenance-without-changing-the-build) explains why the record matters.

 

## What we tested

 

Every command on this page ran in a Jenkins Pipeline from SCM on 2026-10-17, with CI/lock 4.7.0-rc.2, on a Linux agent with Docker 29 (containerd image store) and a registry with the OCI referrers API. The app was a Maven web app that builds a WAR and an image. Host names, repository names and credential IDs on this page are examples. Lines marked **adapted** differ from the tested text, and the comment says how.

 

`cilock attach` and `cilock download` are new in 4.7. Use 4.7.0-rc.2 or later.

 

## Why a Pipeline from SCM

 

Jenkins has three job types that can run a Maven build. Only one gives a traced build and build steps that live in Git:

 

| Job type | Can CI/lock trace it? | Are the build steps in Git? |
| --- | --- | --- |
| Maven job type | No. It does not run `mvn`. Jenkins starts its own JVM. | No |
| Freestyle | Yes, if you wrap `mvn` | No. The steps live in the Jenkins configuration and can change with no commit. |
| Pipeline from SCM | Yes, if you wrap `mvn` | Yes. The Jenkinsfile is in the repository. |

 

Set **Definition** to `Pipeline script from SCM`, not a pasted script. Then a change to a build step is a commit.

 

To convert a Maven job, keep its node label, parameters, credential bindings, Maven goals, `MAVEN_OPTS`, settings file, archived artifacts and timeout. Two things do not carry over by themselves:

 

- The Maven job type passes job parameters to Maven as `-D` properties. A `sh` step does not. Pass each `${...}` that your pom expects on the `mvn` line.
 - A `tools { jdk ... }` line forces that JDK. If your pom compiles for a newer release, javac stops with `release version 25 not supported`. Our Jenkinsfile has no `tools` line and prints `java -version` first.

 

## Before you start

 

- Jenkins plugins: Pipeline, Git, Credentials Binding, SSH Agent, Copy Artifact, Timestamper and JUnit.
 - A Linux agent with Docker, `curl`, `sha256sum` and `git`.
 - Two key pairs. The CI key signs the records in the compile job. The VSA key signs the VSA in the nightly job, and Kyverno trusts only the VSA key:
  
   
  
  ```bash
mkdir -p keys
openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out keys/ci.key
openssl pkey -in keys/ci.key -pubout -out keys/ci.pub
openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out keys/vsa.key   # adapted: we tested one key for both
openssl pkey -in keys/vsa.key -pubout -out keys/vsa.pub
```
  
   
  
  Keep the two keys apart. The compile job gives the CI key to Maven's step, so a malicious build plugin can read it. If the same key signed the VSA, that plugin could sign a fake `PASSED` VSA that Kyverno admits.
 - A signed policy (next section).
 - Four Jenkins credentials of the kind **Secret file**: `cilock-signing-key` (`keys/ci.key`, compile job only), `cilock-vsa-key` (`keys/vsa.key`, nightly job only), `cilock-policy` (`policy.signed.json`) and `cilock-policy-pub` (`policy.pub`). Restrict `cilock-vsa-key` to the nightly job, for example in a folder that holds only that job.

 

## The policy

 

The policy has three steps. The CI key must sign each record, and each record must carry an RFC 3161 timestamp. These are the rules we used:

 

| Step | Rule |
| --- | --- |
| `checkout` | The command is `git clone --branch <branch> <approved URL> app`, exit code 0, traced. |
| `build` | Each file that the build read and that the checkout wrote has the same SHA-256 digest (`artifactsFrom: ["checkout"]`). A file that is not from the checkout fails, except the toolchain, the Maven cache, the Docker login and `target/` (`allowedUntracked`). |
| `build` | The command is `mvn`, exit code 0, traced, and `java` ran. |
| `build` | No process wrote, renamed or deleted a file in `src/`. No source file was read with two different contents. |
| `build` | Each program that the build started is on an approved list and runs from a toolchain directory. |
| `push` | The image went to the approved repository, and it is the image that Maven built. |
| `push` | The WAR in the pushed image has the digest of the WAR that the build wrote (`artifactsFrom: ["build"]`). |

 

The last rule lets the nightly job start from the image digest only. Verify finds the push record that names the digest. The push material leads to the build record that made the WAR, and the build reads lead to the checkout.

 

This is the approved-programs rule. The list is the programs that a clean build of our app ran. A name alone is not enough: a build can write its own `target/java`. So the rule also requires each program to run from a toolchain directory that the build cannot write to:

 

```rego
package example.webapp.build_commands

# Only approved programs run during the build. The trace records every program the build starts.
allowed := ["dirname", "expr", "java", "jspawnhelper", "ls", "mvn", "sh", "uname"]
# adapted: the tested rule checked the name only. Use the directories of your agent image.
toolchain := ["/usr/", "/bin/", "/opt/"]

deny[msg] {
    p := object.get(input, "processes", [])[_]
    i := object.get(p, "execPathId", -1)
    i >= 0
    exe := input.paths[i]
    not approved(exe)
    msg := sprintf("UNAPPROVED COMMAND: pid %v ran %v during the build. Command line: %v",
        [p.processid, exe, cmdline(p)])
}

approved(exe) {
    parts := split(exe, "/")
    allowed[_] == parts[count(parts) - 1]
    startswith(exe, toolchain[_])
}

cmdline(p) := input.cmdlines[i] {
    i := object.get(p, "cmdlineId", -1)
    i >= 0
}

cmdline(p) := "unknown" {
    object.get(p, "cmdlineId", -1) < 0
}
```

 

The rule checks the path of each program, not its digest. Keep the toolchain directories out of the build's writable paths: the build runs as a user that cannot write to `/usr`, `/bin` or `/opt`.

 

[Policies](https://www.testifysec.com/docs/cilock/concepts/policy-verification) gives the policy format. A different key, the policy key, signs the policy. Keep the policy key off the build agent. Sign on an offline machine:

 

```bash
openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out policy.key
openssl pkey -in policy.key -pubout -out policy.pub
cilock sign --offline -k policy.key -f policy.json -o policy.signed.json
```

 

## Step 1: Install a pinned CI/lock in the pipeline

 

The first stage of both Jenkinsfiles installs CI/lock into a tool folder of the job, outside the source tree. The agent image does not change. The stage verifies `install.sh` against a sha256 in the Jenkinsfile before it runs the script. `install.sh` verifies the download against the [cilock.dev manifest](https://cilock.dev/dl/manifest.json). Then the stage verifies the binary against a second sha256 in the Jenkinsfile. If cilock.dev changed, either check stops the job. The next builds use the same file and do not download it again.

 

`CILOCK_SHA256` is the sha256 of the `cilock` file in `cilock-4.7.0-rc.2-linux-amd64.tar.gz`. `CILOCK_INSTALLER_SHA256` is the sha256 of `install.sh` at that release. To upgrade, change `CILOCK_VERSION`, `CILOCK_SHA256` and `CILOCK_INSTALLER_SHA256` together.

 

```groovy
  environment {
    CILOCK_VERSION = 'v4.7.0-rc.2'
    CILOCK_SHA256  = 'b32d6821a602702c4795dbcb9f327950a9496efc93df183f12312f307006b5c3'
    CILOCK_INSTALLER_SHA256 = '362c3635624d95b0dcf906736bf77a295789648c4481f0ddc7ee7f3aed55323a'   // adapted: install.sh at v4.7.0-rc.2
    CILOCK_DIR     = "${env.WORKSPACE_TMP}/cilock-${CILOCK_VERSION}"   // per job, outside the source tree
    CILOCK         = "${CILOCK_DIR}/cilock"
    CILOCK_MAX_ATTESTATION_BYTES = '16MiB'   // adapted: added later, a cold Maven cache wrote a 4.4 MiB build record
  }
  stages {
    stage('Install cilock') {
      steps {
        sh '''
          pinned() { [ -x "$CILOCK" ] && [ "$(sha256sum < "$CILOCK" | cut -d' ' -f1)" = "$CILOCK_SHA256" ]; }
          if pinned; then
            echo "cilock $CILOCK_VERSION already installed at $CILOCK, sha256 matches"
          else
            mkdir -p "$CILOCK_DIR"                                  # install.sh does not create it
            curl -fsSLo "$CILOCK_DIR/install.sh" https://cilock.dev/install.sh
            echo "$CILOCK_INSTALLER_SHA256  $CILOCK_DIR/install.sh" | sha256sum -c -   # adapted: verify before it runs
            CILOCK_VERSION="$CILOCK_VERSION" CILOCK_BIN_DIR="$CILOCK_DIR" sh "$CILOCK_DIR/install.sh"
            rm -f "$CILOCK_DIR/install.sh"
            if ! pinned; then
              echo "STOP: $CILOCK has sha256 $(sha256sum < "$CILOCK" | cut -d' ' -f1), but the Jenkinsfile pins $CILOCK_SHA256. Not using it."
              rm -f "$CILOCK"; exit 1
            fi
          fi
          "$CILOCK" version
        '''
      }
    }
```

 

`CILOCK_MAX_ATTESTATION_BYTES` raises the size limit for each signed record. With a cold Maven cache, the build record of our app was 4.4 MiB, over the 4 MiB default, and the build step failed with `attestation too large`. Put the line in the `environment` block of both Jenkinsfiles.

 

The log shows `cilock 4.7.0-rc.2` and `Commit: 5e2786c4`. On the next build it shows `already installed at ..., sha256 matches`.

 

To put CI/lock in the agent image instead, add these lines to its Dockerfile. Then set `CILOCK_DIR = '/opt/cilock'`, and the stage only verifies the sha256:

 

```dockerfile
FROM <your agent image>
USER root
RUN mkdir -p /opt/cilock \
 && curl -fsSLo /tmp/cilock-install.sh https://cilock.dev/install.sh \
 && echo "362c3635624d95b0dcf906736bf77a295789648c4481f0ddc7ee7f3aed55323a  /tmp/cilock-install.sh" | sha256sum -c - \
 && CILOCK_VERSION=v4.7.0-rc.2 CILOCK_BIN_DIR=/opt/cilock sh /tmp/cilock-install.sh \
 && rm /tmp/cilock-install.sh
```

 

If your image sets a `USER`, put that line again after these lines.

 

## Step 2: The compile job

 

The compile job runs three traced or recorded steps. This is the Jenkinsfile, without the install stage from step 1:

 

```groovy
pipeline {
  agent { label 'linux-docker' }                         // adapted: your agent label
  triggers { githubPush() }
  options {
    timeout(time: 150, unit: 'MINUTES')
    timestamps()
    disableConcurrentBuilds()
    copyArtifactPermission('webapp-nightly')            // the nightly job copies evidence/ from this job
    skipDefaultCheckout()   // no second, untraced copy of the repository in the workspace
  }
  parameters {
    string(name: 'SRC', defaultValue: 'registry.example.com/apps/webapp:build', description: 'Image name')
    string(name: 'git_branch', defaultValue: 'main')
  }
  environment {
    REPO_URL     = 'git@git.example.com:acme/webapp.git'
    BRANCH       = "${params.git_branch}"   // params.*: on the first run Jenkins has not set $git_branch yet
    IMAGE        = "${params.SRC}"
    ROOT         = 'app'                    // clone folder; the policy expects app/
    // CILOCK_VERSION, CILOCK_SHA256, CILOCK_DIR, CILOCK and CILOCK_MAX_ATTESTATION_BYTES: see step 1
    MVN_SETTINGS = '/home/jenkins/.m2/settings.xml'                                  // adapted: your settings file
    WAR_IN_IMAGE = '/opt/jboss/wildfly/standalone/deployments/webapp.war'            // where the Dockerfile puts the WAR
    TRACE        = '--trace --capture-mode trace:ptrace'
    TSA          = '--timestamp-servers http://timestamp.digicert.com --timestamp-allow-http'
  }
  stages {
    // stage('Install cilock') from step 1
    stage('Checkout (traced)') {
      steps {
        sh 'rm -rf "$ROOT" "$ROOT.git" evidence push && mkdir evidence'
        sshagent(credentials: ['git-ssh']) {
          withCredentials([file(credentialsId: 'cilock-signing-key', variable: 'CILOCK_KEY')]) {
            sh '''
              "$CILOCK" run --step checkout $TRACE --signer-file-key-path "$CILOCK_KEY" $TSA --offline \
                -a environment --material-manifest -o evidence/checkout.json -- \
                git clone --branch "$BRANCH" "$REPO_URL" "$ROOT"
              git -C "$ROOT" rev-parse HEAD > evidence/commit
              mv "$ROOT/.git" "$ROOT.git"    # keep git's own folder out of the build inputs
            '''
          }
        }
      }
    }
    stage('Build (traced)') {
      steps {
        withCredentials([
          usernamePassword(credentialsId: 'registry-push', usernameVariable: 'DOCKER_REPOSITORY_USER', passwordVariable: 'DOCKER_REPOSITORY_PASSWORD'),
          file(credentialsId: 'cilock-signing-key', variable: 'CILOCK_KEY')]) {
          dir(env.ROOT) {
            sh 'java -version; mvn -v'
            // adapted: the tested goals were the job's own goals, ending in a docker-maven-plugin goal that builds the image
            sh '''
              "$CILOCK" run --step build $TRACE --signer-file-key-path "$CILOCK_KEY" $TSA --offline \
                -a environment,maven --material-manifest -o ../evidence/build.json -- \
                mvn -B -s "$MVN_SETTINGS" -gs "$MVN_SETTINGS" \
                  -DSRC="$IMAGE" -DBUILD_NUMBER="$BUILD_NUMBER" -DBUILD_URL="$BUILD_URL" \
                  clean install docker:build
            '''
          }
        }
      }
    }
    stage('Push (recorded)') {
      steps {
        withCredentials([
          usernamePassword(credentialsId: 'registry-push', usernameVariable: 'DOCKER_REPOSITORY_USER', passwordVariable: 'DOCKER_REPOSITORY_PASSWORD'),
          file(credentialsId: 'cilock-signing-key', variable: 'CILOCK_KEY')]) {
          sh '''
            WAR_BUILT=$PWD/$ROOT/target/docker/webapp.war              # adapted: where your build writes the WAR
            CID=$(docker create "$IMAGE"); docker cp -q "$CID:$WAR_IN_IMAGE" evidence/war-in-image.war; docker rm "$CID" >/dev/null
            mkdir -p push && cd push
            "$CILOCK" run --step push --signer-file-key-path "$CILOCK_KEY" $TSA --offline -a oci --material-manifest \
              --attestor-material-bind "$WAR_BUILT=../evidence/war-in-image.war" -o ../evidence/push.json -- docker push "$IMAGE"
            cd ..
            # The image digest, read from the signed push record, for the nightly job.
            sed -n 's/.*"payload": *"\\([^"]*\\)".*/\\1/p' evidence/push.json | base64 -d \
              | grep -o "registrydigest:$IMAGE@sha256:[0-9a-f]*" | awk -F@ 'NR==1 {print $2}' > evidence/image-digest
            echo "pushed $IMAGE = ${IMAGE%:*}@$(cat evidence/image-digest)"
          '''
        }
      }
    }
  }
  post {
    always {
      archiveArtifacts allowEmptyArchive: false, artifacts: "evidence/**,${env.ROOT}/target/surefire-reports/*.xml"
      junit allowEmptyResults: true, testResults: "${env.ROOT}/target/surefire-reports/*.xml"
    }
  }
}
```

 

Each line in this Jenkinsfile has a reason:

 

- **`skipDefaultCheckout()`.** Without it, Jenkins also checks the repository out into the workspace. The docker-maven-plugin then reads that copy's `.git`, and the build record lists files that the traced clone did not write. Verify fails.
 - **The traced clone.** The Git plugin checks out only the Jenkinsfile. The clone under `cilock run` is the record of which files the build used.
 - **`mv app/.git app.git`.** The docker-maven-plugin reads `.git/config`. git rewrites that file during the clone, so the checkout record has the first version, and verify fails with `mismatched digests for .../app/.git/config`.
 - **`docker push` in place of a Maven push goal.** The `oci` attestor records the digest that the registry returns. The nightly job verifies by that digest.
 - **`--attestor-material-bind`.** The push record's material is the WAR taken from the pushed image, under the path where the build wrote it. This links the push record to the build record.

 

The compile job prints these lines and `Finished: SUCCESS`:

 

```text
cilock run: step checkout ok, command exit 0; signer file, outfile evidence/checkout.json; --summary full for details
cilock run: trace ptrace+seccomp PARTIAL (gaps: fanotify-unavailable, syscalls-untraced)
[INFO] BUILD SUCCESS
cilock run: step build ok, command exit 0; signer file, outfile ../evidence/build.json; --summary full for details
cilock run: trace ptrace+seccomp PARTIAL (gaps: fanotify-unavailable, syscalls-untraced)
cilock run: step push ok, command exit 0; signer file, outfile ../evidence/push.json; --summary full for details
pushed registry.example.com/apps/webapp:build = registry.example.com/apps/webapp@sha256:8e8949f7...
```

 

The `PARTIAL` line is the trace coverage. See [What is proven and what is declared](https://www.testifysec.com/docs/cilock/guides/jenkins-record-and-verify-a-pipeline/#what-is-proven-and-what-is-declared).

 

### ptrace or eBPF

 

We tested with `--capture-mode trace:ptrace`. ptrace needs no extra container capabilities. Its cost grows with the number of JVM threads, and the JVM starts threads for each CPU. In a container with 8 CPUs, a warm Maven build of our app took 11 s to 12 s without CI/lock and 22 s to 27 s under ptrace (CI/lock before 4.7). With 4.7.0-rc.2 on 32 CPUs, the traced build in the Jenkins job took about 50 s.

 

`--capture-mode trace:ebpf` traces in the kernel. It costs less, but the `cilock` binary needs `CAP_BPF` and `CAP_PERFMON`, and the run fails if eBPF cannot load. We did not test eBPF in this Jenkins setup. [Capture modes](https://www.testifysec.com/docs/cilock/concepts/capture-modes) compares the backends.

 

## Step 3: The nightly job: verify, promote, attach

 

The nightly job copies `evidence/` from the last successful compile build. It verifies the three records by the image digest, then promotes and attaches. `set -euo pipefail` stops the shell at a failed verify, before the promotion.

 

```groovy
pipeline {
  agent { label 'linux-docker' }                                   // adapted: your agent label
  triggers { cron('H 2 * * *') }
  options {
    disableConcurrentBuilds()                                      // one writer for each image
    timestamps()
  }
  environment {
    // CILOCK_VERSION, CILOCK_SHA256, CILOCK_DIR, CILOCK and CILOCK_MAX_ATTESTATION_BYTES: see step 1
    IMAGE        = 'registry.example.com/apps/webapp'              // the compile job's SRC, no tag
    STABLE_IMAGE = 'registry.example.com/apps/stable/webapp'       // where the promotion pushes
  }
  stages {
    // stage('Install cilock') from step 1
    stage('Get the records') {
      steps {
        deleteDir()
        copyArtifacts(projectName: 'webapp-compile', selector: lastSuccessful(), filter: 'evidence/**')
      }
    }
    stage('Verify, promote, attach') {
      steps {
        withCredentials([
          file(credentialsId: 'cilock-vsa-key', variable: 'VSA_KEY'),       // adapted: the VSA key, not the CI key
          file(credentialsId: 'cilock-policy', variable: 'POLICY'),
          file(credentialsId: 'cilock-policy-pub', variable: 'POLICY_PUB')]) {
          sh '''#!/bin/bash
            NIGHTLY_TAG=$BUILD_NUMBER
            set -euo pipefail
            SIGN="--signer-file-key-path $VSA_KEY"
            VSA_TSA="--vsa-timestamp-servers http://timestamp.digicert.com --vsa-timestamp-allow-http"
            DIGEST=$(cat evidence/image-digest)

            # 1. Verify the three records against the signed policy, from the image digest alone. Write a signed VSA.
            "$CILOCK" verify --offline -p "$POLICY" -k "$POLICY_PUB" \
              -a evidence/checkout.json -a evidence/build.json -a evidence/push.json \
              --subjects "$DIGEST" $SIGN $VSA_TSA --vsa-outfile evidence/vsa.json

            # 2. The promotion: a copy with a new tag, never a new build.
            docker pull -q "$IMAGE@$DIGEST" > /dev/null
            docker tag "$IMAGE@$DIGEST" "$STABLE_IMAGE:$NIGHTLY_TAG"
            docker push -q "$STABLE_IMAGE:$NIGHTLY_TAG"

            # 3. The promotion must keep the digest. Then the VSA (which names that digest) is valid for the stable image.
            docker pull -q "$STABLE_IMAGE:$NIGHTLY_TAG" > /dev/null
            STABLE_DIGEST=$(docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$STABLE_IMAGE:$NIGHTLY_TAG" \
              | awk -F@ -v r="$STABLE_IMAGE" '$1 == r {print $2; exit}')
            if [ "$STABLE_DIGEST" != "$DIGEST" ]; then
              echo "STOP: the stable image has digest $STABLE_DIGEST, the verified image has $DIGEST"; exit 1
            fi

            # 4. Attach the VSA to the stable image. --replace removes the older VSA, so Kyverno sees one verdict per image.
            "$CILOCK" attach "$STABLE_IMAGE@$DIGEST" --envelope evidence/vsa.json --replace
            echo "promoted and attached: $STABLE_IMAGE:$NIGHTLY_TAG = $STABLE_IMAGE@$DIGEST"
          '''
        }
      }
    }
  }
  post {
    always { archiveArtifacts allowEmptyArchive: true, artifacts: 'evidence/vsa.json' }
  }
}
```

 

The job prints:

 

```text
level=info msg="Verification succeeded"
promoted and attached: registry.example.com/apps/stable/webapp:3 = registry.example.com/apps/stable/webapp@sha256:8e8949f7...
Finished: SUCCESS
```

 

Some registries keep attestations per repository path. A VSA attached to `apps/webapp` is then not on `apps/stable/webapp`, also for the same digest. For this reason the job attaches after the promotion, to the stable image.

 

To read the VSA back from the registry:

 

```bash
mkdir -p vsa-dl && cilock download "$STABLE_IMAGE@$(cat evidence/image-digest)" --predicate-type https://slsa.dev/verification_summary/v1 --outdir vsa-dl
python3 -c 'import base64,json,glob; e=json.load(open(glob.glob("vsa-dl/*.json")[0])); p=json.loads(base64.b64decode(e["payload"])); print(p["predicate"]["verificationResult"], p["predicate"]["timeVerified"])'
```

 

The second command prints `PASSED` and the time of the verify.

 

## Step 4: Tamper drills

 

Run each drill on a test tag, then run the nightly job. In each drill the compile job gives `SUCCESS`, because it only records. The nightly job must fail, and the image must not be promoted.

 

| Drill | Nightly job result |
| --- | --- |
| Clean build | `Verification succeeded`, promoted and attached |
| A build plugin runs `curl` | `UNAPPROVED COMMAND: pid 5805 ran /usr/bin/curl during the build`. Not promoted. |
| SUNSPOT: another process changes a source file while `javac` runs, then puts it back. `git status` is clean. | `mismatched digests for .../app/src/main/java/com/example/webapp/HealthServlet.java`. Not promoted. |
| A source file changed after the checkout, before Maven | `mismatched digests for .../HealthServlet.java`. Not promoted. |
| A different image pushed under the same tag | `no passed build record to compare with (the build step failed its own checks; see its reasons)`. Not promoted. |
| A wrong `CILOCK_SHA256` in the Jenkinsfile | The compile job fails in the `Install cilock` stage: `STOP: ... has sha256 b32d6821..., but the Jenkinsfile pins 000d6821... Not using it.` Nothing else runs. |

 

The SUNSPOT drill is the SolarWinds pattern. The compiler read a file that `git clone` did not write. The attacker restored the file after the compile, but the record of the read stays. A failed verify prints the reason for each step:

 

```text
level=error msg="Verification failed"
level=error msg="verification failure: Reason: \"failed to verify artifacts for step build: \nfailed to verify artifacts: mismatched digests for /home/jenkins/agent/workspace/webapp-compile/app/src/main/java/com/example/webapp/HealthServlet.java\""
level=error msg="collection rejected: \"push\", Reason: \"... policy was denied due to: no passed build record to compare with (the build step failed its own checks; see its reasons)"
Finished: FAILURE
```

 

Read the first step that failed. The other steps fail because they link to it.

 

## What is proven and what is declared

 

CI/lock finds tampering. It does not prevent it. The verify runs after the build, so a tampered build completes, and the nightly job refuses to promote it.

 

**Proven by the records and the policy:**

 

- The source came from `git clone` of the approved repository URL.
 - Each source file that Maven read has the digest of the file that `git clone` wrote. An untracked file fails.
 - No process in the build wrote, renamed or deleted a file in `src/`. Only approved programs from toolchain directories ran.
 - The WAR in the pushed image is the WAR that the traced build wrote, and the registry digest is the image that Maven built.

 

**Declared in the record, not proven.** Each traced record carries `summary.coverage`: the tracer that ran, `complete`, and a list of gaps. The `cilock run` line prints it: `trace ptrace+seccomp PARTIAL (gaps: fanotify-unavailable, syscalls-untraced)`.

 

- `fanotify-unavailable`: the container did not allow fanotify. File digests are open-time path hashes, not kernel-synchronous.
 - `syscalls-untraced`: ptrace traces a fixed set of syscalls. This gap is always present under ptrace.
 - `path-name-unproven`: CI/lock was not able to prove that a recorded file name is the name that the kernel used. Examples are a rename or unlink under ptrace, or a name that eBPF did not bind in the kernel.

 

A policy that needs a complete trace denies unless `input.summary.coverage.complete == true`. A policy that accepts known limits allows a named set of gap kinds and denies any other kind. Then a new gap kind is not accepted silently.

 

**Not covered:**

 

- The CI key is a file that the build step can read. A malicious build plugin can sign a fake record with it. The key does not prove which process signed. [Sign Jenkins builds as the job](https://www.testifysec.com/docs/cilock/guides/jenkins-job-identity) replaces the file key with a short-lived certificate that names the job, so Jenkins holds no long-lived key.
 - The Docker daemon is not a child of the build, so the tracer does not follow it. The policy proves that the image has the traced WAR. It does not prove what the other Dockerfile lines added. Pin the `FROM` image by digest. Verify the sha256 of each file that a `RUN curl` line downloads, in the same `RUN` line.
 - The record does not name a process outside the build. The SUNSPOT drill fails by the digests, but the attacker process is not in the record.
 - The image rule needs the containerd image store (the default in Docker 29). With the older store, the image ID is not the registry digest.

 

## Troubleshooting

 

| You see | Do this |
| --- | --- |
| `mismatched digests for .../app/.git/config` on a clean build | Put back the `mv "$ROOT/.git" "$ROOT.git"` line after the clone. |
| `mismatched digests for .../src/...` on a clean build | Something changed the source after the clone. Look for a plugin or script in the job that writes source files. |
| `UNAPPROVED COMMAND: ... ran /usr/bin/<x>` on a clean build | Examine the program. If it is correct, add it to the approved list, and sign the policy again. If the path is not under `/usr`, `/bin` or `/opt`, find out why the build runs a program from there. |
| `sha256sum: WARNING: 1 computed checksum did NOT match` in `Install cilock` | The downloaded `install.sh` is not the file that the Jenkinsfile pins. Do not run it. If you changed `CILOCK_VERSION`, change `CILOCK_INSTALLER_SHA256` to the sha256 of `install.sh` for that version. |
| `did not come from the approved repository` | The repository URL in the Jenkinsfile and in the policy are not the same text. Make them the same, also the `.git` at the end. |
| `no passed build record to compare with` | Read the reasons for the `build` step first. Make sure that the push step has `--attestor-material-bind` and that the policy has `artifactsFrom` on `push`. |
| `STOP: .../cilock has sha256 ..., but the Jenkinsfile pins ...` | Do not use the downloaded binary. If you changed `CILOCK_VERSION`, change `CILOCK_SHA256` to the sha256 for that version. |
| `STOP: the stable image has digest ...` | The promotion made a new image. Change it to copy the image with a new tag. |
| `release version 25 not supported` | The job forces an older JDK. Remove the `tools { jdk ... }` line, and make sure that the agent image has the JDK that the pom needs. |
| `attestation too large` in the build step | The build record is over the 4 MiB default, usually on a cold Maven cache. Set `CILOCK_MAX_ATTESTATION_BYTES = '16MiB'` in the `environment` block (step 1). |
| `unknown flag: --timestamp-servers` on `cilock verify` | The VSA flags are `--vsa-timestamp-servers` and `--vsa-timestamp-allow-http`. |
| `cilock-trace: killed 1 process(es) the traced command left running ...` | CI/lock stops each process that a traced step leaves running. Start long-lived services outside the traced step. Run Gradle with `--no-daemon`. |
| The traced build stops, or takes more than 3 times as long as plain `mvn` | Make sure that `cilock version` shows 4.7.0-rc.2 or later. If it still stops, set `JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=8` as a temporary fix. |
| `cilock attach` fails | The registry can have no OCI referrers API. See [`cilock attach`](https://www.testifysec.com/docs/cilock/reference/cli#cilock-attach-reposha256hex) for the routes. |

 

## Next

 

- [Kyverno: admit images only with a fresh, passing VSA](https://www.testifysec.com/docs/cilock/guides/kyverno-admit-with-a-fresh-vsa) to deploy only an image that the nightly job passed.
 - [Verify in a release gate](https://www.testifysec.com/docs/cilock/guides/verify-in-a-release-gate) for other places to put the verify.

Reference generated from the product documentation. Match commands and support details to your installed release.