# Configure CI/lock tracing

Source: https://www.testifysec.com/docs/cilock/tracing-setup/

The kernel versions, capabilities and security settings each cilock tracer needs, what you lose without them, and tested setups for Docker, Kubernetes, GitHub Actions, GitLab, non-root and macOS.

cilock picks the best tracer the environment allows and records what it could not see. It does not stop a run because a privilege is missing, unless you ask it to. So the common failure is quiet: a build in a locked-down container gets the ptrace fallback and no fanotify, the run passes, and the trace reports gaps nobody reads. This page shows which privileges produce which trace, then gives setups per environment. Read [How tracing works](https://www.testifysec.com/docs/cilock/tracing) first for the terms used here.

 

## What each setup gets

 

We ran the same small command under `cilock run --trace` in each setup below and read the signed coverage record. The host was Ubuntu 24.04.5 on kernel 6.8 with AppArmor on, Docker 29.1.3, and cilock built from `main` on 2026-11-09. macOS was 27.2 on arm64 with cilock 4.9.0.

 

| Where cilock runs | Tracer | fanotify | `files` | `exec` |
| --- | --- | --- | --- | --- |
| Root on a Linux host or VM | eBPF | active | complete | complete |
| Docker, default capabilities | ptrace | unavailable | incomplete | incomplete |
| Docker, `--cap-add BPF --cap-add PERFMON` | eBPF | unavailable | incomplete | complete |
| Docker, `BPF`, `PERFMON` and `SYS_ADMIN`, default AppArmor profile | eBPF | unavailable (mount denied) | incomplete | complete |
| Docker, `BPF`, `PERFMON` and `SYS_ADMIN`, `--security-opt apparmor=unconfined` | eBPF | active | complete | complete |
| Docker, `--privileged` | eBPF | active | complete | complete |
| Not root, no capabilities (host or container) | ptrace | unavailable | incomplete | incomplete |
| Not root, file capabilities `cap_bpf,cap_perfmon` | eBPF | unavailable | incomplete | incomplete |
| Not root, file capabilities `cap_bpf,cap_perfmon,cap_sys_admin,cap_sys_chroot` | eBPF | active | incomplete | incomplete |
| Not root, file capabilities with `cap_sys_admin` but not `cap_sys_chroot` | the run fails |  |  |  |
| macOS, any user | macOS sandbox | n/a | incomplete | incomplete |

 

The `network` and `products` dimensions were incomplete in every run. `network-paths-unobserved` is present on every Linux trace (see [Reading the coverage record](https://www.testifysec.com/docs/cilock/tracing)). `products-outsider-unguarded` appeared on every Linux run because nothing in a trace rules out a process outside it writing the same file after the build's last write.

 

The two non-root eBPF rows are incomplete because of `probe-misses-unmeasured`: cilock could not read how many probe hits the kernel skipped, and that loss could be of any type. Running cilock as uid 0, including root inside a container, avoided it in our runs.

 

The short version: **run cilock as root with `CAP_BPF`, `CAP_PERFMON` and `CAP_SYS_ADMIN`, and let it mount**. cilock takes those privileges away from the build it traces; "The build's own privileges" below explains how.

 

## What each part needs

 

### eBPF tracer

 

- **Kernel:** Linux 5.11 or later with BTF type information (`/sys/kernel/btf/vmlinux`). The tracer's programs use BPF task storage, added in 5.11. On an older kernel the eBPF probe can pass while the tracer still fails to load; `auto` then falls back to ptrace and says why.
 - **Capabilities:** `CAP_BPF` and `CAP_PERFMON`, or root.
 - **Seccomp:** the container's profile must allow `bpf` and `perf_event_open`. Docker 29.1's default profile allowed them once the capabilities were added.
 - **Not needed:** `kernel.perf_event_paranoid` and `kernel.unprivileged_bpf_disabled` do not matter when cilock holds the capabilities. Our host had them at 4 and 2, and a non-root cilock with `cap_bpf,cap_perfmon` still ran the eBPF tracer.

 

If the eBPF probe fails, `auto` falls back to ptrace and prints the reason, for example `BPF_MAP_CREATE failed: operation not permitted`. Set `CILOCK_TRACE_MODE=ebpf` or `--capture-mode trace:ebpf` to make that a hard failure before the build starts.

 

### fanotify

 

fanotify needs more than one capability, because cilock gives the build a mount namespace of its own and marks only the build's mounts. That way a slow fanotify handler can stall the build, never the host.

 

- `CAP_SYS_ADMIN`, for `fanotify_init` and for `unshare(CLONE_NEWNS)`.
 - Permission to remount `/` as a slave mount inside that namespace. Docker's default AppArmor profile denies it, and the warning reads `make build mounts slave: permission denied`. Run the container with `--security-opt apparmor=unconfined`, or a profile that allows the mount.
 - `CAP_SYS_CHROOT`, so cilock can move its own thread back out of the build's namespace with `setns`. Root in a Docker container has it by default. A non-root cilock with file capabilities needs it granted explicitly. When `CAP_SYS_ADMIN` is present and `CAP_SYS_CHROOT` is not, fanotify starts and **the run then fails** with `cilock's thread could not be restored: re-enter mount namespace: operation not permitted`.

 

Under the default `--hardening standard`, missing fanotify is a warning and a `fanotify-unavailable` gap. Under `--hardening strict` or `CILOCK_FANOTIFY=1` the run fails with `fanotify required but unavailable`.

 

On kernels before 5.3 there is no `pidfd_open`, so fanotify events from the build's process group can arrive that cilock cannot tie to the build. They are counted in `fanotify-pgid-leader-unbound` and their files are not hashed for the build.

 

### ptrace tracer

 

- **Capabilities:** none for the usual case. cilock starts the build as its own child under `PTRACE_TRACEME`.
 - **Yama:** `kernel.yama.ptrace_scope` 0 or 1 allows it. At 2 the kernel also requires `CAP_SYS_PTRACE` in cilock. At 3 ptrace is disabled entirely.
 - **Seccomp:** the profile must allow `ptrace`. Docker's default profile does on kernels 4.8 and later.
 - **Kernel:** kernels before 5.3, such as RHEL 8's 4.18, are supported. cilock tracks syscall entry and exit itself there, because those kernels do not report it.

 

When ptrace itself is refused, cilock does not fall back to an untraced run. The run fails with `fork/exec <command>: operation not permitted` and a non-zero exit. The envelope it writes has no command-run attestation, so a policy that requires one refuses it. We checked this with a Docker seccomp profile that denies `ptrace`.

 

Even with every privilege, a ptrace trace carries `syscalls-untraced`, which marks all four dimensions incomplete. Use it when eBPF is impossible, not as the target.

 

### macOS

 

No root, no entitlement and no install beyond cilock. `/usr/bin/sandbox-exec` and `/usr/bin/log` must be present. Sandboxes do not nest: a build that runs `sandbox-exec` itself, such as Bazel's darwin sandbox, fails under `--trace` with `sandbox_apply: Operation not permitted`. cilock refuses to sign rather than record an untraced run as traced.

 

### What the tracer does not need

 

cilock's tracer does not read `/proc/<pid>/map_files`, so it does not need `CAP_CHECKPOINT_RESTORE`, which the kernel requires for that directory from 5.9 on (or `CAP_SYS_ADMIN` before). The Docker run with `BPF`, `PERFMON` and `SYS_ADMIN` and no `CHECKPOINT_RESTORE` got complete file coverage.

 

### The build's own privileges

 

cilock keeps the capabilities it needs and starts the build without them. The rule it applied is recorded as `_meta.host.tracee.drop`:

 

| `drop` | When | The build runs as |
| --- | --- | --- |
| `root-capless` | cilock is root, not under sudo (every root container) | uid 0, no capabilities, `no_new_privs` |
| `sudo` | cilock is root under `sudo` | the invoking user (`SUDO_UID`/`SUDO_GID`) |
| `user` | `CILOCK_TRACEE_USER=<uid>:<gid>` | that uid and gid, no capabilities |
| `inherited` | `CILOCK_TRACEE_USER=inherit` | cilock's uid and capabilities |
| `none` | cilock is not root | cilock's own user |

 

A build step that needs a capability itself breaks under `root-capless`. Use `CILOCK_TRACEE_USER=inherit` only if you accept that the build could then interfere with the tracer; the attestation says so, and a policy can refuse it.

 

## Setups

 

Each setup says whether we ran it. "Verified" means we ran it as written, or with only the image and command changed, and read the coverage record. "Not run" setups follow from the verified ones, and you should check your first trace with the [jq command](https://www.testifysec.com/docs/cilock/tracing) before relying on them.

 

### Docker (verified)

 

```bash
docker run --rm \
  --cap-add BPF --cap-add PERFMON --cap-add SYS_ADMIN \
  --security-opt apparmor=unconfined \
  -v "$PWD:/src" -w /src your-build-image \
  cilock run --trace --step build -o build.att.json -- make
```

 

This gave eBPF with fanotify active and complete `files` and `exec` coverage. `--privileged` gives the same trace, with far more access than cilock needs.

 

| Drop this | And you get |
| --- | --- |
| `--security-opt apparmor=unconfined` | eBPF, no fanotify (`fanotify-unavailable`): `exec` complete, `files` incomplete |
| `SYS_ADMIN` too | The same |
| `BPF` and `PERFMON` too | ptrace: nothing complete |

 

On a host that uses SELinux instead of AppArmor we have not tested the mount; if fanotify reports a permission error there, the container's SELinux label is the first thing to check.

 

### Kubernetes (not run)

 

The container that runs cilock needs the same three capabilities, uid 0, and an AppArmor profile that allows the mount:

 

```yaml
containers:
  - name: build
    image: your-build-image
    securityContext:
      runAsUser: 0
      capabilities:
        add: ["BPF", "PERFMON", "SYS_ADMIN"]
      appArmorProfile:
        type: Unconfined
```

 

`securityContext.appArmorProfile` needs Kubernetes 1.30 or later. On older clusters use the annotation `container.apparmor.security.beta.kubernetes.io/build: unconfined` on the pod. The Pod Security Standards `baseline` and `restricted` levels do not allow these capabilities, so the namespace needs the `privileged` level. If the pod uses `seccompProfile: RuntimeDefault` and cilock still reports `BPF_MAP_CREATE failed: operation not permitted` with the capabilities added, the runtime's profile is blocking `bpf`.

 

**Actions Runner Controller.** cilock runs wherever the workflow step runs. With the default runner mode the step runs in the `runner` container, so put the `securityContext` above on that container in the runner scale set's pod template. With `containerMode: kubernetes`, a job that sets `container:` runs in a separate job pod, and that pod needs the settings instead.

 

### GitHub Actions, hosted runners (not run)

 

Hosted Ubuntu runners are VMs, and the runner user has passwordless `sudo`. Run cilock under `sudo` and it gets the same privileges as the root-on-a-host row above, while the build drops back to the runner user:

 

```yaml
- name: Build
  run: |
    sudo -E "$(command -v cilock)" run --trace --step build \
      -o "$RUNNER_TEMP/build.att.json" -- make
```

 

`-E` keeps the `GITHUB_*` and `ACTIONS_ID_TOKEN_*` variables that attestors and keyless signing read. `$(command -v cilock)` avoids `sudo` resetting `PATH`.

 

cilock-action with `trace: true` runs cilock as the runner user without `sudo`. That is the not-root, no-capabilities row: ptrace and no fanotify. Use the CLI under `sudo` as above when you need complete coverage. For a job with `container:`, add the Docker options from the Docker setup to `container.options`.

 

### GitLab runners (not run)

 

For the Docker executor, set the capabilities and the AppArmor override in the runner's `config.toml`:

 

```toml
[runners.docker]
  cap_add = ["BPF", "PERFMON", "SYS_ADMIN"]
  security_opt = ["apparmor=unconfined"]
```

 

The job image must run as root. For the Kubernetes executor, apply the settings from the Kubernetes setup to the build container. On a shell executor, run cilock with `sudo` as on a GitHub hosted runner.

 

### Not root (verified)

 

When cilock cannot run as root, grant it file capabilities:

 

```bash
sudo setcap cap_bpf,cap_perfmon,cap_sys_admin,cap_sys_chroot+ep "$(command -v cilock)"
```

 

This gave eBPF with fanotify active. `files` and `exec` were still incomplete because of `probe-misses-unmeasured`, which root avoids. Do not grant `cap_sys_admin` without `cap_sys_chroot`: the run fails. With only `cap_bpf,cap_perfmon` you get eBPF without fanotify. File capabilities are lost when the binary is replaced, so reapply them after every upgrade.

 

### macOS (verified)

 

```bash
cilock run --trace --step build -o build.att.json -- make
```

 

No setup. A macOS trace is never complete in any dimension; see the gaps listed on [How tracing works](https://www.testifysec.com/docs/cilock/tracing).

 

## Check your trace

 

After a first run in a new environment, read three things from the attestation (the [jq command](https://www.testifysec.com/docs/cilock/tracing) prints them):

 

1. `_meta.host.tracer.backend` is `ebpf`. If it is `ptrace+seccomp`, `reason` says why.
 2. `summary.coverage.gaps` has no `fanotify-unavailable`.
 3. `summary.coverage.dimensions.files.complete` and `dimensions.exec.complete` are `true`.

 

Then turn that into policy, so a later change to the runner cannot thin the trace without failing verification. [Require coverage in policy](https://www.testifysec.com/docs/cilock/tracing) has the rules.

 

## Troubleshooting

 

| Symptom | Cause | Fix |
| --- | --- | --- |
| stderr: `eBPF unavailable (BPF_MAP_CREATE failed: operation not permitted)`; tracer is `ptrace+seccomp` | No `CAP_BPF`/`CAP_PERFMON`, or seccomp blocks `bpf` | Add `BPF` and `PERFMON`, or run as root; check the seccomp profile |
| tracer is `ptrace+seccomp` and `reason` says the probe passed but the tracer could not start | Kernel older than 5.11, or no BTF | Use a 5.11+ kernel with `/sys/kernel/btf/vmlinux` |
| Gap `fanotify-unavailable`, detail `unshare mount namespace: operation not permitted` | No `CAP_SYS_ADMIN` | Add `SYS_ADMIN` |
| Gap `fanotify-unavailable`, detail `make build mounts slave: permission denied` | AppArmor (or another LSM) denies the mount | `--security-opt apparmor=unconfined`, or `appArmorProfile: Unconfined` |
| Run fails: `re-enter mount namespace: operation not permitted` | `CAP_SYS_ADMIN` without `CAP_SYS_CHROOT` | Grant `CAP_SYS_CHROOT` too |
| Run fails: `fanotify required but unavailable` | `--hardening strict` or `CILOCK_FANOTIFY=1` without the fanotify privileges | Grant them, or use `--hardening standard` and enforce coverage in policy |
| Run fails: `fs-verity required but unsupported on workspace FS` | `--hardening strict` on a filesystem without fs-verity | Set `CILOCK_FSVERITY=auto` beside `--hardening strict` |
| Run fails: `attestation rejected (--require-zero-drops)` | Events were lost, often on a heavily loaded host | Re-run; give the build host headroom |
| Gap `events-dropped`, all dimensions incomplete | The ring buffer overflowed under load | Re-run; reduce parallel load on the host |
| Gap `probe-misses-unmeasured` | cilock could not read the kernel's missed-probe counts, as in non-root runs | Run cilock as root |
| Gap `syscalls-untraced` | The ptrace tracer ran | Fix whatever made eBPF unavailable |
| Run fails: `fork/exec ...: operation not permitted`, no command-run attestation | ptrace refused: seccomp, or `ptrace_scope` 3 | Allow `ptrace`; or get eBPF working |
| `CILOCK_TRACE_MODE="..." is not recognized` | Typo in the variable | Use `auto`, `ebpf` or `ptrace` |
| macOS: `sandbox_apply: Operation not permitted` | The build uses its own sandbox | Run that build without `--trace`, or turn off its sandboxing |
| Few or no materials, with gap `opens-unrecorded` | fanotify hashed files the build opened while the tracer recorded no open | Treat the material list as wrong; re-run, and check the tracer and its gaps |
| Few or no materials, no coverage record at all | The run was not traced (no `--trace`), so materials came from a directory walk | Add `--trace` |