# How CI/lock tracing works

Source: https://www.testifysec.com/docs/cilock/tracing/

What cilock run --trace records, the eBPF, ptrace and macOS tracing backends, how cilock picks one, and how to read summary.coverage to see what a trace missed.

`cilock run --trace` watches the command it wraps from the kernel's side and signs what it saw into the `command-run/v0.2` attestation. Without `--trace`, materials and products come from a directory walk before and after the step. With it, they come from the files the build's processes opened and wrote.

 

How much a trace sees depends on the privileges cilock has where it runs. A container with Docker's default capabilities gets a thinner trace than a VM where cilock runs as root, and the run still succeeds. Nothing fails, but the attestation records which one you got. This page explains what a trace contains, how cilock chooses a tracer, and how to read the coverage record. [Configure tracing](https://www.testifysec.com/docs/cilock/tracing-setup) covers the privileges each tracer needs and the setup for Docker, Kubernetes, GitHub Actions, GitLab and macOS.

 

## What a trace records

 

| Record | Where it lands | What it is used for |
| --- | --- | --- |
| Processes | `processes[]`: program path and digest, argv, parent | Which tools the build really ran |
| Files read | each process's opened files, with a content digest | The step's materials |
| Files written | write records, then the products list | The step's products |
| Network | connections and sends, with the peer | Egress rules in policy |
| Coverage | `summary.coverage` | What the tracer could not see |

 

The last row is the one this page is about. Every tracer misses something. cilock writes each known blind spot into `summary.coverage` as a named gap, so a verifier can decide whether a given trace is good enough instead of trusting it blindly.

 

## The tracers

 

| Tracer | `coverage.tracer` | Platform | How it observes the build |
| --- | --- | --- | --- |
| eBPF | `ebpf` | Linux | Kernel programs on the open, exec, write and network paths stream events to cilock |
| ptrace | `ptrace+seccomp` | Linux | cilock stops the build at each system call and reads its arguments |
| macOS sandbox | `sandbox-exec+oslog` | macOS | The command runs under `/usr/bin/sandbox-exec` with a profile that allows everything and reports exec, fork and network operations, which cilock reads through `/usr/bin/log` |

 

On Linux, **fanotify** runs beside either tracer. It is an integrity gate, not a tracer: the kernel holds each open until cilock has hashed the file, so the digest is of the bytes the build was about to read. When fanotify is not running, cilock hashes files from their path at open time, and a file rewritten in between can be recorded with the wrong digest. That difference is the `fanotify-unavailable` gap.

 

The ptrace tracer's recorded name still says `seccomp`. The current tracer installs no seccomp filter; it stops the build at every system call with `PTRACE_SYSCALL`. It records the syscalls it has handlers for (exec, the open family, file operations and network calls) and reports the rest as the `syscalls-untraced` gap on every run. It refuses `io_uring` by default, because reads and writes through a ring are not system calls it can see. Set `CILOCK_PTRACE_IOURING=allow` to let a build use `io_uring` anyway; its ring I/O is then counted in the `io-uring-untraced` gap.

 

The macOS tracer needs no root and no entitlement. It cannot confirm that an exec succeeded or bind an exec's digest to the image the kernel loaded, so `exec-success-unconfirmed` and `exec-digest-unbound` are on every macOS trace. File reads are observed only inside the working directory (`file-reads-scoped`). `--trace-file-content`, which is macOS-only, also keeps bounded text snapshots of the files read there. Those snapshots can contain secrets, so turn it on only where that capture is approved.

 

On any other platform `--trace` fails the run.

 

## How cilock picks a tracer on Linux

 

The tracer is chosen once, before the build starts, from these settings in order:

 

1. The `--capture-mode` suffix: `trace:auto`, `trace:ebpf` or `trace:ptrace`. A suffix overrides the environment variable.
 2. `CILOCK_TRACE_MODE`: `auto`, `ebpf` or `ptrace`. Unset means `auto`. Any other value fails the run.

 

| Mode | What happens |
| --- | --- |
| `auto` (default) | cilock probes eBPF: it creates a small BPF map and loads a two-instruction kprobe program. If both work it starts the eBPF tracer. If the probe fails, or the probe passes but the full tracer cannot load (a kernel older than 5.11, such as RHEL 8's 4.18, lacks BPF task storage), it falls back to ptrace and prints why. |
| `ebpf` | Same probe. If eBPF is unavailable the run fails before the build starts, with a message that lists ways to grant the capabilities. |
| `ptrace` | No probe. ptrace is used directly. |

 

The fallback is printed on stderr:

 

```text
cilock-trace: eBPF unavailable (BPF_MAP_CREATE failed: operation not permitted); falling back to ptrace+seccomp. For the full probe diagnostic, set CILOCK_TRACE_MODE=ebpf to force a loud failure.
cilock: tracing mode = ptrace+seccomp
```

 

and recorded in the signed attestation under `_meta.host.tracer`:

 

```json
{ "backend": "ptrace+seccomp", "reason": "auto; eBPF unavailable: BPF_MAP_CREATE failed: operation not permitted" }
```

 

fanotify is chosen separately, by `--hardening` or `CILOCK_FANOTIFY` (the variable wins when both are set):

 

| Setting | fanotify | When it cannot start |
| --- | --- | --- |
| `--hardening standard` (default), `CILOCK_FANOTIFY=auto` | Used when available | The run continues, prints a warning, and records `fanotify-unavailable` |
| `--hardening strict`, `CILOCK_FANOTIFY=1` | Required | The run fails: `fanotify required but unavailable: ...` |
| `--hardening off`, `CILOCK_FANOTIFY=off` | Not used | Recorded as `fanotify-disabled` |

 

`--hardening strict` also requires fs-verity on the workspace filesystem and turns on `--require-zero-drops`. On ext4 without the verity feature, which is the common case, strict fails with `fs-verity required but unsupported on workspace FS`. Keep fanotify required and make fs-verity opportunistic with `CILOCK_FSVERITY=auto`. `--require-zero-drops` fails the run when events were lost: ring-buffer drops, missed probe hits, unreadable ptrace stops, lost fanotify events, fs-verity seal failures, or opens the tracer did not record. It does not fail on structural gaps such as `fanotify-unavailable` or `syscalls-untraced`; write policy for those.

 

## Reading the coverage record

 

`summary.coverage` in the command-run attestation looks like this. This one is from a root run on an Ubuntu 24.04 host (kernel 6.8), with detail strings shortened:

 

```json
{
  "tracer": "ebpf",
  "complete": false,
  "gaps": [
    { "kind": "network-paths-unobserved", "detail": "network paths this trace does not record every occurrence of: [connect accept receive] ..." },
    { "kind": "products-outsider-unguarded", "count": 1, "detail": "products bound to a write by inode identity only ..." }
  ],
  "network": { "model": 1, "observed": ["send"], "lost": 0, "complete": false },
  "dimensions": {
    "files":    { "complete": true },
    "exec":     { "complete": true },
    "network":  { "complete": false, "gaps": ["network-paths-unobserved"] },
    "products": { "complete": false, "gaps": ["products-outsider-unguarded"] }
  }
}
```

 

- `gaps[]` lists each known blind spot. `kind` is a stable string a policy can match. `count`, when present, is how many events it covers. `detail` says what was missed in words.
 - `complete` is true only when there are no gaps at all.
 - `dimensions` splits the gaps into four questions: did the trace see every file read and written (`files`), every process and its argv (`exec`), every way the build reached the network (`network`), and is every product bound to a write the trace saw (`products`). A gap kind can affect more than one dimension. A lost event of unknown type affects all four.

 

**`complete` is false on every Linux trace today.** Neither Linux tracer records every network path: the eBPF tracer records sends on every path but has no accept or receive probe, and its connect probe misses connects made through `io_uring` or the 32-bit compat table. The ptrace tracer claims no network path. So `network-paths-unobserved` is always present. Check the dimension your decision depends on. For "which files did this build read, and which programs did it run", that is `files` and `exec`. A best-case Linux trace, from cilock running as root with eBPF and fanotify, has both complete.

 

A trace from the ptrace tracer is never complete in any dimension, because `syscalls-untraced` affects all four. Neither is a macOS trace.

 

### Check a trace from the command line

 

The output file is a DSSE envelope. Pull the tracer and coverage out with `jq`:

 

```bash
jq -r '.payload | @base64d | fromjson
  | .predicate.attestations[]
  | select(.type == "https://aflock.ai/attestations/command-run/v0.2")
  | .attestation
  | { tracer: ._meta.host.tracer,
      gaps: [.summary.coverage.gaps[]?.kind],
      dimensions: (.summary.coverage.dimensions | map_values(.complete)) }' build.att.json
```

 

A run that has the privileges it needs prints something like:

 

```json
{
  "tracer": { "backend": "ebpf", "requested": "auto", "reason": "auto; eBPF probe passed" },
  "gaps": ["network-paths-unobserved", "products-outsider-unguarded"],
  "dimensions": { "files": true, "exec": true, "network": false, "products": false }
}
```

 

`_meta.host` also records what the trace ran with: the kernel release, the relevant sysctls (`kernel.yama.ptrace_scope`, `kernel.perf_event_paranoid`, `kernel.unprivileged_bpf_disabled`), the effective capability set as `process.capEffective`, container indicators, and `tracee.drop`, the privileges the traced build was started with. When a trace is thinner than you expected, start with `capEffective`.

 

### Require coverage in policy

 

A step's Rego policy receives the command-run attestation as `input`. This rule refuses a trace whose file coverage is not complete, and also refuses an attestation from an older cilock that records no coverage:

 

```rego
package cilock.trace_files

deny[msg] {
  not input.summary.coverage.dimensions.files.complete
  msg := "trace does not claim complete file coverage, or records no coverage"
}

deny[msg] {
  not input.summary.coverage.dimensions.exec.complete
  msg := "trace does not claim complete exec coverage, or records no coverage"
}
```

 

Read the field inside the negation as shown. Binding `cov := input.summary.coverage` first and testing `not cov.dimensions.files.complete` fails open when `coverage` is absent. For a policy that accepts some named gaps and denies every other kind, see [Requiring a complete trace](https://www.testifysec.com/docs/cilock/guides/high-assurance-attestation) in the high-assurance guide, and add `network-paths-unobserved` to its accepted set if your policy does not depend on network coverage.

 

## Gap reference

 

Every gap kind cilock emits, and the dimensions it makes incomplete. The kind strings are stable; a policy matches on them.

 

| Kind | Dimensions | Meaning |
| --- | --- | --- |
| `tracer-unknown` | all | No coverage model for the tracer. |
| `fanotify-disabled` | files | fanotify was turned off; file digests are hashed from the path, not at the open. |
| `fanotify-unavailable` | files | fanotify could not start; file digests are hashed from the path, not at the open. |
| `fanotify-state-unknown` | files | No record of whether open-time hashing ran. |
| `fanotify-events-lost` | files, products | fanotify open and close-write events were lost. |
| `fanotify-rewrite-window` | files | A read of a rewritten file may not have been delivered. |
| `fanotify-reader-failed` | files | Opens after the fanotify reader failed were not observed. |
| `fanotify-kernel-denied` | files | An open was denied by the kernel, so the build read differently. |
| `fanotify-pgid-leader-unbound` | files | File events from the build's process group could not be tied to the build (kernels before 5.3 have no `pidfd_open`). |
| `fanotify-lineage-undecided` | files, products | File events not tied to the build's lineage or ruled out of it. |
| `fanotify-pre-start` | files | File events before the build was known were not attributed. |
| `fanotify-closers-unattributed` | products | Close-writes not shown to be the build's: products without a digest. |
| `opens-unhashed` | files | An open was recorded without the bytes it read. |
| `writers-unproven` | files | Reads of build-created files not shown free of outside writers. |
| `opens-unrecorded` | files | The build opened files the tracer did not record. |
| `opens-unwitnessed` | files | No witness counted opens outside the hooked open syscalls. |
| `files-inherited` | files | Reads through a descriptor inherited from before the trace make no open. |
| `path-name-race` | files | A recorded path named other bytes than the kernel opened. |
| `path-name-unproven` | files | A file name could not be proven to be the kernel's. |
| `path-resolved-at-entry` | files | A rename or unlink name was resolved before the kernel's lookup. |
| `write-hooks-missing` | products | Writes were judged by name, not by the file written. |
| `clock-stepped-back` | products | Products decided by mtime may be missing. |
| `products-unproven` | products | Products admitted on their times alone. |
| `products-unverified` | products | Products not bound to a write by identity. |
| `products-birth-bound` | products | Products bound by birth time on a filesystem without inode generations. |
| `products-outsider-unguarded` | products | Products bound by inode identity alone: an outside writer is not excluded. |
| `products-dropped` | products | Files opened for write and changed in the trace were left out of the products. |
| `file-reads-unobserved` | files | macOS: file reads were not proven delivered. |
| `file-reads-scoped` | files | macOS: file reads are observed only under a scope. |
| `file-reads-unproven` | files | macOS: file-read reports whose ownership was not proven. |
| `open-probe-missed` | files | Open-syscall probes the kernel did not run: opens were lost, nothing else. |
| `io-uring-untraced` | files, network, products | io\_uring file and socket I/O is not recorded. |
| `exec-cmdlines-unread` | exec | An exec's argv was not delivered. |
| `exec-cmdlines-truncated` | exec | An exec's argv was longer than 128 KiB and was truncated. |
| `exec-success-unconfirmed` | exec | macOS: an exec's success is not confirmed. |
| `exec-digest-unbound` | files, exec | macOS: an exec'd image's digest is not bound to the file executed. |
| `images-unhashed` | files, exec | Exec'd images with no digest. |
| `execs-undigested` | files, exec | In-tree execs without an image digest. |
| `network-paths-unobserved` | network | Network paths the trace does not record every occurrence of. |
| `network-unobserved` | network | macOS: the network report channel was not proven to deliver. |
| `network-hosts-unobservable` | network | macOS: network endpoints carry ports only. |
| `network-reports-unproven` | network | macOS: network reports whose ownership was not decided. |
| `events-dropped` | all | eBPF ring-buffer events of any type were dropped. |
| `events-rejected-by-userspace` | all | Events of any type were discarded by cilock before recording. |
| `probe-events-missed` | all | Probe hits of any probe never ran the tracer's program. |
| `probe-misses-unmeasured` | all | Probes whose missed count could not be read. |
| `syscalls-untraced` | all | ptrace: the handler set does not record every file, exec and network syscall. |
| `syscall-stops-lost` | all | ptrace: syscall stops of any type could not be read. |
| `descendants-unconfirmed-dead` | all | A surviving descendant may act after the trace. |
| `diagnostics-missing` | all | No diagnostics to derive coverage from. |
| `processes-unproven` | all | Processes whose facts vanished: their activity is unattributed. |
| `unproven-execs-omitted` | all | Omitted processes' activity is not in the trace. |
| `ancestry-unresolved` | all | Processes not placed in the tree: their activity is unattributed. |
| `descendants-unobserved` | all | Descendants that produced no report, or children created with `CLONE_UNTRACED`. |
| `processes-reparented` | all | Reparented processes' activity is unattributed. |
| `processes-group-only` | all | Processes whose descent was not proven. |
| `reports-unattributed` | all | Reports of any type dropped for unproven ownership. |
| `reports-unreadable` | all | Reports of any type could not be read. |
| `pid-reuse` | all | A reused pid may have mixed two processes' events. |
| `reports-coalesced` | all | Folded reports lose processes of any activity. |
| `fork-reports-unmatched` | all | Children that left no other trace. |

 

A kind your policy does not recognise should be treated as affecting every dimension; cilock's own derivation does the same.