How CI/lock tracing works

cilock run --trace watches the command it wraps from the kernel's side and signs what it saw into the command-run/v0.2 attestation. Without --trace, materials and products come from a directory walk before and after the step. With it, they come from the files the build's processes opened and wrote.

How much a trace sees depends on the privileges cilock has where it runs. A container with Docker's default capabilities gets a thinner trace than a VM where cilock runs as root, and the run still succeeds. Nothing fails, but the attestation records which one you got. This page explains what a trace contains, how cilock chooses a tracer, and how to read the coverage record. Configure tracing covers the privileges each tracer needs and the setup for Docker, Kubernetes, GitHub Actions, GitLab and macOS.

What a trace records

Record	Where it lands	What it is used for
Processes	`processes[]`: program path and digest, argv, parent	Which tools the build really ran
Files read	each process's opened files, with a content digest	The step's materials
Files written	write records, then the products list	The step's products
Network	connections and sends, with the peer	Egress rules in policy
Coverage	`summary.coverage`	What the tracer could not see

The last row is the one this page is about. Every tracer misses something. cilock writes each known blind spot into summary.coverage as a named gap, so a verifier can decide whether a given trace is good enough instead of trusting it blindly.

The tracers

Tracer	`coverage.tracer`	Platform	How it observes the build
eBPF	`ebpf`	Linux	Kernel programs on the open, exec, write and network paths stream events to cilock
ptrace	`ptrace+seccomp`	Linux	cilock stops the build at each system call and reads its arguments
macOS sandbox	`sandbox-exec+oslog`	macOS	The command runs under `/usr/bin/sandbox-exec` with a profile that allows everything and reports exec, fork and network operations, which cilock reads through `/usr/bin/log`

On Linux, fanotify runs beside either tracer. It is an integrity gate, not a tracer: the kernel holds each open until cilock has hashed the file, so the digest is of the bytes the build was about to read. When fanotify is not running, cilock hashes files from their path at open time, and a file rewritten in between can be recorded with the wrong digest. That difference is the fanotify-unavailable gap.

The ptrace tracer's recorded name still says seccomp. The current tracer installs no seccomp filter; it stops the build at every system call with PTRACE_SYSCALL. It records the syscalls it has handlers for (exec, the open family, file operations and network calls) and reports the rest as the syscalls-untraced gap on every run. It refuses io_uring by default, because reads and writes through a ring are not system calls it can see. Set CILOCK_PTRACE_IOURING=allow to let a build use io_uring anyway; its ring I/O is then counted in the io-uring-untraced gap.

The macOS tracer needs no root and no entitlement. It cannot confirm that an exec succeeded or bind an exec's digest to the image the kernel loaded, so exec-success-unconfirmed and exec-digest-unbound are on every macOS trace. File reads are observed only inside the working directory (file-reads-scoped). --trace-file-content, which is macOS-only, also keeps bounded text snapshots of the files read there. Those snapshots can contain secrets, so turn it on only where that capture is approved.

On any other platform --trace fails the run.

How cilock picks a tracer on Linux

The tracer is chosen once, before the build starts, from these settings in order:

The --capture-mode suffix: trace:auto, trace:ebpf or trace:ptrace. A suffix overrides the environment variable.
CILOCK_TRACE_MODE: auto, ebpf or ptrace. Unset means auto. Any other value fails the run.

Mode	What happens
`auto` (default)	cilock probes eBPF: it creates a small BPF map and loads a two-instruction kprobe program. If both work it starts the eBPF tracer. If the probe fails, or the probe passes but the full tracer cannot load (a kernel older than 5.11, such as RHEL 8's 4.18, lacks BPF task storage), it falls back to ptrace and prints why.
`ebpf`	Same probe. If eBPF is unavailable the run fails before the build starts, with a message that lists ways to grant the capabilities.
`ptrace`	No probe. ptrace is used directly.

The fallback is printed on stderr:

cilock-trace: eBPF unavailable (BPF_MAP_CREATE failed: operation not permitted); falling back to ptrace+seccomp. For the full probe diagnostic, set CILOCK_TRACE_MODE=ebpf to force a loud failure.
cilock: tracing mode = ptrace+seccomp

and recorded in the signed attestation under _meta.host.tracer:

{ "backend": "ptrace+seccomp", "reason": "auto; eBPF unavailable: BPF_MAP_CREATE failed: operation not permitted" }

fanotify is chosen separately, by --hardening or CILOCK_FANOTIFY (the variable wins when both are set):

Setting	fanotify	When it cannot start
`--hardening standard` (default), `CILOCK_FANOTIFY=auto`	Used when available	The run continues, prints a warning, and records `fanotify-unavailable`
`--hardening strict`, `CILOCK_FANOTIFY=1`	Required	The run fails: `fanotify required but unavailable: ...`
`--hardening off`, `CILOCK_FANOTIFY=off`	Not used	Recorded as `fanotify-disabled`

--hardening strict also requires fs-verity on the workspace filesystem and turns on --require-zero-drops. On ext4 without the verity feature, which is the common case, strict fails with fs-verity required but unsupported on workspace FS. Keep fanotify required and make fs-verity opportunistic with CILOCK_FSVERITY=auto. --require-zero-drops fails the run when events were lost: ring-buffer drops, missed probe hits, unreadable ptrace stops, lost fanotify events, fs-verity seal failures, or opens the tracer did not record. It does not fail on structural gaps such as fanotify-unavailable or syscalls-untraced; write policy for those.

Reading the coverage record

summary.coverage in the command-run attestation looks like this. This one is from a root run on an Ubuntu 24.04 host (kernel 6.8), with detail strings shortened:

{
  "tracer": "ebpf",
  "complete": false,
  "gaps": [
    { "kind": "network-paths-unobserved", "detail": "network paths this trace does not record every occurrence of: [connect accept receive] ..." },
    { "kind": "products-outsider-unguarded", "count": 1, "detail": "products bound to a write by inode identity only ..." }
  ],
  "network": { "model": 1, "observed": ["send"], "lost": 0, "complete": false },
  "dimensions": {
    "files":    { "complete": true },
    "exec":     { "complete": true },
    "network":  { "complete": false, "gaps": ["network-paths-unobserved"] },
    "products": { "complete": false, "gaps": ["products-outsider-unguarded"] }
  }
}

gaps[] lists each known blind spot. kind is a stable string a policy can match. count, when present, is how many events it covers. detail says what was missed in words.
complete is true only when there are no gaps at all.
dimensions splits the gaps into four questions: did the trace see every file read and written (files), every process and its argv (exec), every way the build reached the network (network), and is every product bound to a write the trace saw (products). A gap kind can affect more than one dimension. A lost event of unknown type affects all four.

complete is false on every Linux trace today. Neither Linux tracer records every network path: the eBPF tracer records sends on every path but has no accept or receive probe, and its connect probe misses connects made through io_uring or the 32-bit compat table. The ptrace tracer claims no network path. So network-paths-unobserved is always present. Check the dimension your decision depends on. For "which files did this build read, and which programs did it run", that is files and exec. A best-case Linux trace, from cilock running as root with eBPF and fanotify, has both complete.

A trace from the ptrace tracer is never complete in any dimension, because syscalls-untraced affects all four. Neither is a macOS trace.

Check a trace from the command line

The output file is a DSSE envelope. Pull the tracer and coverage out with jq:

jq -r '.payload | @base64d | fromjson
  | .predicate.attestations[]
  | select(.type == "https://aflock.ai/attestations/command-run/v0.2")
  | .attestation
  | { tracer: ._meta.host.tracer,
      gaps: [.summary.coverage.gaps[]?.kind],
      dimensions: (.summary.coverage.dimensions | map_values(.complete)) }' build.att.json

A run that has the privileges it needs prints something like:

{
  "tracer": { "backend": "ebpf", "requested": "auto", "reason": "auto; eBPF probe passed" },
  "gaps": ["network-paths-unobserved", "products-outsider-unguarded"],
  "dimensions": { "files": true, "exec": true, "network": false, "products": false }
}

_meta.host also records what the trace ran with: the kernel release, the relevant sysctls (kernel.yama.ptrace_scope, kernel.perf_event_paranoid, kernel.unprivileged_bpf_disabled), the effective capability set as process.capEffective, container indicators, and tracee.drop, the privileges the traced build was started with. When a trace is thinner than you expected, start with capEffective.

Require coverage in policy

A step's Rego policy receives the command-run attestation as input. This rule refuses a trace whose file coverage is not complete, and also refuses an attestation from an older cilock that records no coverage:

package cilock.trace_files

deny[msg] {
  not input.summary.coverage.dimensions.files.complete
  msg := "trace does not claim complete file coverage, or records no coverage"
}

deny[msg] {
  not input.summary.coverage.dimensions.exec.complete
  msg := "trace does not claim complete exec coverage, or records no coverage"
}

Read the field inside the negation as shown. Binding cov := input.summary.coverage first and testing not cov.dimensions.files.complete fails open when coverage is absent. For a policy that accepts some named gaps and denies every other kind, see Requiring a complete trace in the high-assurance guide, and add network-paths-unobserved to its accepted set if your policy does not depend on network coverage.

Gap reference

Every gap kind cilock emits, and the dimensions it makes incomplete. The kind strings are stable; a policy matches on them.

Kind	Dimensions	Meaning
`tracer-unknown`	all	No coverage model for the tracer.
`fanotify-disabled`	files	fanotify was turned off; file digests are hashed from the path, not at the open.
`fanotify-unavailable`	files	fanotify could not start; file digests are hashed from the path, not at the open.
`fanotify-state-unknown`	files	No record of whether open-time hashing ran.
`fanotify-events-lost`	files, products	fanotify open and close-write events were lost.
`fanotify-rewrite-window`	files	A read of a rewritten file may not have been delivered.
`fanotify-reader-failed`	files	Opens after the fanotify reader failed were not observed.
`fanotify-kernel-denied`	files	An open was denied by the kernel, so the build read differently.
`fanotify-pgid-leader-unbound`	files	File events from the build's process group could not be tied to the build (kernels before 5.3 have no `pidfd_open`).
`fanotify-lineage-undecided`	files, products	File events not tied to the build's lineage or ruled out of it.
`fanotify-pre-start`	files	File events before the build was known were not attributed.
`fanotify-closers-unattributed`	products	Close-writes not shown to be the build's: products without a digest.
`opens-unhashed`	files	An open was recorded without the bytes it read.
`writers-unproven`	files	Reads of build-created files not shown free of outside writers.
`opens-unrecorded`	files	The build opened files the tracer did not record.
`opens-unwitnessed`	files	No witness counted opens outside the hooked open syscalls.
`files-inherited`	files	Reads through a descriptor inherited from before the trace make no open.
`path-name-race`	files	A recorded path named other bytes than the kernel opened.
`path-name-unproven`	files	A file name could not be proven to be the kernel's.
`path-resolved-at-entry`	files	A rename or unlink name was resolved before the kernel's lookup.
`write-hooks-missing`	products	Writes were judged by name, not by the file written.
`clock-stepped-back`	products	Products decided by mtime may be missing.
`products-unproven`	products	Products admitted on their times alone.
`products-unverified`	products	Products not bound to a write by identity.
`products-birth-bound`	products	Products bound by birth time on a filesystem without inode generations.
`products-outsider-unguarded`	products	Products bound by inode identity alone: an outside writer is not excluded.
`products-dropped`	products	Files opened for write and changed in the trace were left out of the products.
`file-reads-unobserved`	files	macOS: file reads were not proven delivered.
`file-reads-scoped`	files	macOS: file reads are observed only under a scope.
`file-reads-unproven`	files	macOS: file-read reports whose ownership was not proven.
`open-probe-missed`	files	Open-syscall probes the kernel did not run: opens were lost, nothing else.
`io-uring-untraced`	files, network, products	io_uring file and socket I/O is not recorded.
`exec-cmdlines-unread`	exec	An exec's argv was not delivered.
`exec-cmdlines-truncated`	exec	An exec's argv was longer than 128 KiB and was truncated.
`exec-success-unconfirmed`	exec	macOS: an exec's success is not confirmed.
`exec-digest-unbound`	files, exec	macOS: an exec'd image's digest is not bound to the file executed.
`images-unhashed`	files, exec	Exec'd images with no digest.
`execs-undigested`	files, exec	In-tree execs without an image digest.
`network-paths-unobserved`	network	Network paths the trace does not record every occurrence of.
`network-unobserved`	network	macOS: the network report channel was not proven to deliver.
`network-hosts-unobservable`	network	macOS: network endpoints carry ports only.
`network-reports-unproven`	network	macOS: network reports whose ownership was not decided.
`events-dropped`	all	eBPF ring-buffer events of any type were dropped.
`events-rejected-by-userspace`	all	Events of any type were discarded by cilock before recording.
`probe-events-missed`	all	Probe hits of any probe never ran the tracer's program.
`probe-misses-unmeasured`	all	Probes whose missed count could not be read.
`syscalls-untraced`	all	ptrace: the handler set does not record every file, exec and network syscall.
`syscall-stops-lost`	all	ptrace: syscall stops of any type could not be read.
`descendants-unconfirmed-dead`	all	A surviving descendant may act after the trace.
`diagnostics-missing`	all	No diagnostics to derive coverage from.
`processes-unproven`	all	Processes whose facts vanished: their activity is unattributed.
`unproven-execs-omitted`	all	Omitted processes' activity is not in the trace.
`ancestry-unresolved`	all	Processes not placed in the tree: their activity is unattributed.
`descendants-unobserved`	all	Descendants that produced no report, or children created with `CLONE_UNTRACED`.
`processes-reparented`	all	Reparented processes' activity is unattributed.
`processes-group-only`	all	Processes whose descent was not proven.
`reports-unattributed`	all	Reports of any type dropped for unproven ownership.
`reports-unreadable`	all	Reports of any type could not be read.
`pid-reuse`	all	A reused pid may have mixed two processes' events.
`reports-coalesced`	all	Folded reports lose processes of any activity.
`fork-reports-unmatched`	all	Children that left no other trace.

A kind your policy does not recognise should be treated as affecting every dimension; cilock's own derivation does the same.