Skip to main content
View Markdown ↗

Copy this page

Select and copy the Markdown below, then paste it into your LLM.

Keep the result of a recovery rehearsal

Record both a successful database restore and a rehearsal that misses writes made after the backup.

Jump to the written guide ↓

Follow along

Goal: record a successful recovery rehearsal and a failed one as signed evidence. A signed failed result remains a failed result.

Before you start

  • Install CI/lock and connect it to your platform using the approved human or agent enrollment flow.
  • Work in a disposable test repository with a committed recovery rehearsal at scripts/restore-rehearsal.sh. It must return zero only when your recovery checks pass, and nonzero when they fail.
  • Prepare disposable test data and an isolated database environment. The commands below wrap your own reviewed rehearsal, not a built-in CI/lock recovery test.
  • The second command uses the fixture's REHEARSAL_LATE_WRITES switch. Your script must implement this switch, or you must use its documented failure injection instead. Merely setting an environment variable does not create a recovery gap.

1. Confirm the rehearsal and identity

cilock version
cilock agent status
git rev-parse HEAD
test -x ./scripts/restore-rehearsal.sh

Expected: the script exists and is executable, and the intended enrolled agent has a usable signing identity. If running as a person, follow signing and identity instead of assuming an agent session is a human session.

2. Record the successful rehearsal

Use the platform endpoint configured by your approved enrollment instructions. CI/lock captures command output and file observations; keep secrets out of the rehearsal output.

cilock run --step recovery-rehearsal \
  -o rehearsal.json -- ./scripts/restore-rehearsal.sh

Expected: the restore runs against the disposable environment, its checks pass, and rehearsal.json is written. Choose checks that test your recovery requirements. Comparing restored row counts and fingerprints is narrower than proving complete application recovery.

3. Record a known recovery gap

For a rehearsal implementing late-write injection with the REHEARSAL_LATE_WRITES input:

REHEARSAL_LATE_WRITES=37 cilock run \
  --step recovery-rehearsal --ignore-command-exit-code \
  -o gap.json -- ./scripts/restore-rehearsal.sh

Expected: the rehearsal identifies missing data and exits nonzero, while CI/lock writes the evidence. The flag keeps evidence collection running after the wrapped command fails. It must not be used as a reason to treat the rehearsal as successful.

4. Inspect both records

With jq installed:

jq -r .payload rehearsal.json | base64 -d | jq '.predicate'
jq -r .payload gap.json | base64 -d | jq '.predicate'

Expected: the records identify the same rehearsal step and their respective command results. Decoding a payload is inspection, not signature or policy verification. If platform storage is configured, also inspect the stored records and their subjects in the platform.

Troubleshooting and cleanup

A signing or upload error is different from a failed restore. Preserve the complete stderr and exit status while diagnosing it. If the injected run succeeds, check that your script actually implements the failure switch. Clean up only the disposable resources your rehearsal created, using that script's documented teardown.

Next: evaluate both results against a policy. Configure technical-control mappings separately; completing this exercise does not establish compliance with a framework or control.

Browse the other walkthroughs