# Operate and evaluate the appliance

Source: https://www.testifysec.com/docs/platform/operations/

Where each appliance operating procedure lives, what it needs, and whether it has been validated, plus the documents a buyer reviews before a pilot.

This is an index, not a new procedure. Each entry points to the authoritative source, lists what you need before you start, and states plainly whether the procedure has been validated. Where nothing exists yet, the entry says so. Do not treat a gap as covered because it appears in this list.

 

It applies to the single-host appliance described in [Appliance deployment](https://www.testifysec.com/docs/platform/appliance): one process, one data directory, SQLite by default, and **no built-in failover**. The [evaluation walkthrough](https://www.testifysec.com/docs/platform/appliance-walkthrough) shows local evaluation. It is not a production installation guide, and nothing on this page makes it one.

 

Most procedures below live in the operator runbook that ships inside the appliance binary, so it always matches the version you run:

 

```bash
judge-api runbook                     # list the runbooks in this binary
judge-api runbook self-host-minimal   # print the Linux runbook; sections are cited as §n below
```

 

## Operator procedures

 

Every procedure below needs administrative access to the appliance host. Your team performs it; see [responsibilities](https://www.testifysec.com/docs/platform/operations#for-buyers).

 

### Back up and restore

 

- **Source:** runbook §9.2 (backup) and §9.8 (restore). With local storage, include `blobs/`.
 - **Before you start:** a complete backup set from one §9.2 run, including `root-key.pem`, and the release that wrote it. Also record one artifact, its signed policy, and a `cilock verify` command that passes.
 - **Rollback:** the current data directory is moved aside, not deleted, and is moved back if any check fails.
 - **Check afterwards:** 
  
  - health and readiness;
   - sign-in;
   - the recorded `cilock verify` exits 0 without accepting a changed trust bundle;
   - policy releases and assignments match;
   - a new run uploads and verifies.
 - **Status:** documented, not yet rehearsed. It moves to validated only after a rehearsal in a disposable appliance passes every check.
 - **Root-key warning:** if `root-key.pem` is missing beside existing data, the service refuses to start rather than generate a new root key. A new key would be a new identity, not a restore: every credential derived from the old root would stop working. Releases before #12143 did generate one silently. Confirm the file is in place before the first start after a restore.

 

### Verify historical evidence

 

- **Source:** [Verify a release offline](https://www.testifysec.com/docs/cilock/getting-started/verify-a-release-offline), and verification against your own platform in [CI/lock trust](https://www.testifysec.com/docs/cilock/trust).
 - **Before you start:** the signed policy, and network access to your appliance, or the offline trust material.
 - **Rollback:** none needed. Verification is read-only.
 - **Check afterwards:** `cilock verify` exits 0 for evidence recorded before the change you are checking.
 - **Status:** verification is documented. Verifying historical evidence after a restore depends on the restore gap above, so it is not validated.

 

Offline verification of the appliance package before installation has its own runbook, tracked separately. Until it is published, follow the integrity check in runbook §2.3.

 

### Rotate the root and re-pin clients

 

- **Source:** runbook §9.9, which records the decision and its reasons.
 - **Before you start:** do not. Protect and back up `root-key.pem` instead (runbook §9.2), and restore the same key if you need it back (§9.8).
 - **Rollback:** not applicable. There is no rotation to roll back.
 - **Status: not supported (decision recorded, #12145).** On an appliance that holds data, a new root key makes stored secrets unreadable and invalidates every session and token. `cilock` clients would replace their pinned trust instead of adding to it, and the server does not detect a swapped key. If you believe the key is compromised, contact TestifySec. Runbook §9.9 lists what rotation would need first.

 

### Upgrade and roll back

 

- **Source:** runbook §9.3.
 - **Before you start:** back up first (runbook §9.2), and keep the previous binary or release package.
 - **Rollback:** swap the previous binary back and restart. Schema migrations run automatically at start and only add, never drop columns or tables, so the immediately prior release still runs against the upgraded database. There are no down-migrations. Test a jump of several versions in a disposable environment first.
 - **Check afterwards:** the health endpoints below, then a sign-in.
 - **Status:** documented. No rollback rehearsal is recorded for the appliance.

 

### License expiry

 

- **Source:** runbook §2.6 and §15.2.
 - **Behaviour:** 
  
  - The license is checked when the service starts. An expired license, a missing license, or a clock set back more than an hour stops the service from starting.
   - A running service is not stopped when its license expires. The next restart fails instead.
 - **Before you start:** a replacement license from TestifySec.
 - **Rollback:** keep the previous license file until the new one boots.
 - **Check afterwards:** the boot log line `entitlement verified method=signed-license`.
 - **Warning:** the web UI shows a license banner with the days remaining. It escalates in the last 14 days and once the license has expired. A dismissed banner returns when its severity changes.
 - **Status:** documented. Renew before the expiry date, because a restart after it fails.

 

### Identity provider (OIDC/JWKS) outage

 

- **Source:** runbook §9.6, and [Appliance deployment, license and identity](https://www.testifysec.com/docs/platform/appliance#license-and-identity).
 - **Impact:** 
  
  - Within 10 minutes, new federated sign-ins fail, and CI runners that use an OIDC token are refused with `invalid API credential`.
   - Existing sessions keep working, and password sign-in for a local administrator stays available.
 - **Before you start:** shell access to the appliance host, and the issuer URL of each provider you use.
 - **Rollback:** none needed. The procedure fixes connectivity and changes no credentials.
 - **Check afterwards:** the provider's discovery and key endpoints answer from the appliance host. Then a federated sign-in and one CI upload succeed.
 - **Status:** documented. Written from the code paths and not executed end to end against a simulated outage.

 

### Storage outage

 

- **Source:** runbook §9.7, and §8.
 - **Impact:** 
  
  - Uploads fail without leaving a partial record, and reading evidence bodies fails.
   - `cilock run` keeps each signed envelope it could not upload and prints how to upload it later.
 - **Before you start:** access to the store (bucket and instance role, or the local data disk) and the service log.
 - **Rollback:** none needed. Do not edit database rows to work around the outage.
 - **Check afterwards:** the store probe succeeds and a new run uploads. Then upload every envelope kept during the outage. A duplicate upload is safe.
 - **Status:** documented. Written from the code paths and not executed end to end against a simulated outage. The readiness endpoint still does not check blob storage, so watch the log as the runbook describes.

 

### Retention

 

- **Status: gap.** No evidence retention or deletion procedure exists for the appliance. The appliance does not delete evidence on a schedule. A platform setting can hide older evidence for some tenant states, but it hides the data and does not delete it.

 

### Monitoring

 

- **Source:** runbook §6 and §9.1.
 - **What exists:** 
  
  - `GET /healthz` reports that the process is up.
   - `GET /readyz` checks the database and that identity and the evidence store have started.
   - `judge-api healthcheck --url <url>` exits non-zero on anything but HTTP 200.
   - Service logs go to the system journal.
   - A metrics listener runs on port 9090. Keep it closed to users and CI runners.
 - **Status:** the endpoints exist. No alerting guide is published, and readiness does not cover blob storage, the license expiry date, or the identity provider.

 

## For buyers

 

- **Data flows:** the [network requirements](https://www.testifysec.com/docs/platform/appliance#network-requirements), [system architecture](https://www.testifysec.com/docs/concepts/architecture), and runbook §8 list each outside service the appliance calls. AI and object-storage providers receive data only when you enable them.
 - **Responsibilities:** your team operates the host, access, network, storage, monitoring, backup, restore, and updates. TestifySec supplies the software, releases, documentation, and support under your agreement. See [operations and availability](https://www.testifysec.com/docs/platform/appliance#operations-and-availability).
 - **Security documents:** the [Trust Center](https://www.testifysec.com/trust) lists what is public and what is available on request, including the policy package and subprocessor list. TestifySec does not currently hold a SOC 2 report.
 - **Pilot prerequisites:** a host sized for your workload, a signed license, an administrator, a trusted certificate and DNS name, and persistent storage for the data directory and signing material. Runbook §15 lists what changes from a test appliance to production. [Plan an appliance evaluation](https://www.testifysec.com/solutions/private-deployment).
 - **What evidence proves:** [Bring evidence you already have](https://www.testifysec.com/docs/platform/evidence-inputs) states what each input path's signature proves.

 

Questions this page cannot answer, such as failover, contractual commitments, or a procedure marked as a gap, need a written answer from TestifySec. [Contact TestifySec](https://www.testifysec.com/contact).