Backup, restore & retention

One property shapes everything here: integrity travels with the data, not with the backup. Every record is hash-chained and origin-signed, every checkpoint is signed and re-verified on load, and a store whose disk was edited refuses to open instead of serving rewritten history. A backup therefore protects availability and confidentiality, never integrity. The corollary: a restored copy proves itself or refuses to load, so every restore ends with a machine-checkable verdict.

What there is to save

Artifact	Written by	Contents
WAL directory	gateway	one <chain>.jsonl per audit chain (HTTP transport: one per session)
Ledger store	ledger	per chain: <chain>.checkpoints.jsonl, <chain>.anchors.jsonl; store-wide: keys.json
Evidence packs	obsign-ledger export	self-contained proof — the retention artifact
Trust material	ops	trusted-keys.json, the signed deployment bundle — treat as configuration
Sealing key	HSM	not yours to file-copy: the vendor’s backup ceremony applies. Losing it stops new seals; it invalidates nothing already sealed. A replacement takes a new key_id

Everything in the first three rows is JSONL or JSON, append-only. That is deliberate, and it makes backup boring: files only grow, a torn final line is tolerated and trimmed by the owning process, and keys.json is replaced atomically. Hot copies are safe at the file level.

Backing up

Copy order matters. The store’s files reference each other, and Store::open refuses a store where references dangle. Copy in reference order, and store before WAL (sealed coverage must never run past the end of the log):

# Hot backup, safe while gateway and ledger run.
rsync -a store/*.anchors.jsonl      backup/store/ 2>/dev/null || true
rsync -a store/*.checkpoints.jsonl  backup/store/
rsync -a store/keys.json            backup/store/
rsync -a wal/                       backup/wal/

A single atomic filesystem snapshot (LVM, ZFS, EBS) of both directories is equivalent and simpler.

Frequency. Your backup interval is your RPO for the audit trail. The WAL is append-only and fsync’d per record, which makes continuous replication cheap; at minimum, back up at the sealing interval.

Verify every backup. A backup nobody has opened is a hope:

for f in backup/wal/*.jsonl; do
  chain=$(basename "$f" .jsonl)
  obsign-ledger export --wal backup/wal --store backup/store \
      --chain-id "$chain" --out "/tmp/$chain.pack.json" \
      --deployment-bundle deployment-bundle.json
  obsign verify --strict --trusted-keys trusted-keys.json \
      "/tmp/$chain.pack.json" || echo "BACKUP BAD: $chain"
done

Exit code 0 or the backup is not a backup. Then encrypt and follow 3-2-1: the copies carry no key material, but they do carry who did what.

Restoring

Ground rules: restore WAL and store from the same backup set. A newer WAL with an older store is fine (the unsealed tail gets sealed on the next pass); an older WAL with a newer store is TruncatedLog.

• Gateway host lost — restore the WAL onto a volume that honours fsync, restart. The gateway replays the log, verifies every record’s origin signature, and resumes where the copy ends. ForeignRecord on resume means either tampering or a missing rotation key in the deployment bundle: nothing is trimmed or adopted; a human decides. Acts performed after the last backup are absent. Say so in the incident record: a documented gap is defensible, a quietly shortened chain is not.
• Ledger host lost — restore the store and run one seal pass. A successful Store::open is the integrity check. DivergedLog means WAL and store disagree about sealed history: one of them is not the original, and the anchors arbitrate: an RFC 3161 token fixes what the checkpoint said, and when.
• Both hosts lost — the archived evidence packs are self-contained and verify offline with no service: history is not lost, it is already in its long-term form. Operations resume on fresh chains. Do not reconstruct WAL files from packs. Packs are the proof of record; new work gets new chains.
• Sealing key lost — nothing already sealed is affected. Provision a new key, seal under a new key_id. The old id stays bound to the old key forever, and that binding is what keeps old seals verifiable.

Retention

The retention artifact is the evidence pack rather than the raw WAL: one JSON file, offline-verifiable for as long as you can run sha256 and Ed25519. Per chain, the lifecycle runs: close, seal, anchor, export (the pack self-verifies), verify independently with the out-of-band keys, then archive the pack and its sha256 into immutable storage (e.g. S3 Object Lock, two locations). Only then prune the chain’s working files. Pruning removes whole chains’ files, never lines within a file. A shortened file is indistinguishable from tampering, and the tooling will treat it as such.

Periodic re-verification

Quarterly, and after every restore:

fail=0
for pack in archive/*/*/*.pack.json; do
  sha256sum -c "$pack.sha256" || fail=1
  obsign verify --strict --trusted-keys trusted-keys.json "$pack"
  [ $? -eq 0 ] || { echo "FAIL: $pack"; fail=1; }
done
exit $fail

Treat any non-zero exit as a failure, including 3. In a scheduled job, exit 3 means the trusted-keys file did not reach the verifier; it is not a pass.

Two long-horizon points: the trusted-keys.json used for re-verification must be the out-of-band copy under your control, and it must accumulate, never shrink: a key retired from new deployments still verifies two years of old seals. And archive the TSA’s certificate chain alongside the packs: proving who issued a timestamp in year two requires the issuer’s certificates, and the TSA will have rotated by then.

Once a year, run a restore drill: restore a random backup set to a scratch host and run the loop above. The drill’s exit code is the only evidence that this runbook works.