Alerting on divergence

When the ledger finds that sealed history no longer matches the WAL, it refuses to seal and exits non-zero. That is deliberate: a rewritten log is an incident, not a heartbeat, and a service that retried it every thirty seconds would dress the incident up as one. These conditions never self-heal:

Error	Meaning
DivergedLog	the record at the sealed boundary no longer hashes to the sealed head — the WAL was rewritten after sealing
TruncatedLog	sealed records have disappeared from the WAL
UnauthenticatedRecord	a record no trusted origin key vouches for — someone other than the gateway wrote to the WAL
StoreBroken	the checkpoint store itself is corrupt
KeyConflict	a key id reused with different key material

Everything else (an I/O blip, a WAL segment caught mid-write) is retried inside run mode’s loop and never exits. So any exit of run mode is in the never-self-heals class by construction.

Exiting is only half the job; who is told? Not by the ledger itself, which makes no network calls, ever: a webhook that cannot leave the enclave is an alert nobody receives. Telling someone is the supervisor’s job, and on the target deployments the supervisor is systemd.

The OnFailure= pattern

[Unit]
Description=Obsign ledger — seals the gateway audit log
OnFailure=obsign-alert@%n.service

[Service]
Type=exec
User=obsign-ledger
ExecStart=/usr/local/bin/obsign-ledger run \
    --wal /var/lib/obsign/wal --chain-id prod \
    --store /var/lib/obsign/ledger \
    --hsm-module /usr/lib/pkcs11/vendor.so \
    --hsm-key-label seal-prod \
    --hsm-pin-file /etc/obsign/hsm-pin \
    --key-id seal-prod
# No Restart=. Transient failures are retried inside the loop and never
# exit; whatever does exit never self-heals. Restart=on-failure would
# re-detect the same divergence at every start — the heartbeat the exit
# code exists to avoid.
Restart=no

[Install]
WantedBy=multi-user.target

The alert unit (obsign-alert@.service, a template instantiated with the failing unit’s name) runs a site-specific script. Everything in it is local or stays on the management network:

#!/bin/sh
unit="$1"

# 1. A critical journal entry — the SIEM collector that already reads
#    this host's journal picks it up without any new plumbing.
echo "AUDIT INCIDENT: $unit failed — sealed history no longer matches \
the WAL. journalctl -u $unit for the refusal." | systemd-cat -t obsign -p crit

# 2. Everyone logged in on the host, immediately.
echo "AUDIT INCIDENT: $unit failed — see journalctl -u $unit" | wall

# 3. Local mail through the site's internal relay, evidence attached.
{ printf 'Subject: [obsign] AUDIT INCIDENT on %s: %s\n\n' "$(hostname)" "$unit"
  journalctl -u "$unit" -n 50 --no-pager
} | sendmail ops@site.internal

The ledger’s refusal message says which sequence number diverged and whether an authentic prefix was sealed. It is on stderr, hence in the failed unit’s journal, and the alert only has to point at it.

Slack, without breaking the air gap

A chat alert is worth having, as long as the same rule holds: the network call lives in the supervisor. The shipped obsign-slack-alert.sh posts to an Incoming Webhook under three constraints:

• Best-effort, never load-bearing. It runs alongside the local channels (journal, wall, mail), which always deliver inside the enclave. If Slack cannot be reached the script logs a warning and exits 0. An alert channel must never become the incident.
• Bounded. Connect and total timeouts; the webhook URL is read from a root-owned file, never baked into a unit or an argument.
• Egress-aware. Honours https_proxy; a site with no egress runs a store-and-forward relay on the management network instead; the enclave itself never opens an outbound connection.

Cron-style deployments: seal + a timer

One-pass seal is the cron- and air-gap-friendly mode, with one addition. A timer keeps firing after a failure, and each pass would re-detect and re-alert the same divergence. The latch stops sealing after the first incident until an operator explicitly clears it:

# obsign-seal.service
[Unit]
Description=Obsign sealing pass
OnFailure=obsign-alert@%n.service
# The latch: once the flag exists the timer stops producing alerts (and
# seals) until the operator removes the file. Clearing it is a deliberate
# act that concludes an investigation, not a retry.
ConditionPathExists=!/var/lib/obsign/ledger/INCIDENT

[Service]
Type=oneshot
User=obsign-ledger
ExecStart=/usr/local/bin/obsign-ledger seal ...
ExecStopPost=/bin/sh -c '[ "$$SERVICE_RESULT" = success ] || \
    touch /var/lib/obsign/ledger/INCIDENT'

One honest caveat: unlike run mode, a one-shot seal cannot retry internally, so a transient failure (HSM briefly unreachable) also raises the alert. The journal tail says which class you are in. If that distinction matters at your scale, run the daemon; its exits are all incident-class.

Containerized ledgers

The distroless ledger image has no systemd inside, so keep the supervisor outside. Run the container from a systemd unit (ExecStart=docker run --rm ...): the container’s exit code becomes the unit’s, and OnFailure= applies unchanged. A compose restart: always policy on the ledger converts the incident back into a heartbeat, with nobody told.

When it fires

Treat it as an incident. The WAL and the ledger store are the evidence; do not “fix” them to get sealing green again. obsign-ledger export still works and writes the pack even when it fails verification: a failing pack on disk is exactly what the investigation wants. The last sealed checkpoint (and its RFC 3161 anchor, if any) bounds when the rewrite happened: everything up to to_seq is still provable, and that boundary is the starting point of the timeline.