Operations

The operating model

You operate the whole stack through one service: the admin gateway on :8082. It launches and supervises every process (cluster nodes, media drivers, OMS, gateways, the backup agent), watches their health, and exposes everything as plain HTTP. You never kill -9 a node by hand or edit state files; you ask the gateway, and it does it in the right order.

Start and stop

# Bring everything up, in dependency order:
curl -X POST localhost:8082/api/admin/processes/start-all

# Health: one LEADER, all nodes live, counters fresh:
curl -s localhost:8082/api/admin/status | jq

# Stop everything, in reverse order:
curl -X POST localhost:8082/api/admin/processes/stop-all

/status tells the truth: it reports per-node process liveness and whether each node's counters are actually advancing, not just whether a PID exists.

Zero-downtime updates

The cluster survives losing one node, and rolling updates use exactly that: each node is stopped, updated, restarted, and fully rejoined before the next one starts. Trading continues throughout.

curl -X POST localhost:8082/api/admin/rolling-update

If a node fails to rejoin or catch up in time, the update hard-fails and stops rather than marching on and eroding the quorum.

Snapshots, backups, restore

• Snapshots compact the cluster's replay history so restarts are fast. They can run on a schedule, and disk below the latest snapshot is reclaimed live.
• Backups run continuously to disk through a dedicated agent. Freshness is monitored by heartbeat: /api/admin/backup-info says fresh or not, and why, because a backup process that is "running" but silently wedged is the most dangerous state in operations.
• Restore is one call (/api/admin/recover-from-backup), and the procedure has been validated with a power-loss drill: machine yanked mid-load, recovered with effectively zero data loss.

The guardrails

The gateway refuses operations that would hurt a degraded cluster:

• No snapshots or archive housekeeping while any member is down or lagging.
• Operations claim an exclusive slot: a snapshot cannot interleave with a rolling update.
• Cleanup endpoints preserve archives unless you explicitly confirm otherwise, twice.
• A crashed media driver takes its node down cleanly instead of leaving it wedged.
• Node restarts are capped: a crash-looping process gets stopped, not restarted forever.

Monitoring

Everything speaks Prometheus: each cluster node (:9500+), the OMS (/metrics on :8080), and the admin gateway. The match repo ships a Grafana starter kit and alert rules under grafana/. The admin gateway logs structured JSON with correlation ids, so an operation can be traced across services.

The full runbooks