Fleet Scaling Plan: Postgres Registry, Redis Caching, RBAC and Audit

1. Goal and decisions

2. Where the code stands today (survey, 2026-10-09)

3. Target architecture

                 ┌──────────── MCP clients (Claude, agents, A2A) ────────────┐
                 │  Bearer token → TokenPrincipal(role, controller selector) │
                 └──────────────────────────────┬────────────────────────────┘
                                                │ (load balancer; sticky per client OR stateless_http)
                      ┌─────────────────────────┼─────────────────────────┐
                      ▼                         ▼                         ▼
              ┌──────────────┐          ┌──────────────┐          ┌──────────────┐
              │  API replica │   ...    │  API replica │          │ Worker (n)   │
              │ tool wrapper │          │              │          │ sync / health│
              │  → resolver  │          │              │          │ leased shards│
              │  → ClientPool│          │              │          └──────┬───────┘
              └──────┬───────┘          └──────┬───────┘                 │
                     │                         │                         │
        ┌────────────┴───────┬─────────────────┴────────┬────────────────┘
        ▼                    ▼                          ▼
 ┌─────────────┐     ┌──────────────────────────┐   ┌───────────────────────────────┐
 │  Postgres   │     │          Redis           │   │ UniFi gateways / api.ui.com   │
 │ registry,   │     │ read cache, session      │   │ (hundreds–thousands)          │
 │ creds(enc), │     │ state, rate-limit        │   └───────────────────────────────┘
 │ sites,      │     │ buckets, leases, pub/sub │
 │ inventory,  │     │ (webhooks, invalidation) │
 │ tokens,audit│     └──────────────────────────┘
 └─────────────┘

3.1 The central idea: route in the wrapper, not in 275 tools

1. Add an optional controller: str | None = None parameter to the public signature (skipped for tools that are controller-agnostic, e.g. registry management tools).
2. On each call, resolve the target: explicit argument → ctx.get_state("active_controller") → registry default → error.
3. Build (or fetch from an LRU) a per-controller Settings via base_settings.model_copy(update=profile.settings_overrides()), and inject that as settings.
4. Bind controller_id into a ContextVar for the duration of the call, so the client pool, cache keys, audit records, and logs all pick it up without signature changes.

3.2 Postgres (system of record)

• EnvRegistry: a single controller named default, built from today’s UNIFI_* env vars. Always available, so this is the zero-config path.
• PostgresRegistry: backed by the tables above, with an in-process snapshot refreshed on a short interval and on registry:changed pub/sub events, so the hot path never queries Postgres per tool call.

3.3 Redis (shared, ephemeral state)

3.4 Connection management at thousands of controllers

• ClientPool keyed by controller_id, holding one long-lived httpx.AsyncClient per controller. LRU-bounded (FLEET_MAX_OPEN_CLIENTS, default 256) with idle eviction, since keeping a thousand TLS pools open per replica isn’t viable. UniFiClient.__aenter__ takes the pooled transport when a ContextVar pool is active, and __aexit__ does not close it. Without a pool, behavior is unchanged.
• Per-controller circuit breaker: after consecutive failures, mark health=down and fail fast with a clear error for a cool-off period instead of burning timeouts. The worker’s health checks close it again.
• Fan-out reads (new fleet_* tools, read-only): bounded concurrency (asyncio.Semaphore, configurable), a per-controller timeout, and partial results: each controller returns {ok, data} or {error}, never all-or-nothing. They prefer device_inventory snapshots when fresh enough, and go live only on request.

3.5 Worker tier

• Controllers are sharded across workers by consistent hashing over Redis-registered worker IDs. Shard ownership is held by leases, so a dead worker’s shard is picked up after lease expiry.
• Jobs: health check (cheap probe, updates health / capabilities), site sync → sites, inventory sync → device_inventory (staggered with jitter so a thousand controllers aren’t all polled at :00), and cloud-account discovery (Site Manager /v1/hosts → upsert controllers with cloud_account_id).
• API replicas never poll. They read snapshots and do targeted live calls.

3.6 Access control (RBAC)

• DB tokens are 32 random bytes generated by the server, shown once, and stored as SHA-256 (enough for high-entropy tokens; no password hashing needed). They support expires_at and revoked_at. Revocation reaches every replica through the tokens:changed pub/sub message, and token lookups are cached for at most 30 seconds.
• MCP_AUTH_TOKEN tokens map to admin, which keeps today’s behavior. They serve as break-glass access and are flagged in audit records.
• stdio is a local, single-user channel: its principal is admin (configurable with UNIFI_STDIO_ROLE).

3.7 Audit logging

• mutating call, including dry-run, with result success, error or dry_run
• denied call: RBAC refusals and rejected write targets (§3.1), with result denied
• admin/config event: token create/revoke/role change, controller registry changes, credential and key rotation, audit queries and exports, and server start with its effective safety configuration (read_only, dry_run, profile, registry source)

• JsonlSink: today’s file, still the default without a database. It gets size/date rotation, and the chain continues across files.
• PostgresSink: the append-only audit_log table (§3.2). It’s primary whenever DATABASE_URL is set.
• SIEM exporters: syslog (RFC 5424 over TLS), OTLP logs and an HMAC-signed HTTPS webhook. They’re asynchronous and best-effort, with an on-disk retry buffer, so a slow SIEM never blocks a tool call. They receive the stored (encrypted-payload) form of each record. Configuring a target requires UNIFI_AUDIT_CHAIN_KEY (AC-531).

4. Phased delivery

Phase 0: Foundations and fixes (no behavior change)

• Add redis_url/redis_* fields to Settings and fix CacheClient to honor them (fixes the documented-but-ignored REDIS_HOST). Move to a shared connection pool.
• Add database_url, fleet_* settings. Add [fleet] extra (sqlalchemy[asyncio], asyncpg, alembic, redis).
• Add ControllerProfile (Pydantic) and the ControllerRegistry protocol with EnvRegistry.
• docker-compose.fleet.yml with Postgres 16 + Redis 7 for development and integration tests.
• Verify: full unit suite unchanged; a new test proves REDIS_HOST is honored; an import test proves the base install imports no sqlalchemy/asyncpg/redis.
• Status (2026-10-09): done, with three items deferred to Phase 2 so that no setting ships before the code that reads it: DATABASE_URL/fleet_* settings, the Postgres/SQLAlchemy/Alembic packages in [fleet] (which holds only redis for now), and docker-compose.fleet.yml (the existing docker-compose.yml already runs Redis). The import test blocks the fleet packages instead of checking sys.modules, because a developer venv can carry redis transitively: an installed pydocket makes FastMCP import it, although the locked FastMCP does not depend on it.

Phase 1: Routing core

• Extend _make_tool_wrapper per §3.1: controller parameter, resolver, per-controller Settings LRU, and a controller_id ContextVar.
• New tools: list_controllers, select_controller, get_active_controller.
• Tier map and principal plumbing (§3.6): classify every tool, add DESTRUCTIVE_TOOLS, the tier snapshot test, and the Principal ContextVar. Point A2A’s permission check at the tier map. Every caller is still admin, so behavior doesn’t change yet.
• Wrapper-level audit (§3.7) with schema v2 and the attempt/outcome pattern, written to JsonlSink with the HMAC chain, plus unifi-mcp audit verify. Start migrating the manual audit calls.
• Record controller in every audit record and log line. Expose a per-controller call-count gauge only, with no controller label on the per-tool histogram (275 tools × 1,000 controllers would be a cardinality blow-up; see DEVELOPMENT_PLAN risk register).
• Verify: concurrency tests where two sessions with different selected controllers interleave 1,000 calls and every request hits its own controller (respx-mocked hosts). A mutating tool with only a registry default and ≥2 controllers fails with a clear error. Tool JSON schemas are snapshot-tested: only the added optional controller field changes. Every mutating tool produces exactly one audit record per call, including dry-run and error paths. Editing, deleting or reordering a JSONL line makes audit verify fail.
• Status (2026-10-09): done. Notes from implementation:
  • Each call writes an attempt and an outcome record sharing a call_id, so “one record per call” is one record pair; manual log_audit calls inside a call fold into the outcome’s details. The 167 manual calls remain and are harmless; removing them is cleanup, not a correctness need.
  • The verifier is python -m src.utils.audit_verify (matching audit_decrypt) rather than a unifi-mcp audit verify subcommand. Without UNIFI_AUDIT_CHAIN_KEY the chain uses plain SHA-256.
  • Tiers implemented: read, write, destructive (45 tools). fleet-admin arrives with the first admin tools in Phase 2a/2b.
  • Classifying tiers surfaced that A2A delegation treated restore_backup, upgrade_device and other mutating tools as reads with no confirmation; A2A now uses the registry tiers.
  • MCP resources still read the default controller.

Phase 2a: Postgres registry and encrypted credentials

• Alembic baseline migration for the §3.2 tables.
• PostgresRegistry with snapshot + registry:changed refresh. Credential encryption reusing audit_encryption primitives under UNIFI_FLEET_CREDENTIAL_KEY.
• Management tools (fleet-admin tier, confirm=True, dry-run aware): register_controller, update_controller, disable_controller, register_cloud_account, rotate_fleet_credentials (re-encrypt under the newest key).
• unifi-mcp fleet import CLI that seeds the registry from the current env config, for migration.
• Verify: migration up/down tests against a real Postgres (testcontainers, in CI when Actions is unblocked). No API key or plaintext credential ever appears in logs, audit, errors, or tool output (property test modeled on the PR #193 key-never-leaks tests). Registry outage → the last snapshot keeps serving reads, and registry writes fail loudly.
• Status (2026-10-09): done. Notes from implementation:
  • Tables created: credentials, cloud_accounts, controllers. sites, device_inventory, api_tokens and the audit tables come with the phases that use them (2b, 4), as the Phase 0 rule requires. controllers has no console_id, health or capabilities yet (Phase 4 discovery and health checks).
  • First start with an empty registry seeds the UNIFI_* controller as default (audited); python -m src.fleet.cli import registers it under another name. Migrations run only through python -m src.fleet.cli migrate; the server checks the revision at startup and refuses to run on an older schema.
  • The snapshot refreshes on an interval and right after this server’s own changes. Cross-replica registry:changed pub/sub is deferred to Phase 4 with the rest of the multi-replica work; until then other servers see a change within the refresh interval.
  • disable_controller is update_controller(enabled=false).
  • Tests run against a real Postgres 16 started by pgserver (a dev dependency), so CI needs no Docker. The docker-compose.fleet.yml image build was not exercised locally (Docker Hub pulls fail on the development machine).

Phase 2b: Access control and the audit store

• api_tokens, roles and enforcement (§3.6): create/list/revoke/update_api_token tools and the unifi-mcp tokens CLI, module filters, controller_selector, revocation via tokens:changed, PermissionDeniedError and denied records, optional per-session tool hiding.
• PostgresSink as primary audit store with the INSERT/SELECT-only role, audit_chain_heads, fail-closed write-ahead, and admin/config events.
• search_audit_log, verify_audit_chain, export_audit_log (fleet-admin only).
• Verify: a role × tier matrix test that runs every role against a sample tool of every tier, plus module and controller filters, and checks allowed calls succeed and denied ones neither reach the (mocked) controller nor skip the denied record. A revoked token stops working on every replica within 30 seconds. With the audit store down, mutating calls are refused and reads still work. The database role can’t UPDATE or DELETE audit_log (tested against real Postgres). No token secret appears in any log, audit record or tool output.
• Status (2026-10-10): done. Notes from implementation:
  • Decisions: break-glass callers (MCP_AUTH_TOKEN, stdio) are fleet-admin, not admin, so enforcement removes nothing they could do; UNIFI_STDIO_ROLE lowers stdio. Tokens default to a 90-day lifetime. With DATABASE_URL, audit records go to Postgres only.
  • Append-only is a trigger refusing UPDATE/DELETE/TRUNCATE for every role, plus an optional least-privilege runtime role (MIGRATION_DATABASE_URL for the schema owner; grants in SECURITY.md) rather than a required one. A tamper test bypasses the trigger as a superuser and the hash chain still catches the edit.
  • Revocation propagates through the verifier’s 30-second cache rather than tokens:changed pub/sub (Phase 4).
  • audit_log is not partitioned yet; partitioning and the archive-then-drop retention arrive together in Phase 5.
  • Optional per-session tool hiding is not implemented; enforcement does not depend on it.
  • Refusals of reads are audited too (the plan listed RBAC refusals without limiting them to writes).

Phase 3: Connection pool and caching on the hot path

• ClientPool with LRU/idle eviction and circuit breakers (§3.4). Auth/site-map caching (§3.3).
• Wrapper-level read-through cache for an allowlist of high-traffic read tools (devices, clients, sites, networks, WLANs), plus mutation-driven invalidation.
• Verify: benchmark harness with 500 simulated controllers on a mock server: p50/p95 tool latency and controller-side request count before vs. after (target: ≥80% fewer controller requests for repeated reads; no /ea/sites probe per call). A stale-read test confirms a mutation is visible on the next read.
• Status (2026-10-10): done. Notes from implementation:
  • Decisions: the read cache is active only with explicit Redis configuration; TTLs are the existing CacheConfig values; cached tools gain an optional fresh argument.
  • Benchmark (scripts/benchmark_fleet.py, 500 controllers × 5 reads, 20 ms simulated latency): baseline 5,000 controller requests and 2,500 auth probes; pool 3,000 and 500 (p50 43 → 22 ms); pool + cache 1,000 and 500, an 80% reduction. Latency with the cache is not representative there (fakeredis runs in the benchmark’s process).
  • The benchmark exposed that a pool smaller than the fleet gave no benefit for round-robin access, because evicting a connection dropped its auth state. Per-controller state (auth, rate limiter, breaker) now lives in a separate LRU of up to 10,000 controllers, independent of open connections.
  • Shared connections make UNIFI_RATE_LIMIT_REQUESTS effective for the first time (each call used to get a fresh limiter); recorded under Changed.
  • Invalidation clears all of a controller’s cached reads on any write instead of per resource family: simpler, and writes are rare next to reads.
  • The auth/site-map cache is in-process rather than in Redis, and single-flight is per process; both become cross-replica concerns in Phase 4.
  • Statistics and RF-health tools are not cached (live metrics). ProtectClient shares the pooled connection but keeps its own auth probe and has no breaker; SiteManagerClient (one cloud endpoint) is not pooled.

Phase 4: Horizontal scale

• Redis session_state_store (dropped, see the spike below), distributed rate limiting, pub/sub invalidation.
• Session affinity decision: spike done (2026-10-10). Two FastMCP 3.2 replicas sharing one state store, driven with real MCP JSON-RPC:

  Mode	Same replica	Other replica
  Stateful (default)	select_controller persists	404 Session not found: the transport session lives in one replica’s memory
  stateless_http=True	Lost on the next call	Lost

  FastMCP keys session state by the mcp-session-id header, and falls back to a fresh UUID when it is missing. A stateless server never issues one, so every request gets a new key. That rules out both options as written: a shared Redis store cannot carry a session across replicas (the session itself is gone, so the client re-initializes with a new id), and Option B would break select_controller even on one replica. Decision: sticky routing per client is the supported multi-replica mode (MULTI_CONTROLLER.md has a load-balancer example). MCP_STATELESS_HTTP=true runs without sessions so any replica can answer any request; there select_controller is refused with a clear error and calls pass controller=<name> (or use the registry default). The Redis session_state_store item is dropped.
• Done in 4a:
  • MCP_STATELESS_HTTP; the session guard (issue #173) is skipped in that mode, since it would refuse every sessionless call.
  • Per-controller rate limits shared through Redis: a sliding window in a MULTI transaction, one window per API key for api.ui.com, and a local fallback when Redis is down.
  • registry:changed pub/sub, so every replica reloads its snapshot right after a change.
• Done in 4b (worker tier, §3.5):
  • python -m src.worker (and a worker service in docker-compose.fleet.yml). Migration 0004 adds sites, device_inventory and the health, health_detail, health_checked_at, last_seen_at and capabilities columns of controllers.
  • Sharding: workers keep a heartbeat in a Redis sorted set and assign each controller by rendezvous hashing (highest hash(worker, controller)), which moves only the leaving or joining worker’s share. A worker works only on controllers whose SET NX PX lease it holds; a dead worker’s controllers move after the lease (UNIFI_FLEET_WORKER_LEASE_SECONDS), and a clean shutdown releases them at once. Without Redis one worker owns everything.
  • Jobs: health (authentication probe plus the Network version on local controllers; degraded on a failure, down after UNIFI_FLEET_WORKER_DOWN_AFTER in a row), site sync for every controller, device inventory per site for local controllers. First runs are offset by a hash of the controller name across the interval, later ones jittered ±10%, with at most UNIFI_FLEET_WORKER_CONCURRENCY jobs at once. Jobs go through the pooled UniFiClient, so with Redis the worker shares each controller’s rate limit with the API replicas.
  • Decisions and gaps: rendezvous hashing instead of a hash ring (same properties, no ring to maintain). Lease renewal is a pipelined GET then PEXPIRE (no Lua, as in 4a); if a lease expires between the two, at worst one job runs twice, and every job is an idempotent snapshot replace. Cloud controllers get sites only, because the cloud APIs list devices per host rather than per site. Cloud-account discovery is deferred: cloud controllers route by API key, not by console, so a discovered console would have nothing to route to until per-console routing exists. API replicas do not read the snapshots yet (the per-client site map stays, and health does not close circuit breakers); both are left for Phase 5 (see 4c below).
• Done in 4c (fleet-wide reads, §3.4):
  • fleet_health, fleet_sites and fleet_inventory answer from the worker’s snapshots (one query, no controller calls, ages on every entry; health older than three intervals is stale). fleet_inventory filters by model, state and firmware_below (numeric version comparison).
  • fleet_fanout runs one read-tier tool live on many controllers with UNIFI_FLEET_FANOUT_CONCURRENCY, a per-controller timeout and an overall deadline, returning {ok, data} or {ok: false, error} per controller. Each per-controller call goes through FastMCP and the tool wrapper, so role, module filters, controller scope and the read cache apply as for a direct call; write, destructive and fleet_* tools are refused.
  • All four are read tier in module fleet, see only controllers in the caller’s scope, and are registered with DATABASE_URL.
  • docker-compose.fleet-scale.yml (Redis, three replicas behind nginx hashing on client address and bearer token, two workers) and make verify-fleet-scale.
  • Not done: snapshot data does not yet replace the per-client site map on API replicas, and worker health does not close replicas’ circuit breakers; both are optimizations rather than correctness gaps, and stay open for Phase 5.
• Verify: a 3-replica + 2-worker compose test. Killing a worker leads to its shard being reassigned within lease TTL + 1 interval. Rate-limit test: 3 replicas combined never exceed one controller’s budget. Fan-out over 1,000 mocked controllers with 5% failing returns partial results within the configured deadline.
• Status (2026-10-11): done. Docker cannot run on the development machine, so the compose file was validated with docker compose config only, and the topology was verified with real processes instead (scripts/verify_fleet_scale.py: three python -m src replicas and two python -m src.worker processes, Postgres from pgserver, Redis from fakeredis over TCP, an HTTPS mock answering as 40 controllers, two of them unreachable). Results, the same over four runs of the final script:
  • The workers split the 40 controllers (17/23 to 21/19); after SIGKILL of one, the survivor held all its controllers 3.1–3.9 s later (lease 3 s, tick up to 5 s), and every controller was health-checked again within 10 s.
  • A controller registered on one replica was listed by the other two within 0.26 s with the periodic refresh at 300 s.
  • Three replicas calling one controller 18 times together, plus its worker, never put more than 5 requests in any 1 s window at a limit of 5/s (measured on arrival, with 50 ms allowed for network jitter).
  • fleet_fanout over 41 live controllers returned 39 results and 2 errors in 0.5–0.7 s; the unit test does the same over 1,000 controllers with 5% failing and one hanging, inside the deadline.
  • fleet_health and fleet_inventory returned the synced snapshots through a replica.

Phase 5: Hardening and documentation

• SIEM export (syslog/OTLP/webhook) with the on-disk retry buffer, and chain-head anchoring to the SIEM. Retention (UNIFI_AUDIT_RETENTION_DAYS): verify, archive, then drop.
• Security drills: an RBAC deny-path review of every tool tier, a tamper test that edits, deletes and truncates records in both stores and expects verify_audit_chain to catch each one, and a SIEM-outage test that confirms tool calls aren’t blocked.
• Degradation drills: Redis down → cache bypass, local rate limiter at budget / expected_replicas, explicit controller required (no session default). Postgres down → snapshot reads only.
• Update MULTI_CONTROLLER.md (registry now real), SECURITY.md (roles, tokens, audit integrity), README configuration, .env.example, API.md (new tools), DEVELOPMENT_PLAN G3 status.
• Verify: chaos tests in the compose environment; docs reviewed against the implemented env vars.
• Done in 5a (drills):
  • RBAC deny-path drill (tests/unit/fleet/test_drill_rbac.py): every one of the 364 tools registered in local mode with the fleet tools (212 read, 93 write, 45 destructive, 11 fleet-admin; the 3 session tools check scope themselves) is called as every role lacking its tier: 149 refusals for viewer, 56 for operator, 11 for admin. Each is refused before any network call and leaves a denied audit record. Nothing escaped.
  • Tamper drill (test_drill_tamper.py): edits, deletions, reordering, deleting the newest records, a whole chain and the whole table, as the superuser with triggers disabled. Found: Postgres verification did not compare chains with audit_chain_heads, so deleting the newest records or a whole chain verified clean. Fixed: verify_audit_chain checks each chain ends exactly at its recorded head (rows and heads read in one snapshot), and audit_verify --head CHAIN:SEQ:HASH does the same for files with an anchor kept elsewhere. Review added two more: a chain whose head row was deleted is reported, and each row’s searchable columns (which search_audit_log filters on) are checked against the sealed record. Rewriting records and heads together still needs an external anchor (5b), and the drill documents that limit.
  • Degradation drills (test_drill_degradation.py, plus the existing outage tests): Redis down falls back to a local rate limit of budget / UNIFI_FLEET_EXPECTED_REPLICAS (new; previously each replica kept the full budget, so N replicas sent N times it; one fallback bucket per controller per process, so clients built outside the pool share it; count workers too, as docker-compose.fleet-scale.yml does with 5), the cache is bypassed and workers pause rather than double-working. Postgres down: the snapshot tools, audit queries and worker fail or retry without echoing database credentials, and the worker keeps going.
  • Not applicable: “explicit controller required (no session default)” when Redis is down. That item assumed sessions in Redis, which the Phase 4 spike dropped; sessions live in each replica’s memory and survive a Redis outage.
• Done in 5b (SIEM export and anchors):
  • src/utils/siem.py: syslog (RFC 5424 over TLS, RFC 5425 framing), HMAC-signed HTTPS webhook and OTLP/HTTP JSON logs, all on the existing httpx/stdlib with no new dependency. UNIFI_AUDIT_SIEM_TARGETS takes several at once. Plain-text transports are refused (webhook) or warned about (syslog, OTLP) unless the host is loopback.
  • Each target has its own on-disk buffer (0700/0600; the directory name is a hash, so no URL lands on disk) of sealed segments delivered in order with backoff. Records are exported only after they are stored, in the stored (encrypted-payload) form, so the SIEM’s copy verifies with the chain key alone. forward() never raises.
  • Anchors: every UNIFI_AUDIT_SIEM_ANCHOR_SECONDS and on shutdown, an anchor event per chain that moved. audit_verify --anchors FILE takes the anchors exported from the SIEM. This closes the 5a limit: the tamper drill now rewrites records and heads together in Postgres, verifies clean on the server, and is caught against the SIEM anchor.
  • SIEM-outage drill (tests/unit/utils/test_siem.py): with the target down, 50 audited calls each cost under 10 ms in the audit path, everything is delivered in order once it comes back, the buffer survives a restart, the cap drops the oldest and counts them, and shutdown while down keeps the buffer.
  • Not done here: chain-head writes to audit_chain_heads stay per-record (5a); anchored_at is still unused, since anchors are tracked by the SIEM rather than the database.
• Done in 5c, part 1 (Postgres retention):
  • python -m src.fleet.cli audit-retention (dry run by default, --confirm to act) with UNIFI_AUDIT_RETENTION_DAYS and UNIFI_AUDIT_ARCHIVE_DIR. It verifies the whole store, verifies the expired records again inside the deleting transaction, writes them to a gzip JSON-lines archive (0600, in a 0700 directory) plus a manifest of each chain’s first and last record and the archive’s SHA-256, fsyncs both and their directory, and only then deletes. Only each chain’s expired prefix is read. The run’s admin event is written in the deleting transaction, so there is no delete without it. A failed delete removes the archive again, unless the database cannot confirm the transaction rolled back (a lost COMMIT reply), in which case the archive is kept. UNIFI_READ_ONLY refuses --confirm, and the job needs no controller credentials. Downgrading below migration 0005 is refused once audit_archives holds checkpoints.
  • Each chain loses a prefix only: everything from its first not-yet-expired record stays, so the rest still links (a record that looks old after a newer one, from clock skew, stays too).
  • New append-only audit_archives table records where each chain was cut. verify_audit_chain resumes each chain right after its latest archive, reports gaps between successive archives, and accepts a fully archived chain whose head is its last archived record (a fully archived chain whose head row was deleted is reported). Deleting archive rows to hide records is caught.
  • audit_verify ARCHIVE.jsonl.gz --manifest ARCHIVE.manifest.json checks an archive on its own: its SHA-256, its record count, that every chain in it is in the manifest, and that each runs exactly from its recorded start to its end. A truncated or corrupt archive fails the check.
  • The append-only trigger now allows one thing: a DELETE from audit_log by the table’s owner in a transaction that set unifi_mcp.audit_retention. Other roles, even with a DELETE grant, and TRUNCATE are still refused; the least-privilege runtime role needs only SELECT on audit_archives.
  • Deviation from the plan: no monthly partitions. Partitioning audit_log would need the partition key (ts) in every unique constraint, which would drop the database’s own guarantee that (chain_id, seq) and event_id are unique. Retention deletes in batches of 1,000 instead, run by an operator or a scheduled job as the schema owner; the server never deletes audit records.
• Done in 5c, part 2 (JSONL rotation and retention):
  • UNIFI_AUDIT_LOG_ROTATE_MB: once the audit file reaches that size the writer renames it to <log>.<UTC time>Z (microseconds, with a -N suffix on a clash, so names sort oldest first) under the same lock that orders the chain. The chain carries on in the next file. A new, empty file is created at once. If the rename fails (for example a file bind mount in Docker), rotation turns itself off with one error until restart. Only files whose whole suffix is a rotation timestamp count as rotated. get_recent_operations reads back through rotated files.
  • audit_verify takes several files as one log (audit.log.[0-9]* audit.log), and reads .gz archives.
  • python -m src.utils.audit_retention LOG (dry run unless --confirm): verifies all files, then takes the leading rotated files whose every record, and the file’s modification time, are older than the cutoff. It never touches the file being written, so it is safe while the server runs. It writes one archive with the same manifest as the Postgres job (shared src/utils/audit_archive.py), appends one line per run to <log>.checkpoints.jsonl holding every chain’s last archived record (a single append, fsynced, and the directory fsynced when the file is new, so a crash leaves at most a torn last line, which is ignored), and only then deletes the files. A run stopped between the checkpoints and the delete is finished by the next one, once that run’s archive is confirmed present and unchanged. A delete that fails is reported rather than ignored. Records are streamed into the archive, so memory use does not grow with the log. audit_verify ... --checkpoints FILE resumes chains there.
  • The run is an audited admin event in the log itself; UNIFI_READ_ONLY refuses --confirm.
  • Limit: someone who can rewrite the files can also rewrite the checkpoint file, so truncating the oldest records and moving the checkpoint forward is caught only by a copy kept elsewhere (the SIEM, or the archives on separate storage); the same holds for audit_archives in Postgres.
• Done in 5d (Phase 4 leftovers and docs):
  • Worker health half-opens replicas’ circuit breakers: after each successful check the worker publishes the controller’s connection key (a hash, no credential) and how long the check took on unifi:health:v1:up (HealthBus, src/fleet/bus.py). Each replica dates the check on its own monotonic clock (arrival minus duration; pub/sub delay only makes it look later), so no two hosts’ clocks are compared, and moves an open breaker for that controller to half-open if it opened before the check started: calls resume at once, one more failure re-opens it. Without Redis the cooldown applies as before; a worker that cannot reach Redis warns once per outage, not per controller.
  • Not done, by decision: replacing the per-client site map with the sites snapshot. The pool already caches the authentication probe and site map per controller for 5 minutes, and the probe is the /ea/sites request the map is built from, so reading the snapshot would save no request.
  • Still deferred: cloud-account discovery, for the reason in 4b (no per-console routing).
  • Docs reviewed against the settings: a test now requires every setting in API.md and .env.example, and that neither names a setting that does not exist. That removed DRY_RUN (the setting is UNIFI_DRY_RUN), UNIFI_CONTROLLERS and UNIFI_WEBHOOK_REDIS_URL (never implemented) and MCP_LOG_LEVEL (the setting is LOG_LEVEL), and documented the 10 settings that were missing (7 from API.md, 6 from .env.example). UNIFI_CACHE_TTL is documented as unused: read-cache lifetimes are fixed per resource type.

5. Risks and open questions

6. Out of scope (for this plan)

• Multi-tenant isolation (row-level security, per-tenant quotas). The schema doesn’t preclude adding tenant_id later, but nothing here builds it.
• External secret stores (Vault, AWS Secrets Manager). Credentials live encrypted in Postgres, per the decision above.
• A web UI for the registry. Management is via MCP tools and the CLI.
• OIDC/JWT identities from an external IdP. The Principal abstraction leaves room for another token verifier later, but this plan only issues its own tokens.
• Auditing read calls, and per-tool (rather than per-tier and per-module) permission lists.