Production hardening

Production is where implicit assumptions break: TLS trust, secret rotation, rate limits, and observability.

What you will learn

• TLS and CA trust patterns for Node clients
• Why gateway and DB limits matter on shared clusters
• Pointers to internal ops docs

The idea

• Prefer NODE_EXTRA_CA_CERTS (or system trust) over disabling TLS verification globally.
• On v2, dashboard mesh probes (FLUX_TENANT_PROBE_GATEWAY_URL) call the tenant API through the gateway with the public tenant Host. Dedicated projects are probed at their PostgREST container (http://flux-<hash>-<slug>-api:3000/) and then the public origin, on /, because dedicated PostgREST treats /health as a table lookup. Archived projects are not probed. That mesh probe is separate from gateway GET /health (liveness) and GET /health/deep (system-database readiness). See docs/OPERATOR-GATEWAY-HEALTH.md.
• Treat gateway signing keys like database superuser passwords: rotation plans, access logging, least privilege.

How it works

Review:

• docs/production-security-audit.md — audit framing
• docs/OPERATIONS.md — namespaced project rebuild runbook (Hetzner)
• docs/OPERATOR-GATEWAY-HEALTH.md — gateway GET /health (liveness) vs GET /health/deep (readiness)

Self-hosted operators — control plane: flux migrate runs pg_dump inside the dashboard/control-plane container against the shared cluster. That image must include PostgreSQL client tools on PATH inside the process that handles /api/cli/v1/migrate; restarting an old container without rebuilding leaves pg_dump missing at runtime. App builders on hosted Flux cannot fix this in their own repo—see hosted vs self-hosted notes under Pooled → dedicated migrate troubleshooting.

Backup storage and verification (self-hosted only)

Three environment variables shape where backups live and how flux backup verify is run. Hosted Flux manages these for you; on a self-hosted install the operator owns them:

• FLUX_BACKUPS_LOCAL_DIR — primary backup volume on the control plane host. Default /srv/flux/backups. Must be writable by the flux-web process (uid 1001 in the shipped image). Backups are stored as <projectId>/<backupId>.dump.
• FLUX_BACKUPS_OFFSITE_DIR — secondary directory the offsite replicator copies to when R2 is disabled. Default /srv/flux/backups-offsite. Useful for same-host replication during testing.
• FLUX_BACKUP_VERIFY_POSTGRES_IMAGE — image used for the disposable Postgres container during flux backup verify. Default postgres:17-alpine. Override only if your tenant Postgres major version differs and you need to align them.

Cloudflare R2 offsite replication (optional)

When FLUX_R2_BACKUPS_ENABLED=true, completed backups upload to R2 (S3-compatible) immediately after the local artifact is written. Object keys:

{FLUX_R2_BACKUP_PREFIX}/flux/v1/{projectHash}/{backupId}.dump   # v1 dedicated{FLUX_R2_BACKUP_PREFIX}/flux/v2/{tenantId}/{backupId}.dump      # v2 shared (catalog project UUID)

Set these in docker/web/.env next to compose (see docker/web/.env.example):

Trust model: R2 is an offsite replication layer only. Destructive gates (flux nuke, dashboard delete, etc.) still require a restore-verified local backup — offsite upload alone does not satisfy the gate.

Free-tier storage headroom (ops-audit): bin/ops-audit.sh always runs a cheap ListObjectsV2 sum against the configured backup bucket when FLUX_R2_BACKUPS_ENABLED=true (the whole bucket — FLUX_R2_BACKUP_PREFIX is not a filter). Example line: R2 storage: 80.0 MiB / 10 GiB free-tier (0.8%). Well under the WARN bar is OK (no email). Crossing WARN/FAIL uses the existing weekday ops-audit alert path (Resend when the run has FAIL/WARN). This does not change retention deletes, FLUX_R2_BACKUPS_STRICT, or ops-watch. Helper: bin/ops-audit/r2-usage.mjs (node bin/ops-audit/r2-usage.mjs --help). Account ids in R2 hostnames are redacted; credentials are never printed.

Platform minimum backup freshness (self-hosted)

Separate from the destructive gate, the hourly backup scheduler enforces a platform minimum backup freshness floor: the newest restore-verified backup should be within FLUX_MIN_BACKUP_INTERVAL_DAYS (default 7). When a project is stale, the scheduler runs create → validate → restore-verify (up to FLUX_MIN_BACKUP_MAX_PIPELINES_PER_TICK projects per tick; default 1 for production safety). Restore-verify stubs per-tenant roles named by the archive (t_<shortId>_role, t_<shortId>_ddl, and other non-reserved roles in policies, owners, grants, or default privileges) as NOLOGIN with no BYPASSRLS, then requires pg_restore to exit 0 with no ignored errors. The scheduler runs inside flux-web. After upgrading that image, re-check an existing artifact with flux backup verify -p <slug> --hash <hash> --latest (this does not write a new dump). The next scheduler tick creates a new backup only for projects that are still not platform-fresh; a restart runs one tick immediately, still capped by FLUX_MIN_BACKUP_MAX_PIPELINES_PER_TICK unless that process has never completed a tick.

Each hourly tick runs retention over every non-excluded project, in project-id order, including projects that did not get a new backup that hour. The sweep deletes restore-verified rows past the count and age window. It deletes failed or restore_failed rows only when a newer restore-verified backup exists, and it never deletes the newest backup. Unverified complete backups are not counted toward the floor of four. When a row is removed, Flux also deletes its offsite replica (R2 object, or the filesystem copy under FLUX_BACKUPS_OFFSITE_DIR). A missing remote object is non-fatal — the sweep logs and continues. Objects uploaded before this behavior shipped may still need a one-time bucket cleanup. KEEP floors (FLUX_MIN_BACKUP_RETENTION_COUNT / DAYS) are unchanged.

Freshness is visible in the dashboard Database tools panel and CLI backup list/create responses. It does not add HTTP 412 blocks beyond the existing restore-verified destructive gate.

The CLI runs on your laptop but backup I/O executes inside flux-web; R2 credentials belong on the control-plane host/container, not in app repos.

Both FLUX_BACKUPS_* paths must exist and be writable by the control-plane process before the first flux backup create. The shipped Docker compose mounts them as named volumes; if you run flux-web outside the canonical compose layout, chown the directories to uid 1001 or set the env vars to paths the process can already write to.

The user-facing trust contract (what backups guarantee, the three trust states) is engine-independent and lives in Backups. This section is purely about where bytes physically land on the operator's host.

Periodic audit (self-hosted): from the repo on your laptop or the server checkout, run bin/ops-audit.sh --remote (SSH defaults match bin/sync-env-remote.sh). Add --deep for backup-catalog trust rows and platform minimum backup freshness (restore-verified age vs FLUX_MIN_BACKUP_INTERVAL_DAYS); add --smoke to GET each tenant API through flux-node-gateway (see bin/ops-audit-smoke.projects.example). The scheduler restore-verifies stale projects automatically; --deep still warns when the newest restore-verified backup is missing or overdue. Schema-only empty v2 tenants (zero user tables) can be restore-verified — they are not treated as restore_failed once the tenant schema and empty dump TOC match. Every run also checks R2 free-tier storage headroom when offsite backups are enabled (WARN at 5 GiB / FAIL at 8 GiB of the 10 GiB Standard allowance; quiet when well under). For unattended paging (error-only), see Error-only Docker / host watcher and bin/ops-watch.sh.

Email alerts for scheduler / ops failures (optional)

Production currently logs backup-scheduler failures; it does not send mail unless you opt in. Set these on the control plane (docker/web/.env, then recreate flux-web).

Cloudflare Email Routing caveat: vsl-base.com inbound mail uses Cloudflare Email Routing. That product is receive-only (MX → destination mailbox). It does not provide SMTP or an outbound API. Pointing FLUX_SMTP_HOST at Cloudflare will not deliver alerts. Use a send provider.

Primary path — Resend HTTP API (SMTP not required):

Resend DNS (brief): In the Resend dashboard add domain vsl-base.com (or a dedicated send subdomain). Add the SPF + DKIM TXT records Resend shows in Cloudflare DNS. Do not replace the Email Routing MX records — those stay for inbound justin@vsl-base.com. Until the domain shows verified, keep FLUX_ALERT_EMAIL_FROM on onboarding@resend.dev. After verify, set From to alerts@vsl-base.com (or another address on the verified domain).

Fallback — generic SMTP (only if you have a real outbound relay; unused for sending when FLUX_RESEND_API_KEY is set):

If FLUX_ALERT_EMAIL_TO is unset, or neither Resend nor SMTP is configured, alerting is a no-op (one debug log).

The backup-scheduler emails when: a platform freshness pipeline fails (includes slug:hash and restore_failed / other errors), offsite replication fails (including hourly retries of a held-in-trust upload), artifact validation fails, retention sweep fails, or a scheduler tick hard-fails. Messages are short: project identity when known, error, UTC timestamp.

Error-only Docker / host watcher (optional)

The same Resend path can page on host/Docker problems — never on a healthy tick. Enable it on the control plane (docker/web/.env), then recreate flux-web. Host crontab is empty on the production box; this watcher is an in-process flux-web scheduler (same pattern as backup-scheduler / fleet-monitor) and shells docker via the socket already mounted in prod.

What it emails (fingerprint examples; same FLUX_ALERT_EMAIL_DEDUPE_HOURS, default 6h):

• Core / v2 stack missing, exited, unhealthy, or restarting — docker:exited:flux-web, docker:unhealthy:flux-postgres-v2, docker:restarting:flux-node-gateway, docker:missing:flux-gateway
• Unexpected tenant-stack exits (flux-<7hex>-*)
• Disk ≥ threshold — disk:root:94
• Core log matches — log:fatal:flux-web

Exited-tenant filter (intentional stops are silent): skip flux-backup-verify-*, flux-ops-watch-*, *-canary; skip exited tenants with no catalog row (orphan / leftover test — same class as ops-audit WARN); skip catalog lifecycle_state dormant / archived; skip catalog health_status=stopped (dashboard power-off / flux reap). Restarting / unhealthy still alert.

Host disk is sampled with a one-shot docker run --rm -v /:/host:ro using the running flux-web image, then df -P on /host, /host/srv, and /host/var/lib/docker only. A bare df walks /host/run/docker/netns/* and fails with Permission denied — that is treated as a tooling skip (logged, not emailed). /srv/apps is not checked — there is no control-plane convention for expected compose projects there.

Tooling failures (disk helper permission errors, docker logs timeout / exit 143) are not findings and do not page. Only real container/disk/log-match problems email Justin.

bin/ops-watch.sh is the host/SSH sibling (--remote, --json): error-only, same filters, does not send mail. Use it for a one-shot check. Do not also cron it while the flux-web tick is enabled or you will double-page after a 6h dedupe window (in-memory vs none). Its docker logs reads run under timeout -k (FLUX_OPS_WATCH_LOG_TIMEOUT_SECONDS, default 8, then SIGKILL after FLUX_OPS_WATCH_LOG_KILL_AFTER_SECONDS, default 5) so a stuck read cannot outlive the check. The fatal/panic patterns are word boundaries (\bfatal\b, \bpanic(ked)?\b).

Docker json-file rotation. flux-gateway and flux-node-gateway already cap logs at max-size: 20m, max-file: 5. The same caps are set on flux-web, flux-postgres-v2, flux-pgbouncer, and flux-postgrest-pool. Docker applies a logging change only when that container is recreated. ./bin/deploy-web.sh brings up only flux-web. docker compose up on docker/v2-shared/docker-compose.yml (./bin/deploy-v2-shared.sh, ./bin/restart-v2-shared.sh, ./bin/deploy-all.sh) recreates a service whose config changed, including flux-postgres-v2. The planned command for that Postgres recreate, the restore-verified backup pre-check, and the expected v2 outage are in docs/OPERATIONS.md.

Limitation: if flux-web itself is down, this tick cannot send. Dashboard down is the signal; ./bin/ops-watch.sh --remote from a laptop still works.

Setup

1. Put FLUX_ALERT_EMAIL_TO=justin@vsl-base.com and FLUX_RESEND_API_KEY=re_… in docker/web/.env (see docker/web/.env.example). SMTP vars are optional.
2. Recreate the control plane so flux-web reloads env: docker compose -f docker/web/docker-compose.yml up -d --force-recreate (or ./bin/deploy-web.sh / ./bin/launch-web.sh --sync-env-apply).
3. Confirm docker logs flux-web has no ops-alert-email: send failed after the next scheduler error.
4. To enable the host/Docker watcher, also set FLUX_OPS_WATCH_ENABLED=1 in the same .env, recreate flux-web, and confirm ops-watch: started in docker logs flux-web. A healthy tick is silent.

This is operator-only visibility. It does not change backup trust tiers or destructive gates.

Example

For multi-region or multi-cluster, document which Postgres cluster holds a tenant before running destructive maintenance.

Next steps

• Threat model
• Environment variables
• Pooled → dedicated migrate