terraform state: backup, dedicated bucket, second copy on OVH, encryption #220
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Review of how Terraform state is managed (2026-10-16). The basics are sound; the gaps are backup and blast radius.
Current
blizzard.jfraeys.com, keyinfra/terraform.tfstate, workspaces for environments.use_lockfile), verified live 2026-08-26;./infra unlockfor stuck locks.CI_STATE_BACKEND_*).prevent_destroyon production nodes.Gaps (most important first)
docs/runbooks/node-failure.md: "the repo has no separate backup of the state object, so snapshot it by hand." Bucket versioning: unknown, not checked.infra-backup). Linode keys scope to the bucket, not a prefix, so the CI staging key can reach prod state and the backups. Unverified: whether that key is read-write.Plan, in order
tofu init -migrate-state; update CI key provisioning (scripts/secrets/provision-ci-state-key.py).tofu state pullsnapshot into restic (both repos), with a check that alerts when it is stale.Related
#187 (OVH migration, DR drill: confirm the state copy is current when the drill runs),
docs/runbooks/node-failure.md.Design for step 3 (scheduled state snapshot): fetch the state objects (infra/terraform.tfstate and env://infra/terraform.tfstate) with curl --aws-sigv4 into /var/lib/infra-backups/terraform-state/ on the services host; the existing restic run already backs that directory up to both repos, so the primary and OVH/Linode secondary both get a copy with no new dependency. A failed fetch marks the backup DEGRADED. Roles change: staging, promotion, then ./infra deploy --tags backups (not on the new OVH node before cutover). Note for step 2: the backup credential and the state backend credential are currently the same Linode key.
Update: step 3 is done and verified. The state snapshot runs in every backup (6 workspace files, mode 0600, in both restic repos); the first deploy failed with 403 on Debian 12's curl 7.88 (missing x-amz-content-sha256) and was fixed in #223. Bucket versioning is off (checked read-only). Costs checked: negligible on both providers. Next: dedicated state bucket, OVH copy, encryption decision.
Status 2026-10-16: plan step 3 (scheduled
tofu state pullsnapshot into restic) is merged (#222, fix #223) and enabled bybackups_state_snapshot_enabled: trueininventory/host_vars/services.yml. It reaches production on the next--tags backupsdeploy and must not run on the new OVH node before cutover. Steps 1, 2, 4, 5, 6 are open; step 4 (second copy on OVH) waits for the OVH primary-repo flip in #187.Status 2026-10-17: backups can now pick the provider per repo (#268), which the planned second state copy on OVH (step 4) builds on. State snapshot into restic (step 3) is deployed as before; steps 1, 2, 5, 6 are still open.