Migrate production compute from Linode to OVH (starting with services) #187
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why now (2026-10-15): the
serviceshost (Linode, 1 vCPU, 960 MB RAM, no swap, root disk 78% of 25 GB, 15 containers) has no headroom, and that blocks another project's artifact-build caching (ci_cache_hostruns on it). The end-state (decided 2026-09-02, amended 2026-10-15) is production on OVH with Linode kept only as the dormant DR provider;docs/provider-portability.mdstill says "nothing on Linode" and needs the same amendment.Already in place: OVH compute module validated against a real
ovh-stagingbox (BHS5);ovh-drworkspace and the Linode -> OVH procedure indocs/runbooks/datacenter-failover.md; OVH S3 is the secondary restic repo; Cloudflare/deSEC DNS records exist in parallel (the registrar NS cutover is a separate track, blocked on email forwarding, so compute can move first with the existing Cloudflare A records).Phases (each ends with a check; nothing is switched until the one before it is verified):
openstack flavor list); OVH credentials and key pair present; a drill of the restore from the secondary repo only (the runbook's own acceptance criterion).--no-ansible), sized from the decision above../manage backup restore), then deploy the stack (./setup --workspace <ovh> --no-terraform).curl --resolveagainst the OVH IP; spot-check data (Forgejo, Vaultwarden, LLDAP, Authelia, Headscale).ovh-drcannot protect it. Build alinode-drworkspace (is_drinterraform/locals.tfcovers both, a DR/production split in the Linode module like OVH's,dr.tfvars); rewritedocs/runbooks/datacenter-failover.mdfor OVH -> Linode with its drill gate; flip backups (OVH Object Storage primary, Linode bucket the dual-write secondary; DR must restore from the secondary alone); keep Terraform state on Linode (the non-production provider, so it stays reachable in an OVH outage; this replaces the earlier "move state to OVH"); keep the Linode token (DR provisioning) on its normal rotation; move the staging workspaces and the runner (d2-2) to OVH.web->g6-standard-1(1 vCPU, 2 GB, 50 GB; equalsd2-2),services->g6-standard-2(2 vCPU, 4 GB, 80 GB; equalsd2-4). Needs a per-node override in the Linode module likeovh_node_flavors(todayinstance_typeapplies to every node:docs/runbooks/resize-node.md), thenlinode/dr.example.tfvarsset to those plans instead of the placeholder. Verify with a drill thatservicesreally runs at 4 GB (it is billed only while the DR nodes exist).Open points to settle in phase 0: per-host instance sizes (the Terraform
instance_typeis one variable for every node innode_labels; seedocs/runbooks/resize-node.md), downtime window for the stateful services, tailnet/Headscale re-enrolment of the new node,ci_cache_hostplacement.Phase 0 decisions, 2026-10-15 (operator: "use recommended and use 2gb for runner"). BHS5 flavors listed from the OVH API (names carry the RAM in GB):
servicesd2-4ci_cache_hostwebd2-2d2-2g6-standard-1Wiring:
instance_type = "d2-2"plusovh_node_flavors = { services = "d2-4" }(#188 adds the per-node override; default unchanged). The runner'sci_ephemeral_provisioner_instance_typeand_tf_workspace/_regionmust change together (pairing rule inroles/ci_ephemeral_provisioner/defaults/main.yml), at the runner cutover, not before.Also found:
scripts/secrets/rotate-ovh-s3-credentials.sh/init-secrets.shfailed withNo module named 'yaml'for the Homebrew python3 (#189 fixes it).Next: merge #188 and #189, then phase 1 (provision with terraform only, no Ansible; real, billed OVH resources, so it waits for an explicit go).
What still points at Linode after the compute cutover (add to phase 5): the nodes are only part of it.
RESTIC_REPOSITORY(primary) is Linode Object Storage;RESTIC_SECONDARY_REPOSITORY(dual-write) is OVH. Keep the dual-write running through the rollback window, so the fenced Linode side can still restore a current backup. At retirement: make the OVH repo the primary, drop the Linode one (or keep it read-only for the retention period), and check one restore from OVH alone (the failover runbook's acceptance criterion).blizzard.jfraeys.com,infra/terraform.tfstate) is on Linode Object Storage. Move it to OVH S3 withtofu init -migrate-statebefore the Linode token is revoked; updateTF_VAR_tf_state_*and the CI state key (OBJECT_STORAGE_*,./infra rotate ci-state-key).linode-stagingandlinode-ci-stagingworkspaces andLINODE_TOKENmove toovh-staging/an OVH ephemeral workspace (runner =d2-2), switchingci_ephemeral_provisioner_instance_type,_tf_workspaceand_regiontogether.TF_VAR_linode_token, delete the Linode secrets from the vault/Forgejo/manifest.Correction to my earlier comment (2026-10-15): the Terraform state backend stays on Linode Object Storage (DR side), it is not moved to OVH, and Linode is not fully retired or its token revoked. The phase list above (phases 5 and 6) now reflects the DR decision.
Gate added 2026-10-15: do not destroy the old Linode production nodes (phase 5) until the DR drill has passed. Until then they are the rollback path (unfence, restore the delta, switch DNS back). The drill is
docs/runbooks/datacenter-failover-ovh-to-linode.mdsteps 3 to 6 with a throwaway DNS name, restoring a freshlinode-drpair from the Linode repo alone, then./setup --workspace linode-dr --confirm-prod -- destroy. Run it right after the OVH cutover, inside the rollback window; record the date, restore time and any failure.Merged for phase 6:
linode-drTerraform (#190), the OVH -> Linode runbook andinventory/dr_linode.example.yml(#195), per-node OVH flavors (#188). Still to do for phase 6: the backup flip (OVH primary, Linode secondary) with a restore check, and the drill itself.Also merged: #194 (stop a still-running runner before reusing its VM after a cancelled job, fixes #192): needs
./infra deploy --tags ci_ephemeral_provisioner --limit services.Added to phase 6 (operator request): when the DR fallback happens, the Linode DR nodes must be the equivalent of the OVH production nodes, not the smaller default. web -> g6-standard-1 (1 vCPU / 2 GB / 50 GB, $12/mo) = d2-2; services -> g6-standard-2 (2 vCPU / 4 GB / 80 GB, $24/mo) = d2-4 (plan names and prices from Linode's public types list; billed only while the DR nodes exist). Prerequisite: a per-node type override in the Linode module (mirror of ovh_node_flavors, merged in #188); the current linode-dr (#190) has a single instance_type for both nodes.
Update 2026-10-16: OVH phase 1 applied (services d2-4, web d2-2, BHS5). Phase 2 rehearsal is next and is run by the operator: docs/runbooks/provider-migration-ovh.md (#217, #218). Findings: restore the full-host snapshot (tags infra,services), not 'latest' (the 6-hourly vaultwarden-only one); deploy with --skip-tags backups,ci_ephemeral_provisioner,ci_ephemeral_wakeup,ci_cache_host so the new node does not act as production; first boot lost the dpkg lock to unattended-upgrades (ufw/fail2ban/auditd missing; fixed for new nodes in #215/#216, Ansible installs them on these two). Decisions: DR stays on Linode (not Hetzner); switch to monthly billing only after the DR drill (CAD 27.45 vs 36.28 per month). Cutover checklist has two undecided items: Headscale re-enrollment and the backup flip.
Status 2026-10-17. Rehearsal (Phase 2) done on both nodes. ovh-services: full-host restore (snapshot f3e5d939), 15 containers healthy, Forgejo 16.0.6 matching production, repo count and newest repo match, tailnet-enrolled. ovh-web: play ok=110, firewall matches production web, fail2ban fixed (#239: it started before auth.log existed). Not done: firewall tag on ovh-services (opens its direct 443 to match the old node, see #233); DR drill; cutover.
Things the rehearsal changed in the plan (all in docs/runbooks/provider-migration-ovh.md): restore the full-host snapshot, never 'latest' (and manage backup restore now validates this first, #226); mint a tailnet pre-auth key per play with --quiet and pass it with -e; scripts/secrets/with-vault.sh --env (traefik reads TF_VAR_cloudflare_api_token from the environment); unset ANSIBLE_PRIVATE_KEY_FILE afterwards; names: services/web are the primary nodes, prefixes mark non-primary, the cutover swaps them; Traefik now reaches web over the tailnet, so it follows the node named web (#243/#244); delete the 6 stale Headscale nodes before the name swap (list in the access tracking issue #251). Open decisions: Headscale name swap details (rename syntax unverified) and the backup flip.
Status 2026-10-17 (evening). Since the last update: access hardening is complete and deployed (#251), Traefik reaches web over the tailnet (#244, verified), fail2ban native role fixed (#239), health check and manage fixes merged (#249, #226, #224), runbook updated with the naming rule and the key/env steps.
Completed: OVH phase 1 (nodes created); Phase 2 rehearsal on both nodes (services restore + deploy, web deploy); firewall behaviour verified on the old node (#233); runbook for the whole procedure.
In progress / not started: (1) firewall tag on the OVH rehearsal node (opens its direct 443 to match the old node); (2) delete 6 stale Headscale nodes; (3) decide the Headscale name swap and the backup flip; (4) cutover; (5) DR drill on Linode (old Linode nodes must not be destroyed before it passes); (6) monthly billing after the drill.
Status 2026-10-16 (baseline plan):
services(d2-4) andweb(d2-2) exist in BHS5.docs/runbooks/provider-migration-ovh.md. The runbook now lists the CI pause window during the freeze and a check that the forge-agent release secrets survived the restore (#258).linode/dr.tfvarswith webg6-standard-1/ servicesg6-standard-2,ovh/dr.tfvarsservicesd2-4). Left unticked until a drill shows services running at 4 GB.serviceshost after cutover without touching Linode or staging.Status 2026-10-17: merged to main since the last note: #260 (Linode prod stays on nanodes until after the migration), #261 (container memory limits are role variables), #262, #265 (shared output lib; manage_cli now also ships lib/output.sh to hosts), #259 (vpn revoke refuses ambiguous names, --delete for stale nodes). Open: #267 (cancel superseded PR CI runs). Waiting on the operator: staging on
3cf5acb-> infra approve -> deploy ci_ephemeral_provisioner,ci_cache_host (CI authkey expires 2026-10-20) and manage_cli; then the OVH rehearsal, cutover, DR drill. Next on the agent side: restic primary/secondary flip prep (cutover step 9).Status 2026-10-17 (evening): merged to main since the last note: #265 (shared output lib; manage_cli ships lib/output.sh), #267 (cancel superseded PR runs), #268 (backups_primary_provider / backups_secondary_provider: the cutover restic flip is now host_vars + repo-URL swap), #269 (OBJECT_STORAGE_* -> LINODE_OBJECT_STORAGE_*: vault, staging file and Forgejo secrets renamed; verified by staging-validate on
3852fa9), #271 (staging ntfy topic), #272 (backup status uses the newest full-host snapshot). Operator deployed ci_ephemeral_provisioner,ci_cache_host and manage_cli on the approved commit. Open: OVH rehearsal (Phase 2), cutover (Phase 3), DR drill (Phase 4). Next on the agent side: second CI runner label and a prebaked OVH runner image.2026-10-18: the services disk (25 GB) hit 100%; follow-ups, including revisiting the CI cache weekly reset (#284) after the OVH move or a disk resize, are tracked in #285. Merged since the last note: #280 (runners resolve Forgejo's public name over the tailnet, tag:ci gains :443), #282 (Forgejo Actions API gets its own Traefik router and a higher rate limit); both deployed or being rolled out by the operator.