jfraeys/coaching-site

Watch

Fork 0

Release A step 4: deployment state, blue/green activation, publishing rewrite #7

New issue

Open

opened 2026-10-16 00:18:49 -04:00 by jfraeys · 2 comments

jfraeys commented

2026-10-16 00:18:49 -04:00

Owner

Full spec: docs/separation-plan.md on branch site-separation, section "4. Deployment and Ansible" + "Revision and deployment model". This issue is Release A, step 4 — the deployment-state/activation/publishing half of the site-separation plan. Steps 2–3 (site contract, extraction, compatibility CI) are already done (see the plan's own Status Log). This step is NOT done and is a precondition for cutting over production to the site-separation model.

Scope, in the plan's own implementation order:

Runtime-input audit first (docs/runtime-inputs.md already has the audit; the "apply on publish" decision for booking hours/email templates/sender name is already made — 2026-10-15). Remaining: actually move those runtime reads off the live checkout onto a release snapshot (see below).
Candidate checkouts, immutable build jobs, attempt journal, recovery. Never build in the live engine checkout — each attempt builds from an immutable candidate engine checkout (git worktree at the target SHA) and an immutable site checkout at the resolved SHA.
Release directories + blue/green activation, replacing deployswap's in-place binary swap:
```
/opt/coach-site/<site>/releases/<attempt-id>/
  bin/coach-api
  public/
  site/              # read-only runtime snapshot: data/*.yaml, language list
  nginx.conf
  provenance.json
/opt/coach-site/<site>/current -> releases/<attempt-id>
```
Two slots (coach-api@<site>-a/-b), one atomic current pointer switch + nginx reload for both HTML and API together. The invariant: the live deployment is always either the previous healthy pair or the new healthy pair, never a mix.
Publishing rewrite: isolated-worktree commit procedure (never commit in the shared checkout's index), publication journal + audit refs (refs/publish-attempts/<attempt-id>), idempotent attempt IDs, forward-only site-history policy, separate "publish accepted" vs. "deployment succeeded/failed" outcomes reported to the admin.
Per-site locking: one flock-based lock per site instance covering the whole attempt (revision selection → activation → manifest update), replacing today's in-process rebuildMu (which doesn't cover deploy.sh run by coach-deploy.service).
Release gate: make release target + a cd.yml first step that both refuse to tag/deploy a commit without a passing cross-site compatibility result bound to that exact SHA.
Build isolation / per-job sandbox: production builds run as transient systemd-run units (DynamicUser=yes, minimal root, no secrets, no network, output as a tar stream over stdout) — treat site/client/layouts as potentially hostile code. Step-0 prototypes already proved the sandbox/blue-green/publication-recovery guarantees on a Debian 12/systemd 252 VM (2026-10-14, see tools/prototypes/README.md); rerun on the actual production host with the pinned Hugo version before cutover.

Production-readiness gates (from the plan, must pass before cutover — not new scope, just the acceptance criteria):

Hostile-layout builds demonstrably can't reach secrets, other sites, or coordinator state.
Publication recovery reconciles the remote branch against the intended commit before any retry/failure report.
Activation crash tests show recorded state always reconciles with what's actually being served (X-Release-Id probe + health checks).
Runtime-input audit traced through actual request paths, not just known files.
Publish commits the validated in-memory snapshot; deploy builds the recorded site commit from a candidate checkout — never a re-read of the mutable checkout.

Do not expand this issue's scope beyond what's above — the plan is explicit that Release A should not grow further once this lands.

Full spec: `docs/separation-plan.md` on branch `site-separation`, section "4. Deployment and Ansible" + "Revision and deployment model". This issue is **Release A, step 4** — the deployment-state/activation/publishing half of the site-separation plan. Steps 2–3 (site contract, extraction, compatibility CI) are already done (see the plan's own Status Log). This step is NOT done and is a precondition for cutting over production to the site-separation model. **Scope, in the plan's own implementation order:** 1. **Runtime-input audit first** (`docs/runtime-inputs.md` already has the audit; the "apply on publish" decision for booking hours/email templates/sender name is already made — 2026-10-15). Remaining: actually move those runtime reads off the live checkout onto a release snapshot (see below). 2. **Candidate checkouts, immutable build jobs, attempt journal, recovery.** Never build in the live engine checkout — each attempt builds from an immutable candidate engine checkout (`git worktree` at the target SHA) and an immutable site checkout at the resolved SHA. 3. **Release directories + blue/green activation**, replacing `deployswap`'s in-place binary swap: ``` /opt/coach-site/<site>/releases/<attempt-id>/ bin/coach-api public/ site/ # read-only runtime snapshot: data/*.yaml, language list nginx.conf provenance.json /opt/coach-site/<site>/current -> releases/<attempt-id> ``` Two slots (`coach-api@<site>-a`/`-b`), one atomic `current` pointer switch + nginx reload for both HTML and API together. The invariant: **the live deployment is always either the previous healthy pair or the new healthy pair, never a mix.** 4. **Publishing rewrite**: isolated-worktree commit procedure (never commit in the shared checkout's index), publication journal + audit refs (`refs/publish-attempts/<attempt-id>`), idempotent attempt IDs, forward-only site-history policy, separate "publish accepted" vs. "deployment succeeded/failed" outcomes reported to the admin. 5. **Per-site locking**: one `flock`-based lock per site instance covering the whole attempt (revision selection → activation → manifest update), replacing today's in-process `rebuildMu` (which doesn't cover `deploy.sh` run by `coach-deploy.service`). 6. **Release gate**: `make release` target + a `cd.yml` first step that both refuse to tag/deploy a commit without a passing cross-site compatibility result bound to that exact SHA. 7. **Build isolation / per-job sandbox**: production builds run as transient `systemd-run` units (`DynamicUser=yes`, minimal root, no secrets, no network, output as a tar stream over stdout) — treat `site/client/layouts` as potentially hostile code. Step-0 prototypes already proved the sandbox/blue-green/publication-recovery guarantees on a Debian 12/systemd 252 VM (2026-10-14, see `tools/prototypes/README.md`); rerun on the actual production host with the pinned Hugo version before cutover. **Production-readiness gates** (from the plan, must pass before cutover — not new scope, just the acceptance criteria): - Hostile-layout builds demonstrably can't reach secrets, other sites, or coordinator state. - Publication recovery reconciles the remote branch against the intended commit before any retry/failure report. - Activation crash tests show recorded state always reconciles with what's actually being served (`X-Release-Id` probe + health checks). - Runtime-input audit traced through actual request paths, not just known files. - Publish commits the validated in-memory snapshot; deploy builds the recorded site commit from a candidate checkout — never a re-read of the mutable checkout. Do not expand this issue's scope beyond what's above — the plan is explicit that Release A should not grow further once this lands.

jfraeys referenced this issue

2026-10-16 00:19:04 -04:00

Hosting model for client/demo sites: shared vs. dedicated nodes #1

jfraeys referenced this issue from a commit

2026-10-16 10:30:13 -04:00

feat(separation): settings apply on publish, not on admin Save

jfraeys commented

2026-10-16 10:30:25 -04:00

Author

Owner

Started. First slice landed: fa797a3 on site-separation — "apply on publish, not on admin Save" (the plan's finding 1 / already-decided 2026-10-15 item from docs/runtime-inputs.md). internal/locales.ReloadOverrides is now startup-only; HandleSaveContent calls a new pure ValidateOverrides for data/email-templates.yaml (validates, never applies), and the site.yaml sender-name live-apply call is removed entirely. Tests updated (TestHandleSaveContent_EmailTemplates_ValidContentSavesButDoesNotApplyLive replaces the old "...AppliesLive" test) and full suite (go build ./..., go test ./...) passes.

Remaining before this issue is done, in the plan's own order:

Candidate checkouts (git-worktree-based builds, never the live checkout).
Immutable build jobs + attempt journal + recovery.
Release directories + blue/green activation (the big structural piece — replaces deployswap's in-place swap).
Runtime snapshot wiring: move site.yaml/features.yaml/email-templates/language-list reads off ContentRoot onto the release's site/ snapshot (needs release directories to exist first).
Publishing rewrite (isolated-worktree commits, publication journal, forward-only site history).
Per-site locking, release gate, build-isolation sandbox.

Admin UI copy noting "applies on publish" (the plan also asks for this) not yet added — deliberately deferred rather than rushing unreviewed copy; needs i18n (en/fr parity, tests/layout_strings_test.go's no-hardcoded-strings check) done properly, not as an afterthought to this commit.

Started. First slice landed: `fa797a3` on `site-separation` — "apply on publish, not on admin Save" (the plan's finding 1 / already-decided 2026-10-15 item from `docs/runtime-inputs.md`). `internal/locales.ReloadOverrides` is now startup-only; `HandleSaveContent` calls a new pure `ValidateOverrides` for `data/email-templates.yaml` (validates, never applies), and the `site.yaml` sender-name live-apply call is removed entirely. Tests updated (`TestHandleSaveContent_EmailTemplates_ValidContentSavesButDoesNotApplyLive` replaces the old "...AppliesLive" test) and full suite (`go build ./...`, `go test ./...`) passes. Remaining before this issue is done, in the plan's own order: 1. Candidate checkouts (git-worktree-based builds, never the live checkout). 2. Immutable build jobs + attempt journal + recovery. 3. Release directories + blue/green activation (the big structural piece — replaces `deployswap`'s in-place swap). 4. Runtime snapshot wiring: move `site.yaml`/`features.yaml`/email-templates/language-list reads off `ContentRoot` onto the release's `site/` snapshot (needs release directories to exist first). 5. Publishing rewrite (isolated-worktree commits, publication journal, forward-only site history). 6. Per-site locking, release gate, build-isolation sandbox. Admin UI copy noting "applies on publish" (the plan also asks for this) not yet added — deliberately deferred rather than rushing unreviewed copy; needs i18n (en/fr parity, `tests/layout_strings_test.go`'s no-hardcoded-strings check) done properly, not as an afterthought to this commit.

jfraeys referenced this issue from a commit

2026-10-16 11:17:46 -04:00

feat(separation): candidate checkouts (git worktree + clone), isolated from the live checkout

jfraeys commented

2026-10-16 11:17:53 -04:00

Author

Owner

Candidate checkouts landed: 3b12511 on site-separation. New internal/deployment package — NewCandidateCheckout(buildsRoot, attemptID, engineRepoDir, engineSHA, siteRepoDir, siteSHA) builds one attempt's isolated engine (git worktree, no network) + site (local clone, real independent .git) pair, never touching either live checkout. Cleanup deregisters the worktree and removes the attempt dir, idempotent. 6 tests against real git repos (no mocks): exact-SHA checkout, live checkouts untouched, duplicate attempt IDs rejected, failure leaves no dangling worktree, Cleanup idempotent, two sites' builds roots independent. Full go build/vet/test ./... passes.

Deliberately NOT done here (next steps, in the plan's order): resolving a branch to a SHA (the coordinator's job, "site revision rule" — this package only ever takes already-resolved SHAs), immutable build jobs, the attempt journal, recovery, release directories + blue/green activation, runtime snapshot wiring, the publishing rewrite, per-site locking, the release gate, and the build-isolation sandbox.

Candidate checkouts landed: 3b12511 on site-separation. New internal/deployment package — NewCandidateCheckout(buildsRoot, attemptID, engineRepoDir, engineSHA, siteRepoDir, siteSHA) builds one attempt's isolated engine (git worktree, no network) + site (local clone, real independent .git) pair, never touching either live checkout. Cleanup deregisters the worktree and removes the attempt dir, idempotent. 6 tests against real git repos (no mocks): exact-SHA checkout, live checkouts untouched, duplicate attempt IDs rejected, failure leaves no dangling worktree, Cleanup idempotent, two sites' builds roots independent. Full go build/vet/test ./... passes. Deliberately NOT done here (next steps, in the plan's order): resolving a branch to a SHA (the coordinator's job, "site revision rule" — this package only ever takes already-resolved SHAs), immutable build jobs, the attempt journal, recovery, release directories + blue/green activation, runtime snapshot wiring, the publishing rewrite, per-site locking, the release gate, and the build-isolation sandbox.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jfraeys/coaching-site#7

No description provided.

Rows
Columns