Infra-Setup

Provisioning and shared conventions for this account's VMs: Ubuntu hosts running behind Caddy, with a shared OTel collector and a shared backup/restore system, provisioning driven by per-host manifests declaring one of two roles.

Two roles

Every VM this repo provisions plays one of two roles, declared in its own hosts/<hostname>.conf manifest (see "Host manifests, roles, and capabilities" below):

• primary — hosts a Forgejo instance and the control plane (platform-control-plane): node identity/registry, all Forgejo API access, the onboarding GUI, and per-app secrets. control_plane is forced on wherever forgejo is — the two can't be split (see "Host manifests, roles, and capabilities" below), and an Operations repo comes with it unconditionally (it's not a capability of its own — the control plane needs one regardless of whether the reconciler also runs locally). Also runs a Forgejo Actions runner (the runner capability) by default, and the always-on shared layer (Caddy, host-only OTel, the backup timer). The reconciler (and, with it, application-telemetry OTel) is optional here — see "augmenting a host" below.
• application — serves applications: Caddy, OTel (host and application telemetry), and the reconciler. A Forgejo Actions runner is optional here too (runner, opt-in, off by default) — runner registration is brokered by the control plane over the network (see "Forgejo Actions runner" below), so it no longer has to be colocated with Forgejo the way it once did.

Only one primary host may ever exist per domain — scripts/ lint-hosts.sh enforces this across every manifest in hosts/, refusing to proceed if two manifests both resolve forgejo=true (or control_plane=true) for the same FORGEJO_DOMAIN (or CONTROL_PLANE_PUBLIC_BASE_URL). Two manifests resolving either true for genuinely different domains — a disposable verification host alongside the real one, say — is not a conflict and is explicitly allowed; the actual risk being guarded against is two hosts racing for the same domain's Let's Encrypt certificate, not "more than one Forgejo anywhere in this repo."

A single VM can hold both roles at once: application's capabilities (reconciler, otel_app) are additive on top of primary's defaults, not a separate deployment — put both roles in ROLES= (e.g. ROLES=primary application, see hosts/Falkenstein-8-16.conf) rather than hand-picking individual CAPABILITY_* overrides, so a host meant to actually deploy applications also gets otel_app and not just reconciler. "Augmenting" a primary host later is just adding the application role to its manifest and re-running bootstrap-all.sh — nothing about primary's own Forgejo/control-plane/runner/Caddy/OTel/backup setup is touched or re-provisioned.

Two layers — don't conflate them

Platform bootstrap — the VM, Docker, the shared networks, Caddy, the otel-collector, the backup system, Forgejo itself, a Forgejo Actions runner, the control plane (node identity, Forgejo access, the onboarding GUI), and the reconciler. This is infrastructure-as-code in the classic sense: version-controlled config and scripts, applied by hand (scripts/bootstrap-all.sh, SSH) rather than a CI pipeline. No promotion pipeline, no image registry round-trip anywhere in this repo — even the reconciler's own service builds straight from the source checked out alongside it (docker compose up -d --build, see infra/platform-reconciler/docker-compose.yml), not from a pulled image. Changes here are high-blast-radius (host firewall, the shared proxy, the shared backup system) and low-frequency, so they stay human-gated on purpose — this layer should never auto-deploy on merge, no matter that a build: step now lives inside it.

scripts/bootstrap-all.sh <manifest> is the single entrypoint for all of it, from a bare Ubuntu VM to a fully working instance — see "Bootstrapping a VM from scratch" below. It's safe to run against any starting state, not just a from-scratch one: every script it calls checks real state before acting, so re-running it after a partial run (or one that predates some of these steps entirely) picks up from wherever things actually stand, and re-running it with an augmented manifest just adds whatever capabilities are newly active.

Application deploys — the actual applications running on top of that platform. Once Forgejo exists, these get the GitOps treatment: build once via Forgejo Actions, push to Forgejo's registry, promote through environments via manifest commits in a separate Operations repo, deploy via the reconciler, roll back via git revert.

Forgejo sits at the seam: it's deployed like the rest of the bootstrap layer (by hand, no pipeline), even though its job is to run the pipeline for everything else. That's not a contradiction — it's infrastructure, same as Caddy, that happens to also be the CI/CD engine once it's up. Don't expect changes to this repo to ever flow through Forgejo Actions promotion — that's for app deploys, not for the platform hosting them, and there's no CI configured against this repo at all (no .forgejo/workflows — not even for the reconciler's own tests, deliberately, matching the rest of this repo being hand-applied and untested).

Layout

• hosts/ — one <hostname>.conf manifest per VM this repo provisions, declaring its role(s) and any capability overrides. See "Host manifests, roles, and capabilities" below.
• caddy/ — shared reverse proxy, one Caddyfile, TLS via Let's Encrypt. caddy/enabled/ (gitignored) is materialized per-host by scripts/sync-caddy-sites.sh from whichever capabilities are actually active — see "Caddy site discovery" below.
• forgejo/ — the whole forgejo capability: Forgejo itself and its Postgres database, one Compose project. See "Forgejo" below.
• infra/platform-runner/ — the runner capability: a Forgejo Actions runner (dind + cache-proxy-forwarder + runner), its own Compose project, deployable on any joined node, not only the primary. See "Forgejo Actions runner" below.
• infra/otel-collector/ — shared OTel collector, ships host and container telemetry to Honeycomb, and (via capabilities.openTelemetry, only when the otel_app capability is active on this host — see docker-compose.otel-app.yml) relays opted-in applications' own OTLP traces/metrics/logs too. No per-site observability container needed.
• infra/backup/ — shared pg_dump / volume-snapshot → rclone → B2 backup system, driven by one manifest per site. infra/backup/sites/enabled/ (gitignored) is materialized per-host by scripts/sync-backup-sites.sh, same mechanism and reasoning as caddy/enabled/.
• infra/platform-reconciler/ — two independently-gated services, split by privilege (see docs/control-plane-architecture.md): platform-control- plane (node identity/registry, all Forgejo API access, the onboarding GUI — including browser-based node management: list/add/regenerate/revoke, see docs/control-plane-architecture.md's "Node management GUI" — and per-app secrets, gated on the control_plane capability, forced on wherever forgejo is) and platform-reconciler (reconciles GitOps deployment specs against this VM's running Docker Compose state, docker.sock only, no Forgejo credential — gated on the reconciler capability). Hand-deployed, bootstrap-layer, same as Forgejo and the runner — but its own source lives right here too (src/Platform.ControlPlane, src/Platform.Reconciler, src/Platform.Operations, tests/Platform.ControlPlane.Tests, tests/Platform.Reconciler.Tests), built locally (docker compose up -d --build <service>), not pulled from a registry. setup-platform-reconciler.sh creates the shared host directories both services need and scaffolds .env files (including auto-detecting DOCKER_GID); bootstrap-operations.sh (triggered only by control_plane) then creates the Operations repo, mints the control plane's own credentials, and registers the onboarding GUI's OAuth2 application — both unattended, see "Bootstrapping a VM from scratch" below.
• scripts/ — VM bootstrap and hardening (Docker install, SSH lockdown, firewall), the host-manifest capability model (lib.sh), the forgejo-/control-plane-uniqueness lint (lint-hosts.sh), the two capability-aware materialization scripts (sync-caddy-sites.sh, sync-backup-sites.sh), and bootstrap-all.sh, the single entrypoint chaining everything above together.
• sites/ — no longer where active sites live (see "Caddy site discovery" below); holds sites/site-template/, the copy-and-customize starting point for a new site, and sites/_headers.caddy, the shared security headers snippet every site.caddy can import ../_headers.caddy.

Host manifests, roles, and capabilities

A host's manifest (hosts/<hostname>.conf, plain KEY=VALUE, bash- sourceable — same shape as every other config file in this repo) declares ROLES= (one or more of primary/application) plus any CAPABILITY_<NAME>=true|false overrides on top of those roles' defaults. scripts/lib.sh's resolve_capabilities unions the listed roles' defaults, applies the overrides, and hard-fails on any unrecognized role or capability name — a typo'd manifest should never silently no-op.

The underlying capabilities: forgejo (Forgejo itself), control_plane (platform-control-plane — forced on wherever forgejo is, never independently settable, see scripts/lib.sh), otel_host (always on), otel_app, caddy (always on), backup (always on — what gets backed up follows from which other capabilities are active, not from a backup flag of its own), reconciler (platform-reconciler), runner (a Forgejo Actions runner — defaults on for primary, off for application, independently settable either way since registration is brokered by the control plane over the network, not minted via local access to Forgejo's own container — see "Forgejo Actions runner" below). The Operations repo isn't in this list — it's not gated by a capability at all, bootstrap-all.sh creates it unconditionally wherever control_plane is (see OPS_REPO_NAME below).

Example — the live primary host, augmented with the application role:

# hosts/Falkenstein-8-16.conf
ROLES="primary application"
OPS_REPO_NAME=operations
CONTROL_PLANE_PUBLIC_BASE_URL=https://platform.joelving.dk
FORGEJO_DOMAIN=code.joelving.dk
BACKUP_PREFIX=falkenstein-8-16

OPS_REPO_NAME, CONTROL_PLANE_PUBLIC_BASE_URL, FORGEJO_DOMAIN, BACKUP_PREFIX, and the optional HEALTHCHECK_URL are examples of the other thing a manifest holds: non-secret, host-specific bootstrap arguments a capability's own setup script needs the first time it materializes itself. The dividing line is deliberate — manifests declare topology plus non-secret bootstrap parameters; .env files hold secrets and generated runtime credentials, exactly as before, never committed. BACKUP_PREFIX is required on every host, not gated on any capability (backup runs unconditionally, see "Host manifests, roles, and capabilities" above) — it namespaces every backup filename this host's infra/backup/backup.sh run writes (<prefix>__<slug>__<type>__<timestamp>.<ext>), since every host shares one B2 bucket by default and NODE_NAME isn't safe to reuse for this (two hosts can legitimately share NODE_NAME=primary, see hosts/test-full.conf's own comment on this). scripts/lint-hosts.sh enforces it's actually unique across every manifest, the same way it does FORGEJO_DOMAIN and CONTROL_PLANE_PUBLIC_BASE_URL — except globally, not per domain, since the collision it guards against (two hosts' backups interleaving in one shared bucket) isn't a per-domain risk. FORGEJO_DOMAIN is required whenever forgejo is active, required (not defaulted to .env.example's own code.joelving.dk) so a fresh checkout can never silently pick up the wrong domain, and is also what scripts/lint-hosts.sh compares across manifests to allow two genuinely different forgejo-active hosts to coexist while still catching two that would actually collide (see "Only one primary per domain" above). It's consumed exactly once, though: forgejo/setup- forgejo.sh scaffolds it into forgejo/.env the first time that file is created, and that file — not the manifest — is the ongoing source of truth after that (same "scaffold once" shape POSTGRES_PASSWORD already had). ControlPlane__ForgejoBaseUrl is inferred from that same forgejo/.env, still with no manifest field of its own, since control_plane can only ever run where forgejo does. A standalone application-role host (reconciler and/or runner active, control_plane not) instead needs CONTROL_PLANE_PUBLIC_BASE_URL to reach its remote control plane — this is what feeds Node__ControlPlaneBaseUrl in that case (see docs/control-plane-architecture.md's "Control plane is a capability of its own, forced together with Forgejo" for the fix that made this actually work end to end, not just in the capability model). RUNNER_LABELS (optional, defaults to build-test) is the same shape for the runner capability specifically — see "Forgejo Actions runner" below.

scripts/bootstrap-all.sh hosts/<hostname>.conf resolves the manifest and gates every phase on CAPS[...]. Augmenting a host is editing its manifest (add a role, or flip a CAPABILITY_* line and supply whatever new arguments that capability needs) and re-running the exact same command — every phase is idempotent and state-checked, so nothing already provisioned gets touched.

Bootstrapping a VM from scratch

sudo bash scripts/bootstrap-all.sh <manifest> — where <manifest> is a hosts/<hostname>.conf file for the VM being bootstrapped (create it first if this is a new host) — takes a bare Ubuntu VM through hardening, Docker, Caddy, the otel-collector, the backup timer, and then whichever of Forgejo, the control plane, the reconciler, and a Forgejo Actions runner that manifest's resolved capabilities call for — in that order, checking real state at every step rather than assuming a specific history, so it's equally safe to run against a fresh VM, one that's partway through a previous run, or one (like the primary host) that ran an old, pre-host-manifest version of this flow before some of these steps existed.

What it deliberately still leaves to a human, and why — all printed as reminders at the relevant point rather than silently skipped:

1. scripts/create-user.sh + scripts/lock-ssh.sh — must be run interactively, never piped; see lock-ssh.sh's own header for why a one-shot pipe risks a total SSH lockout.
2. HONEYCOMB_API_KEY — prompted interactively (read -rsp, skippable) right in the middle of the run, not left as a "go edit a file" reminder — asked once and reused for both infra/otel-collector/.env and infra/platform-reconciler/.env (the latter only if control_plane, reconciler, or runner is active), since it's the same real-world Honeycomb account either way. See prompt_honeycomb_key in scripts/lib.sh.
3. infra/backup/configure-rclone.sh — needs your B2 credentials; still fully interactive (never a script argument or env var), but called directly from the backup phase now rather than left as a separate script to remember to run — safe to call every time, since it skips any remote that already exists.
4. DNS for whatever domain(s) this host's active capabilities route — code.joelving.dk for forgejo, CONTROL_PLANE_PUBLIC_BASE_URL for control_plane — confirm this before running any of the above; see "Caddy site discovery" below for why getting this order wrong burns into Let's Encrypt's rate limit.
5. One confirmation inside forgejo/setup-forgejo.sh, only if Forgejo was already set up by hand (e.g. through the web installer) before this script existed — it can't recover that account's original password, so it asks before resetting it to a freshly generated one.
6. scripts/join-control-plane.sh, if reconciler and/or runner is active on this host — a join token (minted on the primary via scripts/mint-node-token.sh) typed in interactively, once. runner defaults on for a bare primary, so this now applies to a from-scratch primary bootstrap too, not just augmenting one or standing up a worker — see "Forgejo Actions runner" above.

Everything else — the admin account, its API tokens, the Operations repo and its Actions secrets, and the onboarding GUI's OAuth2 application — is created without a human ever typing a password into anything, using Forgejo's own admin CLI (which isn't subject to the API's password requirement for token creation) and API routes confirmed to accept normal scoped tokens rather than assumed from its docs.

Caddy site discovery

caddy/docker-compose.yml does not mount this repo's raw checkout — that was the pre-host-manifest design, and it breaks the moment the same monorepo is checked out on hosts with different capability sets: an application-role host would also see forgejo/site.caddy sitting right there in its checkout and attempt to reverse-proxy — and request its own Let's Encrypt certificate for — a domain it has no business serving.

Instead, caddy/enabled/ (gitignored) is materialized per-host by scripts/sync-caddy-sites.sh <manifest>, which mirrors each active capability's own site.caddy into it (CAPABILITY_SITE in that script maps capability → source file) and — just as important — removes a capability's directory the moment it's no longer active. Caddy's own import sites/*/site.caddy glob in the Caddyfile is unchanged; only what's mounted underneath it changed. bootstrap-all.sh runs this sync automatically before every Caddy startup; run it by hand after editing a manifest without a full bootstrap run.

(A symlink from caddy/enabled/<slug>/ into e.g. forgejo/site.caddy, rather than a copy, was the first idea for that materialization step — it doesn't actually work, same reason a sites-enabled/-style symlink never worked for the original raw-mount design: a Docker bind mount only exposes the specific host directory it's told to mount, so a symlink pointing outside that directory doesn't resolve inside the container's filesystem view. A plain copy sidesteps the problem entirely.)

Consequence worth knowing: because there's no manual step between the sync script running and Caddy seeing the result, a site's config goes live the moment Caddy reads it — either on container start, or on docker compose exec caddy caddy reload after sync-caddy-sites.sh runs following a git pull. Don't git pull + sync a new site's site.caddy onto a VM before its DNS is actually pointed at that host — Caddy requests a certificate the moment it sees the domain in its config, not on first request, and a premature attempt burns into Let's Encrypt's per-hostname failed-validation limit (5/hour — recovers on its own, but avoidable by just getting the order right: confirm DNS, then sync/reload).

Forgejo

Git hosting, PRs/issues, Actions CI, and (eventually) the OCI registry for images built on this VM. Lives in forgejo/, deployed like everything else in the bootstrap layer — by hand, not through a pipeline (see above). Only ever provisioned when the forgejo capability is active on a host, which only one host manifest may declare (see "Host manifests, roles, and capabilities" above and scripts/lint-hosts.sh).

Setup is forgejo/setup-forgejo.sh, called by scripts/bootstrap-all.sh (see "Bootstrapping a VM from scratch" above) — it opens port 2222, scaffolds .env with a random Postgres password, brings forgejo/db up (not the whole Compose project — see that script's step_compose_up for why the Actions-runner services below must stay down until registration completes) with INSTALL_LOCK set (both GITEA__/FORGEJO__ prefixes, per Forgejo's own migration guidance) so the web installer never appears, and creates the admin account plus a short-lived bootstrap API token entirely via docker exec ... forgejo admin ... — no human password typed anywhere, confirmed against Forgejo's actual route table rather than assumed. Two things still worth doing by hand after it runs:

• Test an actual SSH clone against code.joelving.dk:2222 — confirms the port mapping and firewall rule both actually work, not just that the container started.
• Confirm a backup actually runs for the forgejo slug (see "Backup" below) and check the log for success.

Check https://codeberg.org/forgejo/-/packages/container/forgejo for the current LTS tag before deploying to a new primary host — forgejo/docker-compose.yml is pinned to 15-rootless as of 2026-07, may be stale by the time this is redeployed.

Later phases (not yet built): build-and-push workflows, GitOps deploy manifests per environment, and rolling this pattern out to more applications.

Forgejo Actions runner

Status: partially built — see docs/control-plane-architecture.md's "Distributed Forgejo Actions runners" section for the full, current gap list before relying on this for anything beyond the primary's own default runner, and before trusting it against a live host at all.

Its own Compose project, infra/platform-runner/ — gated by the runner capability, independent of forgejo/control_plane. This used to be merged into the forgejo capability's own Compose project, since a runner was only ever provisionable on the one host with local docker exec access into Forgejo's container to mint it a registration token. That's no longer true: registration is brokered by the control plane (POST /nodes/runner-registration, using Forgejo's non-deprecated POST /user/actions/runners API) for any node already joined to it — the same node identity reconciler-active hosts already use, not a separate credential type (see docs/control-plane-architecture.md's "Node identity" section). A host needs NODE_NAME set and to have run scripts/join-control-plane.sh before its runner can register; an optional RUNNER_LABELS (comma-separated, defaults to build-test) controls what jobs it accepts.

Docker-in-Docker (DIND), not rootless Podman — an earlier version of this used rootless Podman for a stronger isolation property (no privilege escalation anywhere), but Testcontainers, used heavily in this user's .NET projects, has real, documented friction with it: its Ryuk cleanup sidecar doesn't reliably work under rootless Podman, and the standard fix (TESTCONTAINERS_RYUK_DISABLED=true) means losing automatic cleanup of orphaned containers after a crashed test run, for every test run, indefinitely. DIND is Forgejo's own documented backend for this, and its isolation is still real, just narrower: job containers build through the dind sidecar's own nested daemon, never the host's real Docker daemon that runs Caddy, Forgejo, the backup system, and every other site — a compromise there needs an actual container-escape exploit against the (privileged: true) sidecar to reach the host, not the immediate access a plain docker.sock mount would hand it.

Three Compose services in this project: dind, cache-proxy- forwarder, and runner, on a private runner-net. dind publishes no ports; runner publishes one, for its Actions cache proxy. Job containers run inside dind's own nested engine, a different network namespace from runner-net where neither dind nor runner resolve by name — so container.docker_host (what the runner process itself uses to reach dind, over the dind-socket volume shared between them) and runner.envs.DOCKER_HOST (what's visible inside job containers, where that same socket always lands at the hardcoded /var/run/docker.sock — confirmed against forgejo/runner's own source) are deliberately two different values, not the same dind:2375 naively pointed at from both sides — that shape was tried first and doesn't work, for the same namespace reason.

Actions' own cache feature (actions/cache@v4) hits the identical namespace problem for a different port, with a different fix: routing job containers back through the VM's own LAN IP was tried first and confirmed not to work (two layers of NAT, traffic leaving and re-entering the host through its own address). cache-proxy-forwarder — a socat relay sharing dind's own network namespace (network_mode: service:dind) rather than joining runner-net itself — sidesteps that entirely: reachable from job containers via their default gateway, and able to reach runner:8082 directly, without ever leaving dind's namespace. runner-config.yml's cache.host is set to host.docker.internal, resolved dynamically via Docker's host-gateway special value (added to job containers via container.options) to whichever gateway a container's own primary network actually has — for a job container, that's dind's internal bridge, exactly where this sidecar listens.

Setup is infra/platform-runner/setup-platform-runner.sh [name] [labels]. scripts/bootstrap-all.sh runs it automatically on a re-bootstrap once this host has already joined the control plane; on a first-ever bootstrap (nothing to join with yet) it just reminds instead, since — unlike platform-reconciler/platform-control-plane, which can start with placeholder secrets and fail to authenticate until joined — this script performs a one-shot registration call that needs a real node token the moment it runs. It calls POST /nodes/runner-registration (see above), writes the returned uuid/token directly into a .runner registration file rather than running forgejo-runner register (which still only knows the now-deprecated registration-token flow this replaces), and patches the generated runner-config.yml with container.docker_host, runner.envs.DOCKER_HOST, runner.labels, and cache.host/ cache.proxy_port itself, same as before. Registration is always scoped to this account's user-level Actions runners (never global) — Forgejo's own Actions security guidance is to register a privileged/docker.sock-capable runner like this one at the tightest useful scope, not globally, since a global scope means any workflow on the instance, not just ones deliberately pointed at this runner, could reach it. A future per-app "allow this app to run Actions here" control belongs on each app repo's own Actions unit (Settings → Units → Actions), not on re-scoping this runner.

The one thing not independently verified: the .runner file's exact JSON schema (WARNING/id/uuid/name/token/address/labels) is act-runner/forgejo-runner's own documented format, confirmed via two independent descriptions of it, but not against a live forgejo-runner binary the way every other Forgejo API call in this repo has been. Verify this against a real instance before trusting it in production.

Backup

Hourly pg_dump / volume-snapshot per registered site, shipped via rclone to a shared, encrypted Backblaze B2 bucket (EU region), with GFS retention (hourly/daily/weekly/monthly, pruned automatically). backup.sh reads two directories: infra/backup/sites/enabled/*.conf (materialized per-host by scripts/sync-backup-sites.sh <manifest> from a capability's own backup.conf — e.g. forgejo/backup.conf — gitignored, not committed directly, same capability-gated mechanism and reasoning as caddy/enabled/, see "Caddy site discovery" above) and a reconciler-owned generated directory (one .conf per application declaring capabilities.backup in its Operations spec, kept in sync by platform-reconciler itself, never git-committed either). See infra/backup/sites/site-template.conf.example for the field reference.

setup-backup.sh (installs rclone, the systemd timer) and configure-rclone.sh (needs your B2 credentials, generates encryption passwords) are both called by scripts/bootstrap-all.sh directly now, one right after the other — configure-rclone.sh stays fully interactive (it still never accepts a credential as a script argument or env var, same reasoning throughout this repo), it's just no longer a separate script you have to remember to run by hand afterward; safe to call on every run, since it skips any remote that already exists. Both run directly on the VM as root (sudo bash ...) — never from a dev machine, so credentials never leave the host. Both must run as root specifically: backup.sh runs as root via the systemd service, so it looks for rclone's config at /root/.config/rclone/rclone.conf — running configure-rclone.sh as a regular user instead writes it to that user's own $HOME, where backup.sh can't find it. Hit this once already; if it recurs, sudo cp ~/.config/rclone/rclone.conf /root/.config/rclone/rclone.conf fixes it without redoing the interactive setup (and without risking re-accepting a fresh generated password instead of Olbowler's real one).

Each registered site is attempted independently — one failing (a stray registration for a compose project that isn't actually running on this host, say) doesn't stop the others from being backed up, but the run still exits non-zero and pings the healthcheck's /fail endpoint with the specifics. That's deliberate, not softened for the capability-gated materialization above: an unexpected registration is a real bug (the materialization step itself malfunctioning, or a hand-edited file) and should be loud, not silently skipped.

Scheduled via a systemd timer (infra-setup-backup.timer/.service), not cron — installing a unit file is always idempotent (no parsing/filtering existing crontab state, the class of bug that took a couple of rounds to get right the first time this was set up), and Persistent=true means a run missed because the VM was down gets caught up on next boot rather than silently skipped. Check status with systemctl list-timers infra-setup-backup.timer — but for actual backup output, use tail /var/log/infra-setup-backup.log, not journalctl -u infra-setup-backup.service: the service unit redirects stdout/stderr straight to that file (StandardOutput=append:...), which replaces the journal destination rather than adding to it, so journald has nothing captured for this unit.

Failures also ping a healthchecks.io dead-man's-switch — kept separate from the Honeycomb-based observability elsewhere in this repo, on purpose: it's specifically good at "did this job run in the last hour," which trace-based observability isn't shaped for.

This bucket is Olbowler's existing bucket — the B2 account is on a free tier limited to one bucket, so there's no separate bucket to stand up for this VM's system; it has to share Olbowler's. That has one real consequence: when running configure-rclone.sh here, don't accept its suggested freshly-generated crypt password — enter Olbowler's existing crypt password and salt instead. Reusing it is fine, not just a workaround:

• Only the crypt password has to match to keep old and new content both readable through one remote config. The B2 application key doesn't — a fresh key scoped to the same bucket works too, if preferred over reusing Olbowler's exact one.
• Olbowler's own filenames (olbowler_20260624_231501.dump.bin, single underscore) don't collide with this system's convention (<prefix>__<slug>__<type>__<timestamp>.<ext>, double underscore) — backup.sh's prune regex and restore.sh's listing regex both require the double-underscore pattern, so neither script will ever touch or even notice Olbowler's existing files. Same bucket, same password, no interference either direction. <prefix> (BACKUP_PREFIX in the host manifest, e.g. falkenstein-8-16) is what keeps this system's own hosts from colliding with each other in the same bucket — the free-tier, one-bucket-per-account constraint above is exactly why this had to be a filename convention rather than "give every host its own bucket."

Registration status: forgejo (on the primary host, Falkenstein-8-16) is registered. justmyluck and Olbowler are not yet — both are planned migrations onto an application host, not gaps. For Olbowler specifically, the plan is not to bulk-rename its existing archive into the new convention: at migration time, restore its most recent backup directly onto the new host (now trivial, since it's the same bucket and password) — that act is the first real restore test, since the scripts read as correct on review but haven't actually been exercised end-to-end yet, and "recoverability" means more than "backups exist somewhere." Once registered, it starts a fresh GFS history in the shared bucket going forward, same as every other site — its old single-underscore files just sit there afterward, ignored by the tooling, until deliberately cleaned up.