Add encrypted offsite backups for Gitea (#8795)

* Add encrypted Gitea offsite backups * Avoid Homebrew cleanup in Gitea backup * Use dedicated Spaces credentials for Gitea backups * Authenticate Gitea backups with a dedicated SSH key * Use SCP for Synology Gitea backups * Skip regenerable unreadable Gitea state * Finalize scheduled Gitea backup workflow * Group Gitea backup retention across runners
2026-07-28 21:35:42 +00:00 · 2026-07-26 21:23:17 -07:00
parent a7990f65aa
commit 809c1e256a
3 changed files with 485 additions and 0 deletions
@@ -0,0 +1,87 @@
+# Gitea offsite backups
+
+Eagle0's Gitea mirror keeps Git and Git LFS data on `zog` for fast local
+access. The `Gitea Offsite Backup` GitHub Actions workflow copies the
+recoverable state to an encrypted Restic repository in the private
+`eagle0-gitea-backups` DigitalOcean Space.
+
+## Backup contents
+
+Each snapshot contains:
+
+- a transactionally consistent SQLite backup of the Gitea database;
+- Gitea configuration and persistent application data;
+- every locally stored Git LFS object;
+- a fresh mirror clone of `admin/eagle0`;
+- the Gitea version and captured `main` commit in `manifest.txt`.
+
+Regenerable indexes, sessions, queues, repository archives, logs, and the
+OAuth2 JWT signing key are excluded. Gitea creates a new JWT signing key when
+none exists, which invalidates pre-restoration OAuth2 tokens. Container-managed
+SSH host keys are not readable by the backup account and must also be
+regenerated after a complete NAS loss.
+
+## Schedule and retention
+
+The backup runs daily at 10:23 UTC on a `halfdan` self-hosted runner. Each
+run validates the SQLite snapshot, runs `git fsck`, uploads an encrypted
+snapshot, and reads a rotating 5% sample of the Restic repository.
+
+Weekly maintenance runs Sunday at 11:53 UTC. It retains 14 daily, 8 weekly,
+12 monthly, and 3 yearly snapshots, prunes unreferenced Restic packs, and
+reads a rotating 10% sample.
+
+Use the workflow's `full_restore_verify` option after storage or credential
+changes. It downloads the latest snapshot, validates the restored SQLite
+database, and runs `git fsck` against the restored repository.
+
+## Credentials
+
+GitHub Actions stores:
+
+- `GITEA_DO_SPACES_ACCESS_KEY`
+- `GITEA_DO_SPACES_SECRET_KEY`
+- `DO_SPACES_ENDPOINT`
+- `GITEA_BACKUP_SSH_KEY`
+- `GITEA_RESTIC_PASSWORD`
+
+`GITEA_BACKUP_SSH_KEY` is a dedicated key accepted by the NAS account used for
+backup reads. It is not a copy of a developer's personal SSH key.
+
+The Restic recovery password also has an offline recovery copy in the
+workstation's macOS Keychain under service `eagle0-gitea-restic`. Never put
+that password in the repository, workflow logs, or a support ticket.
+
+## Restore
+
+Install `restic`, then set the DigitalOcean credentials, Restic repository,
+and Restic password in the shell performing the restore. Restore into an
+empty directory:
+
+```bash
+restic snapshots --host zog --tag gitea
+restic restore latest --host zog --tag gitea --target /path/to/restore
+```
+
+The snapshot preserves the absolute staging path below the restore target.
+Locate `manifest.txt`; its containing directory is the restored backup root.
+
+Before changing Gitea:
+
+```bash
+sqlite3 /path/to/restored/root/gitea/gitea.db "PRAGMA integrity_check;"
+git --git-dir=/path/to/restored/root/repositories/admin/eagle0.git fsck --full
+```
+
+To recover a complete installation:
+
+1. Recreate the same Gitea version shown in `manifest.txt`.
+2. Stop Gitea.
+3. Restore `config`, `gitea`, `lfs`, and the bare repository into the
+   matching paths under `/volume2/docker/gitea`.
+4. Preserve the restored file ownership expected by the container.
+5. Allow Gitea to regenerate indexes, sessions, queues, archives, and SSH
+   host keys.
+6. Start Gitea and verify a historical LFS checkout before enabling writes.
+
+Do not overwrite a running installation during a restore.