From c13ae90c674415fc983d6aa6b7e5280f00dce116 Mon Sep 17 00:00:00 2001 From: Dan Crosby Date: Mon, 8 Jun 2026 06:45:42 -0700 Subject: [PATCH] Document Postgres history migration plan (#6964) --- docs/POSTGRES_HISTORY_MIGRATION_PLAN.md | 149 ++++++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 docs/POSTGRES_HISTORY_MIGRATION_PLAN.md diff --git a/docs/POSTGRES_HISTORY_MIGRATION_PLAN.md b/docs/POSTGRES_HISTORY_MIGRATION_PLAN.md new file mode 100644 index 0000000000..60db178871 --- /dev/null +++ b/docs/POSTGRES_HISTORY_MIGRATION_PLAN.md @@ -0,0 +1,149 @@ +# Postgres History Migration Plan + +## Goal + +Evaluate and, if the measurements support it, migrate production game history persistence from local SQLite plus S3 base +and delta files to a managed Postgres database. + +This is not just a mechanical port. The important product requirement is that battle commands, especially Shardok map +clicks, show results to the player as quickly as possible while still keeping the chance of lost results acceptably low. + +## Proposed Direction + +Use Postgres as the authoritative production store. Local SQLite should remain useful for development, tests, and rollback +during rollout, but production should not have two long-lived authoritative stores for the same game. + +Start with a small DigitalOcean Managed Postgres instance in the same region and VPC as Eagle. The $15 single-node plan is +reasonable for early measurement and staging. If the design works, production can move to a more reliable plan when the +availability requirements justify it. + +## Phase 1: Measure First + +Before porting the full history implementation, add a benchmark or admin utility that runs from the Eagle deployment +environment against the candidate Postgres database. + +Measure at least: + +- single action-result append transaction latency +- batch append latency for 25 and 100 action results +- latest snapshot lookup +- replay from a snapshot to a target index +- `since` and stream-update query latency +- Shardok battle result append latency +- p50, p95, p99, and max latency +- async queue lag, if an async persistence prototype is included + +Useful initial exit criteria: + +- Strategic action-result writes should be comfortably below 50 ms p95. +- Shardok synchronous persistence should be below roughly 20-30 ms p95 to stay on the click-to-animation path. +- If Shardok sync writes are above that, use an async Shardok persistence design rather than blocking player feedback. + +## Phase 2: Schema Prototype + +Add Postgres configuration and a prototype schema that maps the current SQLite history model. + +Expected tables: + +- `games`, for game-level metadata and backend bookkeeping +- `action_results`, for ordered strategic action results +- per-entity decomposition tables currently written by the SQLite dispatchers +- `state_snapshots`, for replay anchors +- Shardok battle result tables, including battle id, sequence number, command/result payloads, and player metadata +- `schema_migrations`, unless the migration mechanism is handled outside the app + +Prefer batched inserts and batched reads. The current SQLite implementation can afford some local-file patterns that will +be too chatty over a network. + +## Phase 3: Implement Postgres History + +Implement a `FullGameHistory` backend backed by Postgres. Keep the existing SQLite backend available behind a config flag +during rollout. + +Core methods to support: + +- `count` +- `last` +- `withNewResults` +- `stateAfter` +- `since` +- `sinceDate` +- round-local recent-result queries +- Shardok result writes and reads + +The implementation should preserve action ordering and idempotency. If retries are possible, writes should include stable +keys or sequence ranges so replaying a write request cannot duplicate results. + +## Phase 4: Decide Shardok Durability Mode + +Shardok latency is the place where this migration can most visibly hurt the player. There are two viable modes. + +### Option A: Synchronous Shardok Writes + +Use this if measured Postgres latency is low enough. The command path writes results to Postgres before telling the client +about them. + +This is the simplest durability model, but only acceptable if it keeps click-to-animation latency low. + +### Option B: Async Shardok Writes + +Use this if synchronous Postgres writes add noticeable latency. + +The model: + +- apply battle results to authoritative in-memory game state immediately +- send client updates immediately after the ordered in-memory apply +- enqueue ordered persistence batches in the same order they were applied +- persist batches with `battle_id`, sequence range, payload hash, and payload +- track `lastAppliedSeq` and `lastPersistedSeq` +- retry failed writes until they succeed +- alert when persistence lag exceeds a small threshold +- block or degrade new commands if the queue grows too large +- force a synchronous checkpoint before battle end reconciliation or strategic-layer handoff +- drain the queue during graceful shutdown + +This creates a small crash-loss window, but only for rare crashes that happen after the player sees a result and before +the async write completes. That tradeoff may be better than adding network persistence latency to every tactical command. + +## Phase 5: Rollout + +Add a backend flag such as: + +```text +EAGLE_HISTORY_BACKEND=sqlite|postgres +``` + +Suggested rollout: + +1. Run benchmarks from the deployed environment. +2. Enable Postgres in staging for new games. +3. Run warmup games and Shardok battle smoke tests. +4. Enable Postgres for new production games. +5. Migrate old games either lazily on first open or through an admin-triggered migration job. +6. Keep SQLite/S3 rollback available until enough new-game production time has passed. + +If Postgres is unavailable, commands should not be acknowledged as durable. Either hold/retry them before applying, or +reject them clearly. Do not show a result to the client while pretending it has already been durably stored unless that is +an explicit async-durability path with lag monitoring. + +## Phase 6: Cleanup + +After production confidence: + +- remove production dependence on S3 base/delta history files +- keep object storage for backups, exports, or emergency archives +- keep SQLite for local development and tests if it remains useful +- document restore, migration, and rollback procedures + +## Open Questions + +- What exact durability guarantee do we want for "the client already saw this result"? +- What Shardok persistence latency is acceptable in real play? +- Should old games migrate lazily or through a bulk/admin process? +- When should the production database move from single-node to HA? +- What should the player experience be if Postgres is reachable but slow? + +## Recommended Next PR + +The next implementation PR should add a Postgres benchmark/admin utility and configuration plumbing, not the full port. +That gives us real latency numbers from the Eagle deployment environment before committing to the larger migration.