Document Postgres history migration plan (#6964)

2026-07-28 21:35:42 +00:00 · 2026-06-08 06:45:42 -07:00
parent eeadacadd4
commit c13ae90c67
1 changed files with 149 additions and 0 deletions
@@ -0,0 +1,149 @@
 # Postgres History Migration Plan
 ## Goal
 Evaluate and, if the measurements support it, migrate production game history persistence from local SQLite plus S3 base
 and delta files to a managed Postgres database.
 This is not just a mechanical port. The important product requirement is that battle commands, especially Shardok map
 clicks, show results to the player as quickly as possible while still keeping the chance of lost results acceptably low.
 ## Proposed Direction
 Use Postgres as the authoritative production store. Local SQLite should remain useful for development, tests, and rollback
 during rollout, but production should not have two long-lived authoritative stores for the same game.
 Start with a small DigitalOcean Managed Postgres instance in the same region and VPC as Eagle. The $15 single-node plan is
 reasonable for early measurement and staging. If the design works, production can move to a more reliable plan when the
 availability requirements justify it.
 ## Phase 1: Measure First
 Before porting the full history implementation, add a benchmark or admin utility that runs from the Eagle deployment
 environment against the candidate Postgres database.
 Measure at least:
 - single action-result append transaction latency
 - batch append latency for 25 and 100 action results
 - latest snapshot lookup
 - replay from a snapshot to a target index
 - `since` and stream-update query latency
 - Shardok battle result append latency
 - p50, p95, p99, and max latency
 - async queue lag, if an async persistence prototype is included
 Useful initial exit criteria:
 - Strategic action-result writes should be comfortably below 50 ms p95.
 - Shardok synchronous persistence should be below roughly 20-30 ms p95 to stay on the click-to-animation path.
 - If Shardok sync writes are above that, use an async Shardok persistence design rather than blocking player feedback.
 ## Phase 2: Schema Prototype
 Add Postgres configuration and a prototype schema that maps the current SQLite history model.
 Expected tables:
 - `games`, for game-level metadata and backend bookkeeping
 - `action_results`, for ordered strategic action results
 - per-entity decomposition tables currently written by the SQLite dispatchers
 - `state_snapshots`, for replay anchors
 - Shardok battle result tables, including battle id, sequence number, command/result payloads, and player metadata
 - `schema_migrations`, unless the migration mechanism is handled outside the app
 Prefer batched inserts and batched reads. The current SQLite implementation can afford some local-file patterns that will
 be too chatty over a network.
 ## Phase 3: Implement Postgres History
 Implement a `FullGameHistory` backend backed by Postgres. Keep the existing SQLite backend available behind a config flag
 during rollout.
 Core methods to support:
 - `count`
 - `last`
 - `withNewResults`
 - `stateAfter`
 - `since`
 - `sinceDate`
 - round-local recent-result queries
 - Shardok result writes and reads
 The implementation should preserve action ordering and idempotency. If retries are possible, writes should include stable
 keys or sequence ranges so replaying a write request cannot duplicate results.
 ## Phase 4: Decide Shardok Durability Mode
 Shardok latency is the place where this migration can most visibly hurt the player. There are two viable modes.
 ### Option A: Synchronous Shardok Writes
 Use this if measured Postgres latency is low enough. The command path writes results to Postgres before telling the client
 about them.
 This is the simplest durability model, but only acceptable if it keeps click-to-animation latency low.
 ### Option B: Async Shardok Writes
 Use this if synchronous Postgres writes add noticeable latency.
 The model:
 - apply battle results to authoritative in-memory game state immediately
 - send client updates immediately after the ordered in-memory apply
 - enqueue ordered persistence batches in the same order they were applied
 - persist batches with `battle_id`, sequence range, payload hash, and payload
 - track `lastAppliedSeq` and `lastPersistedSeq`
 - retry failed writes until they succeed
 - alert when persistence lag exceeds a small threshold
 - block or degrade new commands if the queue grows too large
 - force a synchronous checkpoint before battle end reconciliation or strategic-layer handoff
 - drain the queue during graceful shutdown
 This creates a small crash-loss window, but only for rare crashes that happen after the player sees a result and before
 the async write completes. That tradeoff may be better than adding network persistence latency to every tactical command.
 ## Phase 5: Rollout
 Add a backend flag such as:
 ```text
 EAGLE_HISTORY_BACKEND=sqlite|postgres
 ```
 Suggested rollout:
 1. Run benchmarks from the deployed environment.
 2. Enable Postgres in staging for new games.
 3. Run warmup games and Shardok battle smoke tests.
 4. Enable Postgres for new production games.
 5. Migrate old games either lazily on first open or through an admin-triggered migration job.
 6. Keep SQLite/S3 rollback available until enough new-game production time has passed.
 If Postgres is unavailable, commands should not be acknowledged as durable. Either hold/retry them before applying, or
 reject them clearly. Do not show a result to the client while pretending it has already been durably stored unless that is
 an explicit async-durability path with lag monitoring.
 ## Phase 6: Cleanup
 After production confidence:
 - remove production dependence on S3 base/delta history files
 - keep object storage for backups, exports, or emergency archives
 - keep SQLite for local development and tests if it remains useful
 - document restore, migration, and rollback procedures
 ## Open Questions
 - What exact durability guarantee do we want for "the client already saw this result"?
 - What Shardok persistence latency is acceptable in real play?
 - Should old games migrate lazily or through a bulk/admin process?
 - When should the production database move from single-node to HA?
 - What should the player experience be if Postgres is reachable but slow?
 ## Recommended Next PR
 The next implementation PR should add a Postgres benchmark/admin utility and configuration plumbing, not the full port.
 That gives us real latency numbers from the Eagle deployment environment before committing to the larger migration.