* Add eagle0.net as server name alias
Allows accessing the site via eagle0.net in addition to prod.eagle0.net.
Note: After deploying, run certbot to add eagle0.net to the SSL certificate:
certbot --nginx -d prod.eagle0.net -d eagle0.net
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add admin container image verification in deploy workflow
The admin container was running a stale image after the entrypoint was
changed from /app/admin_server to /app/admin_server_linux_amd64. This
happened because:
1. crane pull + docker load doesn't update the :latest tag locally
2. docker-compose fallback to :latest used the old cached image
3. --force-recreate recreated the container but with the old image
Fixes:
- Tag pulled admin image as :latest after docker load (ensures fallback
uses correct image)
- Add image digest verification after deploy (fails early if wrong
image is running)
- Add admin startup verification in deploy-blue-green.sh (catches
container crashes with helpful logs)
This follows the same pattern used in auth_build.yml for auth service
verification.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
- Remove sudo symlink creation (deploy user lacks passwordless sudo)
- Simplify eagle-exec to read from /opt/eagle0/.active-instance file
- Update deploy-blue-green.sh to write active instance to file
- Keep fallback to checking running containers if file doesn't exist
Users can add their own alias:
alias eagle-exec='/opt/eagle0/scripts/eagle-exec.sh'
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add --no-deps to nginx and admin docker compose commands to prevent
them from cascading to auth. The auth container has secrets (like
TWITCH_CLIENT_ID) that are only available in auth_build.yml, not in
docker_build.yml. Without --no-deps, docker compose would recreate
auth with blank env vars when it detected config changes.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Eagle server already persists to S3 on every game save via CompoundPersister
when S3Credentials.isEnabled. The deploy script's s3cmd backup was redundant
and caused warnings when s3cmd wasn't installed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Auto-invalidate game cache when flush marker is updated
During warmup, the staging server may cache stale game data that was
loaded before the active server flushed. Instead of exposing an RPC
for cache invalidation (which leaks internal state), the server now
automatically detects when the flush marker is updated and invalidates
any cached games.
Changes:
- GamesManager: Added `invalidateCacheIfFlushMarkerUpdated()` that
checks the flush marker's modification time and clears the cache
if it's been updated since the last check
- Called before checking if a game is in cache, so stale data is
cleared before any attempt to use it
- Removed the InvalidateGameCache RPC (no longer needed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix user wait window metric to measure actual user impact
The max user wait window was measuring from when nginx switching
started (before recreation) instead of when it completed. This
inflated the metric by ~12s because nginx recreation time was included.
Users can only experience a wait AFTER nginx starts routing to the
staging server, so we now measure from nginx_switch_end (after
recreation completes) to flush_end.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
During warmup, the staging server may cache stale game data that was
loaded before the active server flushed. Instead of exposing an RPC
for cache invalidation (which leaks internal state), the server now
automatically detects when the flush marker is updated and invalidates
any cached games.
Changes:
- GamesManager: Added `invalidateCacheIfFlushMarkerUpdated()` that
checks the flush marker's modification time and clears the cache
if it's been updated since the last check
- Called before checking if a game is in cache, so stale data is
cleared before any attempt to use it
- Removed the InvalidateGameCache RPC (no longer needed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The saves directory is owned by root (created by Docker) but the deploy
script runs as the deploy user. Use docker exec to create marker files
from inside a running container that has the saves directory mounted.
- create_deployment_marker: uses active container (running before staging starts)
- create_flush_marker: uses staging container (running after active stops)
- cleanup_markers_on_failure: uses active container (still running on failure)
- remove_stale_deployment_marker: finds any running eagle container
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix nginx not picking up config changes during blue-green deploy
Root cause: `docker compose restart nginx` doesn't refresh bind-mounted
volume files. The nginx container keeps using its cached copy of
nginx.conf even after we update the host file with sed.
This caused nginx to keep trying to connect to the old (now deleted)
eagle instance, resulting in 502 errors after deployment.
Fix:
- Use `docker compose up -d --force-recreate nginx` instead of `restart`
This recreates the container, forcing it to read the updated config
- Add verification that nginx picked up the correct backend
- Remove the useless pre-validation (it validated old config in old container)
The progression of failed fixes:
1. `nginx -s reload` - doesn't re-resolve Docker DNS
2. `docker compose restart` - doesn't refresh bind-mounted files
3. `docker compose up -d --force-recreate` - THIS WORKS
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Reorder deploy: switch nginx BEFORE stopping blue to avoid 502s
Previous order (caused 502 errors):
1. Stop blue → nginx still points to blue → 502!
2. Update nginx config
3. Recreate nginx → traffic finally works
New order (eliminates 502 window):
1. Update nginx config
2. Recreate nginx → traffic goes to green (blue still running)
3. Stop blue → flushes state to disk
4. 3-second pause for flush to complete
The stale data race condition is minimized by stopping blue immediately
after the nginx switch. Users reconnecting to green will lazy-load
fresh game data from disk (after blue has flushed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add flush marker coordination for zero-downtime blue-green deploys
This ensures green never serves stale game data during deployments:
1. Deploy script creates .deployment_in_progress marker at start
2. Green's lazy-load waits for flush marker if deployment in progress
3. nginx switches to green BEFORE stopping blue (zero 502 downtime)
4. Blue stops, flushes state to disk
5. Deploy script creates .flush_complete marker
6. Green's waiting lazy-loads proceed with fresh disk data
Key changes:
- GamesManager.scala: Add waitForFlushMarker() that blocks lazy-load
during deployment until flush marker appears (30s timeout)
- GamesManager.scala: Auto-clean stale markers >5 minutes old
- GamesManager.scala: Add deployment ID correlation in logs [DEPLOY:xxx]
- GamesManager.scala: Report flush marker timeouts to Sentry
- deploy-blue-green.sh: Reorder to switch nginx BEFORE stopping blue
- deploy-blue-green.sh: Add marker file coordination with deployment ID
- deploy-blue-green.sh: Add timing metrics (flush duration, user wait window)
- nginx.conf: Keep variable-based routing (Docker DNS only resolves
running containers, so upstream+backup doesn't work)
Monitoring and observability:
- All deployment-related logs tagged with [DEPLOY:timestamp] for correlation
- Wait duration logged for each lazy-load during deployment
- Flush marker timeouts reported to Sentry for alerting
- Deploy script logs total duration, flush duration, max user wait window
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
nginx reload doesn't always force DNS re-resolution in Docker. During
blue-green deployment, after updating nginx.conf to point to the new
instance (e.g., eagle-green:40032), nginx -s reload would sometimes
keep trying to connect to the old (now removed) container, causing
502 errors.
A full restart ensures nginx picks up the new upstream correctly.
The ~1-2 second restart time is acceptable for deployment reliability.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1. Add missing jfr and jvm-tmp volumes to eagle-green
- Without these, JFR sidecar can't attach to the JVM when green is active
- JFR recordings wouldn't work either
2. Remove redundant sync_config_files from deploy script
- CI already copies config files via scp before running deploy
- Fetching from GitHub main could cause version mismatches
- Eliminates unnecessary network calls during deployment
3. Skip image pull if already present locally
- CI already pulls images before running deploy script
- Saves time during CI deployments
- Manual deployments still pull if needed
4. Add nginx config validation before reload
- Run nginx -t before nginx -s reload
- Rollback to backup config if validation fails
- Prevents broken config from taking down nginx
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Problems fixed:
1. CI was recreating admin BEFORE blue-green, causing stale .env
2. CI was restarting nginx AFTER blue-green (double restart)
3. Deploy script used slow nginx restart instead of reload
4. Cleanup was blocking the critical path
Changes:
- CI: Only restart shardok before blue-green, let script handle rest
- CI: Remove duplicate nginx restart and fallback path
- Deploy: Use nginx reload (faster) with restart fallback
- Deploy: Update .env before restarting services
- Deploy: Run container cleanup in background
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix: Move S3 backup to BEFORE stopping old server
Critical fix: The old server may wipe games.e0es on shutdown if it
doesn't have the save() merge fix. The backup must happen BEFORE
stopping to capture valid data.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Admin console shows all games, auto-install s3cmd, restart nginx properly
1. Admin console now shows all games from games.e0es, not just loaded ones:
- Added getAllRunningGamesSummary() to GamesManager
- Updated getRunningGames() to include unloaded games with "[Not loaded]" status
- Clicking into a game triggers lazy loading via getGameHistory()
2. Deploy script improvements:
- Auto-install s3cmd if not present (via pip or apt)
- Auto-configure s3cmd for DigitalOcean Spaces from .env
- Use nginx restart instead of reload to ensure all workers pick up new config
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix critical bug: save() now merges with existing games.e0es
CRITICAL BUG FIX: With lazy loading, gameControllerInfos only contains
games that have been loaded into memory. The old save() would overwrite
games.e0es with only the loaded games, losing all unloaded games.
This caused complete data loss of user-to-game mappings when:
1. New server started (empty gameControllerInfos)
2. Any operation triggered save()
3. games.e0es was overwritten with empty data
Fix: save() now reads existing games.e0es first and merges:
- Unloaded games are preserved from disk
- Loaded games use fresh in-memory state
Also adds logging to diagnose games.e0es read failures.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add S3 backup of games.e0es during blue-green deployment
After stopping the old container (which flushes state to storage),
create a timestamped backup in S3 before switching nginx traffic.
This provides a recovery point if something goes wrong during deployment.
Backups are stored at: s3://eagle0/eagle/save/backups/games.e0es.<timestamp>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix blue-green deployment dependencies and auth routing
1. docker-compose.prod.yml:
- Remove eagle-blue from nginx depends_on (use EAGLE_ADDR variable instead)
- Remove eagle-blue from admin depends_on (use EAGLE_ADDR variable instead)
- Add TODO note about jfr-sidecar limitation during green deployments
2. nginx/nginx.conf:
- Fix auth.Auth location on port 443 to route to auth:40033 instead of
eagle_backend. This was causing auth failures when clients connected
via the main HTTPS port.
These changes allow blue-green deployments to work without hard
dependencies that cause docker-compose to recreate stopped containers.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add jfr-sidecar-green for blue-green JFR profiling support
- Add jfr-sidecar-green service that shares PID namespace with eagle-green
- Make admin's jfr-sidecar address configurable via JFR_SIDECAR_ADDR env var
- Update deploy script to:
- Start appropriate jfr-sidecar with each eagle instance
- Update .env with EAGLE_ADDR and JFR_SIDECAR_ADDR after switching
- Restart admin service to pick up new addresses
- Clean up old jfr-sidecar when removing old eagle instance
This ensures the JFR button in admin console works regardless of
whether blue or green is the active instance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Sync config files from GitHub at start of deployment
Downloads docker-compose.prod.yml and nginx.conf from the main branch
before starting deployment. This ensures new services (like jfr-sidecar-green)
are available when the deploy script runs.
- Preserves the current active instance in nginx.conf
- Creates .bak backups before overwriting
- Continues with existing files if GitHub fetch fails
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix: Move S3 backup to BEFORE stopping old server
Critical fix: The old server may wipe games.e0es on shutdown if it
doesn't have the save() merge fix. The backup must happen BEFORE
stopping to capture valid data.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix critical bug: save() now merges with existing games.e0es
CRITICAL BUG FIX: With lazy loading, gameControllerInfos only contains
games that have been loaded into memory. The old save() would overwrite
games.e0es with only the loaded games, losing all unloaded games.
This caused complete data loss of user-to-game mappings when:
1. New server started (empty gameControllerInfos)
2. Any operation triggered save()
3. games.e0es was overwritten with empty data
Fix: save() now reads existing games.e0es first and merges:
- Unloaded games are preserved from disk
- Loaded games use fresh in-memory state
Also adds logging to diagnose games.e0es read failures.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add S3 backup of games.e0es during blue-green deployment
After stopping the old container (which flushes state to storage),
create a timestamped backup in S3 before switching nginx traffic.
This provides a recovery point if something goes wrong during deployment.
Backups are stored at: s3://eagle0/eagle/save/backups/games.e0es.<timestamp>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix blue-green deployment dependencies and auth routing
1. docker-compose.prod.yml:
- Remove eagle-blue from nginx depends_on (use EAGLE_ADDR variable instead)
- Remove eagle-blue from admin depends_on (use EAGLE_ADDR variable instead)
- Add TODO note about jfr-sidecar limitation during green deployments
2. nginx/nginx.conf:
- Fix auth.Auth location on port 443 to route to auth:40033 instead of
eagle_backend. This was causing auth failures when clients connected
via the main HTTPS port.
These changes allow blue-green deployments to work without hard
dependencies that cause docker-compose to recreate stopped containers.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add jfr-sidecar-green for blue-green JFR profiling support
- Add jfr-sidecar-green service that shares PID namespace with eagle-green
- Make admin's jfr-sidecar address configurable via JFR_SIDECAR_ADDR env var
- Update deploy script to:
- Start appropriate jfr-sidecar with each eagle instance
- Update .env with EAGLE_ADDR and JFR_SIDECAR_ADDR after switching
- Restart admin service to pick up new addresses
- Clean up old jfr-sidecar when removing old eagle instance
This ensures the JFR button in admin console works regardless of
whether blue or green is the active instance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Sync config files from GitHub at start of deployment
Downloads docker-compose.prod.yml and nginx.conf from the main branch
before starting deployment. This ensures new services (like jfr-sidecar-green)
are available when the deploy script runs.
- Preserves the current active instance in nginx.conf
- Creates .bak backups before overwriting
- Continues with existing files if GitHub fetch fails
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
With lazy game loading (merged in #5223), games are loaded on-demand
when users reconnect after nginx switches traffic. No explicit reload
call is needed - the new server reads fresh state from storage.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1. Fix crane installation - don't try to mv crane to itself
(tar extracts to cwd which is already /opt/eagle0)
2. Keep crane binary after deploy for blue-green script to use
3. Update blue-green deploy to use crane instead of docker pull
(fixes OCI/Docker digest mismatch issue)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Handles intermittent Docker registry digest mismatch errors like:
"failed commit on ref: unexpected commit digest"
This is a known Docker/containerd issue that can occur due to:
- Registry caching
- Network/proxy issues
- Race conditions during push
Now retries up to 3 times with 5 second delays.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add blue-green deployment infrastructure for Eagle server
- Add ReloadGames RPC to eagle.proto for reloading game state from disk
- Implement reloadAllGames() and flushToDisk() in GamesManager.scala
- Add reloadGames() handler to EagleServiceImpl.scala
- Update docker-compose.prod.yml with eagle-blue/green services
- Update nginx.conf for switchable upstream
- Create scripts/deploy-blue-green.sh for zero-downtime deployment
- Create Go warmup tool (src/main/go/net/eagle0/warmup) that:
- Uses bidirectional streaming to create test games
- Posts Improve command and verifies ActionResults
- Cleans up test game after warmup
- Create scripts/warmup-eagle.sh wrapper that uses Go tool or falls back to grpcurl
- Update docker_build.yml to build and deploy warmup binary
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add X-Warmup-User header support with localhost restriction
- AuthorizationInterceptor: Accept X-Warmup-User header for warmup authentication
- Only allow X-Warmup-User from localhost connections (security)
- Go warmup tool: Send x-warmup-user metadata header
- Add grpc/metadata dependency to warmup BUILD
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
adminandGitHub