Compare commits

...
Author SHA1 Message Date
adminandClaude Opus 4.5 3563b71543 Reorder deploy: switch nginx BEFORE stopping blue to avoid 502s
Previous order (caused 502 errors):
1. Stop blue → nginx still points to blue → 502!
2. Update nginx config
3. Recreate nginx → traffic finally works

New order (eliminates 502 window):
1. Update nginx config
2. Recreate nginx → traffic goes to green (blue still running)
3. Stop blue → flushes state to disk
4. 3-second pause for flush to complete

The stale data race condition is minimized by stopping blue immediately
after the nginx switch. Users reconnecting to green will lazy-load
fresh game data from disk (after blue has flushed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 21:08:02 -08:00
adminandClaude Opus 4.5 9dcbfccce1 Fix nginx not picking up config changes during blue-green deploy
Root cause: `docker compose restart nginx` doesn't refresh bind-mounted
volume files. The nginx container keeps using its cached copy of
nginx.conf even after we update the host file with sed.

This caused nginx to keep trying to connect to the old (now deleted)
eagle instance, resulting in 502 errors after deployment.

Fix:
- Use `docker compose up -d --force-recreate nginx` instead of `restart`
  This recreates the container, forcing it to read the updated config
- Add verification that nginx picked up the correct backend
- Remove the useless pre-validation (it validated old config in old container)

The progression of failed fixes:
1. `nginx -s reload` - doesn't re-resolve Docker DNS
2. `docker compose restart` - doesn't refresh bind-mounted files
3. `docker compose up -d --force-recreate` - THIS WORKS

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 20:02:03 -08:00
+33 -19
View File
@@ -195,13 +195,8 @@ main() {
log_warn "s3cmd not found, skipping S3 backup"
fi
# Stop the active instance (flushes state to storage)
# We must do this BEFORE switching so staging loads fresh data
log_info "Stopping eagle-${active} (flushing state to storage)..."
docker compose -f "${COMPOSE_FILE}" stop "eagle-${active}"
# Prepare nginx config and .env BEFORE restarting anything
log_info "Switching nginx upstream to eagle-${staging}..."
# Prepare nginx config and .env files
log_info "Updating nginx config to point to eagle-${staging}..."
if [ "$staging" = "green" ]; then
sed -i.bak 's/eagle-blue:40032/eagle-green:40032/g' "${NGINX_CONF}"
else
@@ -221,21 +216,40 @@ main() {
echo "JFR_SIDECAR_ADDR=jfr-sidecar:8081" >> "${env_file}"
fi
# Validate nginx config before restart
log_info "Validating nginx configuration..."
if ! docker exec nginx nginx -t 2>&1; then
log_error "nginx config validation failed, aborting!"
log_error "Rolling back nginx.conf..."
mv "${NGINX_CONF}.bak" "${NGINX_CONF}"
# === CRITICAL SECTION: Minimize time between nginx switch and blue stop ===
# This reduces the window where users might get stale game data.
# Order matters:
# 1. Switch nginx to green (blue still running as fallback)
# 2. Immediately stop blue (flushes state to disk)
# 3. Users reconnecting to green will lazy-load fresh data from disk
# Recreate nginx to pick up new config from bind-mounted volume
# IMPORTANT: We must use --force-recreate, not restart, because:
# - 'restart' doesn't refresh bind-mounted files (keeps cached config)
# - 'reload' doesn't force DNS re-resolution in Docker
log_info "Switching traffic to eagle-${staging}..."
docker compose -f "${COMPOSE_FILE}" up -d --force-recreate nginx
# Verify nginx picked up the correct config
local nginx_backend
nginx_backend=$(docker exec nginx grep -o 'eagle-[a-z]*:40032' /etc/nginx/nginx.conf | head -1)
if [ "$nginx_backend" != "eagle-${staging}:40032" ]; then
log_error "nginx config mismatch! Expected eagle-${staging}:40032, got ${nginx_backend}"
log_error "This should not happen - investigate bind mount issues"
exit 1
fi
log_info "Traffic now routing to eagle-${staging}"
# Restart nginx to pick up new upstream
# Note: We use restart instead of reload because reload doesn't always
# force DNS re-resolution in Docker, causing nginx to keep trying to
# connect to the old (removed) container
log_info "Restarting nginx..."
docker compose -f "${COMPOSE_FILE}" restart nginx
# Immediately stop the old instance to flush state to disk
# This minimizes the window where green might lazy-load stale data
log_info "Stopping eagle-${active} (flushing state to storage)..."
docker compose -f "${COMPOSE_FILE}" stop "eagle-${active}"
# Brief pause to ensure disk flush completes before any lazy loads
log_info "Waiting for state flush to complete..."
sleep 3
# === END CRITICAL SECTION ===
# Restart admin to pick up new .env
log_info "Restarting admin service..."