eagle0/docs/CONNECTION_ARCHITECTURE.md

Eagle0 Client/Server Connection Architecture

Overview

Eagle0 uses a bidirectional streaming gRPC connection between the Unity C# client and the Scala Eagle server. The Shardok tactical combat server (C++) is not directly exposed to clients; instead, Eagle server acts as a proxy, communicating with Shardok via internal gRPC and forwarding updates to clients.

Architecture Diagram

Unity Client (C#) <--[gRPC/HTTP2/TLS]--> nginx <--[HTTP]--> Eagle Server (Scala) <--[gRPC]--> Shardok Server (C++)

Protocol Definition

Service: src/main/protobuf/net/eagle0/eagle/api/eagle.proto

service Eagle {
    rpc StreamUpdates(stream UpdateStreamRequest) returns (stream UpdateStreamResponse) {}
}

This is a bidirectional streaming RPC where:

• Client can send: StreamGameRequest, HeartbeatRequest, EnterLobbyRequest, PostCommandRequest, ErrorRequest, etc.
• Server can send: GameUpdate, HeartbeatResponse, LobbyResponse, etc.

Key Message Types

UpdateStreamRequest:

• StreamGameRequest: Subscribe to game updates, includes:
  • game_id: Which game to watch
  • unfiltered_result_count: Last known Eagle action result count (for delta updates)
  • shardok_view_statuses[]: Array of Shardok game states with their known result counts
  • streaming_text_statuses[]: LLM text streaming progress
• HeartbeatRequest: Keepalive ping with client timestamp
• PostCommandRequest: Send Eagle or Shardok commands

UpdateStreamResponse:

• GameUpdate: Contains either Eagle action results or Shardok action results
• HeartbeatResponse: Keepalive pong with server timestamp

Client Implementation (C#/Unity)

Connection Setup

File: Assets/EagleConnection.cs

private const double KeepAliveSeconds = 45.0;

_channel = GrpcChannel.ForAddress("https://" + url,
    new GrpcChannelOptions {
        HttpHandler = new YetAnotherHttpHandler {
            Http2Only = true,
            Http2KeepAliveInterval = TimeSpan.FromSeconds(KeepAliveSeconds)  // 45 seconds
        },
        // ...
    });

Key Settings:

• HTTP/2 KeepAlive Interval: 45 seconds
• Connection URL: https://<url> (TLS required via nginx)
• HTTP Handler: YetAnotherHttpHandler (Unity-compatible HTTP/2 implementation)

Persistent Connection Management

File: Assets/Eagle/PersistentClientConnection.cs

Constants:

private const double TimeoutSeconds = 300.0;           // 5 minutes
private const double HeartbeatTimerSeconds = 10.0;     // 10 seconds

Connection Lifecycle:

1. Connect() (line 91)

   • Creates bidirectional streaming call with 300-second deadline
   • Starts background thread HandleStreamingCall() to process responses
   • Sends StreamGameRequest for each subscribed game
   • Prevents concurrent connection attempts with _isConnecting flag

2. HandleStreamingCall() (line 466)

   • Runs in dedicated thread
   • Continuously reads from ResponseStream
   • Updates _lastResponseReceived timestamp on every message
   • Sets up heartbeat timer via SetUpTimer() (line 475, 482)
   • On exception: attempts reconnection based on status code

3. HeartbeatTimer Logic (line 648)

   • Timer fires every 10 seconds
   • If no response > 20 seconds: Forces reconnection by disposing stream and calling Connect()
   • If no response 10-20 seconds: Sends heartbeat and creates new timer
   • If response < 10 seconds ago: Heartbeat only sent when stream has new data

4. Reconnection Handling (line 579)

   • StatusCode.Cancelled: Check if intentional cancellation, otherwise reconnect
   • StatusCode.Internal: Immediate reconnection
   • StatusCode.DeadlineExceeded: Immediate reconnection (300-second deadline hit)
   • StatusCode.Unavailable: Wait 5 seconds, then reconnect
   • ObjectDisposedException: Immediate reconnection

Subscription Model

The client maintains a Dictionary<EagleGameId, IClientConnectionSubscriber> of active game subscriptions. When a game is subscribed:

1. Client sends StreamGameRequest with game ID and last known state
2. Server sends incremental updates (only new action results since unfiltered_result_count)
3. For Shardok games: Client sends shardok_view_statuses[] array with each Shardok game's last known result count

Shardok Update Flow

For Shardok tactical battles:

1. Client tracks multiple concurrent Shardok games via ShardokViewStatuses array
2. Each StreamGameRequest includes:

   ShardokViewStatus {
       string shardok_game_id
       int32 filtered_result_count  // Last known action count for this Shardok game
   }
   

3. Server responds with ShardokActionResultResponse containing:

   message SingleShardokGameResultResponse {
       string shardok_game_id
       repeated ActionResultView action_result_views  // Only new actions since filtered_result_count
       int32 new_result_view_count
       AvailableCommands available_commands
   }
   

This delta update mechanism reduces bandwidth during Shardok gameplay.

Server Implementation (Scala)

File: src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala

StreamUpdates Method

def streamUpdates(responseObserver: StreamObserver[UpdateStreamResponse]):
    StreamObserver[UpdateStreamRequest] =
      new SyncStreamObserver(new SyncResponseObserver(AuthorizationUtils.userName, responseObserver))

Creates a bidirectional stream handler that:

1. Wraps the response observer with SyncResponseObserver (username-tagged)
2. Returns SyncStreamObserver to handle client requests

Request Handling

SyncStreamObserver.onNext() dispatches based on request type:

• StreamGameRequest → streamOneUpdate() → delegates to GamesManager or CustomBattleManager
• HeartbeatRequest → handleHeartbeat() → sends HeartbeatResponse immediately
• PostCommandRequest → postCommand() → processes Eagle/Shardok commands
• EnterLobbyRequest → adds user to lobby subscribers, sends lobby state

Heartbeat Handling

private def handleHeartbeat(clientTimestamp: Long, responseObserver: SyncResponseObserver): Unit = {
  println(s"got heartbeat from $userName. Client timestamp $clientTimestamp,
          server timestamp ${System.currentTimeMillis()}")
  responseObserver.onNext(
    UpdateStreamResponse(
      responseDetails = HeartbeatResponse(serverTimestamp = System.currentTimeMillis())
    )
  )
}

Server immediately responds to heartbeats with server timestamp. No server-side timeout enforcement visible in this layer.

Eagle ↔ Shardok Communication

When client sends ShardokCommand or ShardokPlacementCommands:

1. Eagle server checks if game is in gamesManager or customBattleManager
2. Routes to appropriate manager's postShardokCommand() or postPlacementCommands()
3. Manager communicates with Shardok server via internal gRPC
4. Shardok action results flow back through Eagle server to client via ShardokActionResultResponse

Network Layer: nginx Proxy

File: /Users/dancrosby/CodingProjects/github/eagle0/eagle0.net (sibling repo)

Configuration:

location /net.eagle0.eagle.api.Eagle {
    auth_basic "User Area";
    auth_basic_user_file /etc/nginx/.htpasswd;
    grpc_pass grpc://192.168.123.100:40032;
    proxy_connect_timeout 1200s;  # 20 minutes
    proxy_send_timeout 1200s;     # 20 minutes
    proxy_read_timeout 1200s;     # 20 minutes
    grpc_send_timeout 1200s;      # 20 minutes
    grpc_read_timeout 1200s;      # 20 minutes
    grpc_socket_keepalive on;
}

location /net.eagle0.common.ShardokInternalInterface {
    grpc_pass grpc://192.168.123.100:40042;
}

Key Settings:

• Protocol: HTTP/2 over TLS (line 6: listen 443 ssl http2)
• Eagle Server: grpc://192.168.123.100:40032
• Shardok Server: grpc://192.168.123.100:40042 (⚠️ EXPOSED TO CLIENTS!)
• Timeouts: 1200 seconds (20 minutes) - very generous
• Socket Keepalive: Enabled for Eagle endpoint
• TLS: Let's Encrypt certificate for eagle0.net

Analysis:

• nginx timeouts are NOT the problem (20 minutes is very generous)
• HTTP/2 is properly configured
• Socket keepalive is enabled
• ⚠️ CRITICAL: Shardok internal interface is exposed at /net.eagle0.common.ShardokInternalInterface without auth or timeouts!

Server Implementation Details

File: src/main/scala/net/eagle0/eagle/Main.scala

val serverBuilder = ServerBuilder.forPort(grpcPort)  // Default port: 40032

Server Configuration:

• Uses default ServerBuilder.forPort() with NO custom timeout settings
• gRPC Java defaults:
  • maxConnectionIdle: infinity (no automatic closure of idle connections)
  • maxConnectionAge: infinity (connections never expire)
  • keepAliveTime: 2 hours (but NOT enforced by default)
  • keepAliveTimeout: 20 seconds

Key Finding: Server does NOT enforce any timeouts that would cause 2-minute drops.

Complete Timeout Settings Summary

Layer	Component	Setting	Value	Notes
Client	HTTP/2 Handler	KeepAlive Interval	45s	HTTP/2 PING frames (YetAnotherHttpHandler)
Client	gRPC Call	Deadline	300s (5 min)	Per-call timeout for StreamUpdates()
Client	Connection	Heartbeat Timer	10s	Application-level ping when idle
Client	Connection	Heartbeat Timeout	20s	Force reconnect if no response
Client	Reconnection	Unavailable Retry Delay	5s	Backoff when server unavailable
nginx	Proxy	connect_timeout	1200s (20 min)	Connection establishment
nginx	Proxy	send_timeout	1200s (20 min)	Time to send request
nginx	Proxy	read_timeout	1200s (20 min)	Time to read response
nginx	gRPC	send_timeout	1200s (20 min)	gRPC-specific send
nginx	gRPC	read_timeout	1200s (20 min)	gRPC-specific read
Server	gRPC	maxConnectionIdle	∞	No forced idle timeout
Server	gRPC	maxConnectionAge	∞	No forced age limit
Server	gRPC	keepAliveTime	2 hours (unused)	Not enforced

Historical Note: Client deadline was increased from 55s to 300s in commit 7f5b246d59 (Aug 2024) to "extend the timeout".

Identified Issues

1. MYSTERY: Where is the "2 minute timeout"?

User reports: "Connection drops every 2min or so" and "I (think I) put it on a timeout slightly under 2min"

Current settings in code:

• Client gRPC deadline: 300 seconds (5 minutes)
• nginx timeouts: 1200 seconds (20 minutes)
• Server timeouts: infinity (none)

FINDINGS:

• ✅ nginx has very generous 20-minute timeouts - NOT the problem
• ✅ Server has no forced timeouts - NOT the problem
• ✅ Client deadline is 5 minutes - NOT 2 minutes

HYPOTHESIS:

1. User may be misremembering - there is NO 2-minute timeout in the code
2. OR: External network infrastructure - ISP/router/firewall between remote player and nginx may have 2-minute idle timeout
3. OR: The "2 minutes" observation is approximate - actual drops may be at 5 minutes (deadline) or caused by other issues

RECOMMENDATION: Add detailed logging with precise timestamps to measure actual time between drops.

2. 🔴 CRITICAL BUG: Shardok Internal Interface Exposed Without Auth!

nginx configuration (lines 53-57):

location /net.eagle0.common.ShardokInternalInterface {
    grpc_pass grpc://192.168.123.100:40042;
}

PROBLEMS:

• ❌ No auth_basic directive (Eagle endpoint has auth on line 37-38)
• ❌ No timeout settings (Eagle endpoint has timeouts on lines 44-49)
• ❌ No socket keepalive (Eagle endpoint has it on line 49)
• ❌ Exposes internal Shardok interface directly to internet

IMPACT:

• Anyone can bypass Eagle server and call Shardok directly
• No authentication or authorization checks
• Contradicts architecture claim that "Shardok server is not exposed directly to the client"
• Potential security vulnerability

RECOMMENDATION: Either:

1. Remove this location block entirely (Shardok should only be called by Eagle server internally)
2. OR add same auth and timeout settings as Eagle endpoint

3. HTTP/2 KeepAlive vs Application Heartbeat Mismatch

Current settings:

• HTTP/2 KeepAlive: 45 seconds (YetAnotherHttpHandler)
• Application Heartbeat: 10 seconds (PersistentClientConnection)
• Heartbeat Timeout: 20 seconds (force reconnect)

ISSUES:

• HTTP/2 layer sends PING every 45 seconds
• Application layer sends heartbeat every 10 seconds (if no updates)
• Redundant and wasteful - two separate keepalive mechanisms
• Application heartbeat fires 4-5 times per HTTP/2 PING

RECOMMENDATION:

• Align HTTP/2 keepalive with application heartbeat (both 10s)
• OR disable application heartbeat and rely on HTTP/2 PING
• Current setup works but is inefficient

4. Shardok-Specific Connection Issues

User reports: "Issues only for one player who is not on the local network, and apparently only while Shardok game is going"

Why Shardok is Different:

1. Higher message frequency - Tactical combat has more frequent action/reaction cycles than strategic Eagle gameplay
2. Larger message payloads - Shardok sends full unit positions, stats, available actions per turn
3. Multiple simultaneous games - Client tracks array of ShardokViewStatuses[], each with own result count
4. Bidirectional traffic - Player commands and AI responses flowing simultaneously

Possible Root Causes:

A) Network Infrastructure NAT/Firewall Timeout

• Remote player likely behind home router/NAT
• Many consumer routers have 60-120 second idle timeouts for UDP/TCP connections
• During active Shardok gameplay, connection may appear "idle" at transport layer if:
  • Application data is buffered
  • HTTP/2 multiplexing makes connection look inactive
  • Router doesn't recognize gRPC traffic as "active"

B) HTTP/2 Flow Control Window Exhaustion

• Shardok sends bursts of updates during AI turns
• Client may not consume messages fast enough (UI thread blocking?)
• HTTP/2 flow control window fills up
• Server blocks sending, connection appears stalled
• Eventually timeout or client-side buffer overflow

C) nginx Buffer Issues

• nginx may buffer gRPC responses before sending to client
• Default nginx buffers might be too small for Shardok update bursts
• Missing grpc_buffer_size or proxy_buffer_size configuration
• Buffering could add latency that appears as "connection issues"

D) Remote Player Packet Loss/Latency

• High latency + packet loss = TCP retransmission delays
• gRPC is sensitive to head-of-line blocking in HTTP/2
• One lost packet can stall entire stream
• Appears as timeout or "dropped connection"

E) Client-Side Processing Bottleneck

• Shardok updates processed on Unity main thread
• If UI freezes during update processing, background thread can't read from stream
• Server-side buffers fill up
• Connection appears stalled

5. Delta Update State Synchronization Risk

For Shardok, client must track multiple game states:

ShardokViewStatus {
    string shardok_game_id
    int32 filtered_result_count  // Last action index seen
}

Problem: If connection drops mid-update:

1. Client may have partially processed updates
2. filtered_result_count may not match server's expectation
3. Reconnection sends stale count
4. Server either:
   • Sends duplicate updates (client sees repeated actions)
   • Skips updates (client misses actions)

State recovery mechanism (partial):

• Client can request resync by setting filteredResultCount = 0 in ShardokViewStatus
• Server will send all results from the beginning when filteredResultCount = 0
• ⚠️ NOTE: The request_full_resync field exists in the proto (eagle.proto:88) but is NOT read by the server. It's effectively dead code. The resync works via filteredResultCount = 0 instead.
• No checksum/hash to verify client and server are in sync

TODO: Either implement server-side handling of request_full_resync, or remove the field from the proto.

Impact: After connection drop during Shardok, client can recover via resync, but relies on filteredResultCount = 0 workaround.

Recommendations

Critical Security Issue

🔴 PRIORITY 1: Fix Shardok Internal Interface Exposure

Update nginx config /Users/dancrosby/CodingProjects/github/eagle0/eagle0.net:

location /net.eagle0.common.ShardokInternalInterface {
    # This endpoint should NOT be publicly accessible
    # Option 1: Remove this block entirely (recommended)
    # Option 2: Add authentication and timeouts like Eagle endpoint:
    auth_basic "User Area";
    auth_basic_user_file /etc/nginx/.htpasswd;
    grpc_pass grpc://192.168.123.100:40042;
    grpc_send_timeout 1200s;
    grpc_read_timeout 1200s;
    grpc_socket_keepalive on;
}

Recommended: Remove the location block entirely. Shardok should only be accessible to Eagle server on internal network.

Immediate Diagnostic Actions

1. Add Precise Connection Drop Logging

Modify PersistentClientConnection.cs to log:

• Exact timestamp of every connection attempt
• Exact timestamp of every disconnection
• StatusCode of every RpcException
• Time delta between last successful message and disconnect
• Whether disconnect happened during Shardok gameplay

Example log format:

[2024-11-30 03:27:31.026] DISCONNECT after 127.3s idle, StatusCode=Cancelled, InShardok=true

This will answer:

• Is it ACTUALLY ~2 minutes, or something else?
• What is the exact error code?
• Does it only happen during Shardok?

2. Add Remote Player Network Diagnostics

For the affected remote player, collect:

• Ping time to eagle0.net (measure TCP latency, not ICMP)
• Traceroute to eagle0.net (identify network hops)
• MTU discovery (check for fragmentation issues)
• Connection stability test (long-running TCP connection)

Tools:

# On remote player's machine
mtr eagle0.net  # Continuous ping + traceroute
curl -w "@curl-format.txt" -o /dev/null -s https://eagle0.net/

3. Enable nginx Logging for gRPC Endpoint

Add to nginx config:

location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...
    access_log /var/log/nginx/eagle-grpc-access.log;
    error_log /var/log/nginx/eagle-grpc-error.log debug;
}

Look for patterns in nginx logs when remote player disconnects.

Targeted Fixes for Shardok Issues

If Root Cause is NAT/Firewall Timeout:

Solution A: Reduce HTTP/2 KeepAlive to be more aggressive

In EagleConnection.cs, change:

private const double KeepAliveSeconds = 15.0;  // Was 45.0

Sending HTTP/2 PING every 15 seconds should keep NAT sessions alive.

Solution B: Enable TCP keepalive at socket level

Add to nginx config:

location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...
    proxy_socket_keepalive on;  # Add this
}

If Root Cause is HTTP/2 Flow Control:

Solution: Increase nginx HTTP/2 buffer sizes

Add to nginx http block:

http {
    http2_max_field_size 16k;
    http2_max_header_size 32k;
    client_body_buffer_size 128k;
    client_max_body_size 10m;
}

If Root Cause is nginx Buffering:

Solution: Disable nginx buffering for gRPC

Add to nginx gRPC location:

location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...
    grpc_buffering off;  # Add this - disable response buffering
}

If Root Cause is Client-Side Processing Delay:

Solution: Move Shardok update processing off Unity main thread

Profile Unity to check if:

• HandleStreamingCall() thread is blocked waiting to write to queue
• Main thread is slow processing Shardok updates
• GC pauses are causing delays

General Connection Robustness Improvements

1. Add State Resync Mechanism

When reconnecting after drop during Shardok:

• Send StreamGameRequest with filtered_result_count = 0
• Server sends full current state instead of delta
• Client discards partial state and resyncs

2. Align Heartbeat Timings

Current setup is inefficient with two layers:

• HTTP/2 keepalive: 45s
• Application heartbeat: 10s

Option A: Single-layer approach

// In EagleConnection.cs
private const double KeepAliveSeconds = 10.0;  // Align with application heartbeat

// In PersistentClientConnection.cs
// Remove application-level heartbeat entirely, rely on HTTP/2

Option B: Keep both but optimize

// In EagleConnection.cs
private const double KeepAliveSeconds = 30.0;  // Longer HTTP/2 interval

// In PersistentClientConnection.cs
private const double HeartbeatTimerSeconds = 30.0;  // Match HTTP/2

3. Implement Exponential Backoff for Reconnections

Currently, client reconnects immediately on error. For transient network issues:

private int _reconnectAttempts = 0;
private const int MaxReconnectDelay = 30; // seconds

public async Task Connect() {
    if (_reconnectAttempts > 0) {
        int delay = Math.Min(Math.Pow(2, _reconnectAttempts), MaxReconnectDelay);
        await Task.Delay(delay * 1000);
    }
    // ... existing connect logic ...
    _reconnectAttempts++; // Increment on failure
}

// Reset on successful connection:
// _reconnectAttempts = 0;

Summary of Key Findings

✅ What's Working

1. nginx Configuration: Very generous 20-minute timeouts, properly configured for gRPC
2. Client Reconnection Logic: Now fixed with recent PRs (#4596, #4597) to handle heartbeat timeout and race conditions
3. Server Configuration: No forced timeouts that would cause premature disconnections
4. TLS/HTTPS: Properly configured with Let's Encrypt, HTTP/2 enabled

🔴 Critical Issues Found

1. Shardok Internal Interface Exposed Without Auth (Security vulnerability)
2. Mystery "2-minute timeout" - No 120-second timeout found in code
3. No state resync mechanism after connection drops during Shardok
4. Inefficient dual-layer heartbeat (HTTP/2 + application level)

🟡 Hypotheses for Remote Player Disconnects

Most Likely:

1. NAT/Firewall timeout at remote player's router/ISP (60-120s idle timeout)
2. Network packet loss/latency causing HTTP/2 head-of-line blocking
3. nginx buffering adding latency during Shardok update bursts

Less Likely: 4. HTTP/2 flow control window exhaustion 5. Client-side processing bottleneck

Needs Measurement:

• Exact time between connect and disconnect
• Correlation with Shardok gameplay phases (player turn vs AI turn)
• Remote player's network characteristics

Next Steps (Prioritized)

1. 🔴 SECURITY: Fix Shardok internal interface exposure in nginx config
2. 📊 DIAGNOSTICS: Add precise connection drop logging with timestamps
3. 🔍 INVESTIGATE: Get network diagnostics from affected remote player
4. 🐛 FIX: Implement most likely solution (reduce HTTP/2 keepalive to 15s)
5. 🔄 IMPROVE: Add state resync mechanism for Shardok
6. ⚡ OPTIMIZE: Align heartbeat timings to reduce redundancy

Files Examined

• ✅ src/main/protobuf/net/eagle0/eagle/api/eagle.proto - gRPC service definition
• ✅ src/main/csharp/net/eagle0/clients/unity/eagle0/Assets/EagleConnection.cs - Channel setup
• ✅ src/main/csharp/net/eagle0/clients/unity/eagle0/Assets/Eagle/PersistentClientConnection.cs - Connection management
• ✅ src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala - Server-side handler
• ✅ src/main/scala/net/eagle0/eagle/Main.scala - Server startup configuration
• ✅ /Users/dancrosby/CodingProjects/github/eagle0/eagle0.net - nginx configuration

Files Not Yet Examined

• YetAnotherHttpHandler source code (client HTTP/2 implementation)
• Shardok ↔ Eagle internal gRPC interface implementation
• GamesManager/CustomBattleManager Shardok integration
• Client-side Shardok update processing (Unity main thread impact)
• Server-side Shardok result caching/delta calculation

---

Document Created: 2024-11-30 Last Updated: 2024-11-30 Author: Claude Code (automated audit) Status: Includes comprehensive resilience implementation plan Critical Finding: Shardok internal interface security vulnerability Recommended First Action: Add precise connection drop logging to measure actual timeout duration

---

Connection Resilience Implementation Plan

This section outlines a comprehensive, prioritized plan to make the Eagle0 connection system more resilient, stable, and observable.

Design Principles

1. Fail Fast, Recover Fast - Detect failures quickly and reconnect automatically
2. Observability First - Cannot fix what cannot measure
3. Graceful Degradation - System should remain partially functional during issues
4. State Consistency - Game state must be correct or obviously broken (no silent corruption)
5. Security by Default - All endpoints authenticated and rate-limited
6. Progressive Enhancement - Implement improvements incrementally without breaking existing functionality

Priority 1: Critical Fixes & Diagnostics (Week 1)

1.1 Fix Shardok Security Vulnerability

Problem: Shardok internal interface exposed to internet without authentication

Impact: CRITICAL - Anyone can bypass Eagle server and manipulate Shardok games

Implementation:

Option A: Remove Public Access (RECOMMENDED)

# In eagle0.net, REMOVE this entire location block:
# location /net.eagle0.common.ShardokInternalInterface {
#     grpc_pass grpc://192.168.123.100:40042;
# }

Shardok should only be accessible on internal network (192.168.123.x) from Eagle server.

Option B: Add Authentication & Timeouts (if public access needed)

location /net.eagle0.common.ShardokInternalInterface {
    auth_basic "Shardok Internal - Authorized Use Only";
    auth_basic_user_file /etc/nginx/.htpasswd_shardok;  # Separate from Eagle

    # Add same timeout protections as Eagle
    grpc_pass grpc://192.168.123.100:40042;
    grpc_send_timeout 1200s;
    grpc_read_timeout 1200s;
    grpc_socket_keepalive on;

    # Rate limiting
    limit_req zone=shardok_internal burst=10 nodelay;

    # IP whitelist (optional)
    allow 192.168.123.0/24;  # Internal network
    deny all;
}

# In http block, add rate limit zone:
limit_req_zone $binary_remote_addr zone=shardok_internal:10m rate=100r/s;

Testing:

• Verify Shardok calls fail without auth
• Verify Eagle server can still call Shardok internally
• Load test to ensure rate limits work

Rollback Plan: Keep old config file backup, can revert nginx config instantly

Metrics: Monitor nginx 401/403 errors on Shardok endpoint

---

1.2 Add Comprehensive Connection Logging

Problem: Cannot diagnose connection issues without precise timing data

Implementation:

File: Assets/Eagle/PersistentClientConnection.cs

Add connection metrics tracking:

// New class for connection metrics
public class ConnectionMetrics {
    public DateTime? LastConnectAttempt { get; set; }
    public DateTime? LastSuccessfulConnect { get; set; }
    public DateTime? LastDisconnect { get; set; }
    public DateTime? LastMessageReceived { get; set; }
    public DateTime? LastMessageSent { get; set; }

    public int TotalConnectAttempts { get; set; }
    public int TotalDisconnects { get; set; }
    public int TotalReconnects { get; set; }

    public StatusCode? LastDisconnectReason { get; set; }
    public bool IsInShardokGame { get; set; }
    public int ActiveShardokGamesCount { get; set; }

    public double? LastConnectionDurationSeconds =>
        (LastDisconnect.HasValue && LastSuccessfulConnect.HasValue)
            ? (LastDisconnect.Value - LastSuccessfulConnect.Value).TotalSeconds
            : null;

    public double? SecondsSinceLastMessage =>
        LastMessageReceived.HasValue
            ? (DateTime.UtcNow - LastMessageReceived.Value).TotalSeconds
            : null;
}

private ConnectionMetrics _metrics = new ConnectionMetrics();

Enhanced logging in key methods:

public async Task Connect() {
    _metrics.LastConnectAttempt = DateTime.UtcNow;
    _metrics.TotalConnectAttempts++;

    if (_isConnecting) {
        _remoteEagleClientLogger.LogLine(
            $"[CONNECT] Skipped - already connecting. Attempts={_metrics.TotalConnectAttempts}");
        return;
    }

    _remoteEagleClientLogger.LogLine(
        $"[CONNECT] Starting connection attempt #{_metrics.TotalConnectAttempts}. " +
        $"Last disconnect: {FormatTimeDelta(_metrics.LastDisconnect)} ago, " +
        $"Reason: {_metrics.LastDisconnectReason}");

    // ... existing connect logic ...

    _metrics.LastSuccessfulConnect = DateTime.UtcNow;
    var connectDuration = (DateTime.UtcNow - _metrics.LastConnectAttempt.Value).TotalSeconds;
    _remoteEagleClientLogger.LogLine(
        $"[CONNECT] SUCCESS in {connectDuration:F2}s. Stream started.");
}

// In HandleStreamingCall response handling:
private async Task HandleStreamingCall() {
    // ... existing setup ...

    try {
        while (await sc.ResponseStream.MoveNext(_currentThreadToken)) {
            _metrics.LastMessageReceived = DateTime.UtcNow;

            var response = sc.ResponseStream.Current;

            _timingsLogger.LogLine(
                $"[MESSAGE] Received {response.ResponseDetailsCase}, " +
                $"ActiveShardok={_metrics.ActiveShardokGamesCount}, " +
                $"Idle={_metrics.SecondsSinceLastMessage:F1}s");

            // ... existing message handling ...
        }
    } catch (RpcException e) {
        _metrics.LastDisconnect = DateTime.UtcNow;
        _metrics.TotalDisconnects++;
        _metrics.LastDisconnectReason = e.StatusCode;

        var connectionDuration = _metrics.LastConnectionDurationSeconds ?? 0;

        _remoteEagleClientLogger.LogLine(
            $"[DISCONNECT] StatusCode={e.StatusCode}, " +
            $"Duration={connectionDuration:F1}s, " +
            $"InShardok={_metrics.IsInShardokGame}, " +
            $"ShardokGames={_metrics.ActiveShardokGamesCount}, " +
            $"IdleTime={_metrics.SecondsSinceLastMessage:F1}s, " +
            $"TotalDisconnects={_metrics.TotalDisconnects}");

        // ... existing error handling ...
    }
}

// Helper method
private string FormatTimeDelta(DateTime? time) {
    if (!time.HasValue) return "never";
    var delta = (DateTime.UtcNow - time.Value).TotalSeconds;
    if (delta < 60) return $"{delta:F1}s";
    if (delta < 3600) return $"{delta/60:F1}m";
    return $"{delta/3600:F1}h";
}

// Update when entering/leaving Shardok
private void UpdateShardokStatus(IClientConnectionSubscriber subscriber) {
    _metrics.ActiveShardokGamesCount = subscriber.ShardokViewStatuses.Count;
    _metrics.IsInShardokGame = _metrics.ActiveShardokGamesCount > 0;
}

Log Analysis Script:

Create scripts/analyze_connection_logs.py:

#!/usr/bin/env python3
import re
from datetime import datetime
from collections import defaultdict

# Parse connection logs and generate statistics
def analyze_logs(log_file):
    disconnects = []

    with open(log_file) as f:
        for line in f:
            if '[DISCONNECT]' in line:
                # Parse: Duration=X.Xs, InShardok=true/false, IdleTime=X.Xs
                match = re.search(r'Duration=([\d.]+)s.*InShardok=(\w+).*IdleTime=([\d.]+)s', line)
                if match:
                    disconnects.append({
                        'duration': float(match.group(1)),
                        'in_shardok': match.group(2) == 'true',
                        'idle_time': float(match.group(3))
                    })

    # Statistics
    print(f"Total disconnects: {len(disconnects)}")

    durations = [d['duration'] for d in disconnects]
    if durations:
        print(f"Connection duration - Avg: {sum(durations)/len(durations):.1f}s, " +
              f"Min: {min(durations):.1f}s, Max: {max(durations):.1f}s")

    shardok_disconnects = [d for d in disconnects if d['in_shardok']]
    print(f"Shardok disconnects: {len(shardok_disconnects)} " +
          f"({100*len(shardok_disconnects)/len(disconnects):.1f}%)")

    idle_at_disconnect = [d['idle_time'] for d in disconnects]
    if idle_at_disconnect:
        print(f"Idle time at disconnect - Avg: {sum(idle_at_disconnect)/len(idle_at_disconnect):.1f}s")

if __name__ == '__main__':
    import sys
    analyze_logs(sys.argv[1] if len(sys.argv) > 1 else 'connection.log')

Testing:

• Run game for 1 hour, force several disconnects
• Verify logs contain all expected fields
• Run analysis script to validate metrics

Success Metrics:

• 100% of disconnects logged with duration and reason
• Can identify "2-minute pattern" if it exists
• Can correlate disconnects with Shardok gameplay

---

1.3 Aggressive HTTP/2 Keepalive for NAT Traversal

Problem: HTTP/2 keepalive (45s) may not prevent NAT/firewall timeouts (60-120s)

Solution: Reduce keepalive to 15s to ensure consistent NAT session refresh

File: Assets/EagleConnection.cs

private const double KeepAliveSeconds = 15.0;  // Was 45.0

// Reason: Many consumer routers have 60-120s idle timeout for NAT sessions.
// 15-second HTTP/2 PING ensures NAT sees connection as "active"

Tradeoff Analysis:

• ✅ Benefit: Keeps NAT sessions alive, prevents premature disconnects
• ✅ Benefit: Faster detection of broken connections
• ⚠️ Cost: Slightly more network traffic (~8 bytes every 15s = 0.5 KB/min)
• ⚠️ Cost: Slightly more CPU for PING/PONG processing (negligible)

nginx Configuration Enhancement:

Add to Eagle endpoint in eagle0.net:

location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...

    # Ensure nginx doesn't buffer HTTP/2 frames
    proxy_buffering off;
    proxy_request_buffering off;

    # HTTP/2-specific settings
    http2_push_preload on;  # Allow server push if needed
}

Testing:

• Remote player plays for 30+ minutes with no input during Shardok
• Monitor connection logs for disconnects during idle periods
• Verify HTTP/2 PING frames occur every 15s (Wireshark capture)

Success Metrics:

• Connection stays alive for 10+ minutes during Shardok idle periods
• No disconnects with <60s connection duration
• Remote player reports no connection issues

---

Priority 2: State Consistency & Recovery (Week 2)

2.1 Implement State Resync Mechanism for Shardok

Problem: After connection drop during Shardok, client may have inconsistent state due to delta updates

Current Behavior:

• Client sends filtered_result_count for each Shardok game
• Server sends only actions since that count
• If connection drops mid-update, counts can desync

Solution: Add resync request mechanism

Protocol Extension:

Add to eagle.proto:

message StreamGameRequest {
    int64 game_id = 1;
    int32 unfiltered_result_count = 2;

    message ShardokViewStatus {
        string shardok_game_id = 1;
        int32 filtered_result_count = 2;
        bool request_full_resync = 3;  // NEW: Request full state instead of delta
    }
    repeated ShardokViewStatus shardok_view_statuses = 3;
    // ...
}

Client Implementation:

File: Assets/Eagle/EagleGameModel.cs

// Track if we need resync after reconnection
private Dictionary<string, bool> _shardokNeedsResync = new Dictionary<string, bool>();

// After connection drop:
public void MarkShardokForResync(string shardokGameId) {
    _shardokNeedsResync[shardokGameId] = true;
    _remoteEagleClientLogger.LogLine(
        $"[RESYNC] Marked Shardok game {shardokGameId} for full resync");
}

// When building StreamGameRequest:
private StreamGameRequest BuildStreamRequest() {
    var request = new StreamGameRequest {
        GameId = this.GameId,
        UnfilteredResultCount = _lastUnfilteredResultCount
    };

    foreach (var (shardokId, model) in _currentModel.ShardokGameModels) {
        var needsResync = _shardokNeedsResync.GetValueOrDefault(shardokId, false);

        request.ShardokViewStatuses.Add(new StreamGameRequest.Types.ShardokViewStatus {
            ShardokGameId = shardokId,
            FilteredResultCount = needsResync ? 0 : model.LastResultCount,
            RequestFullResync = needsResync
        });

        if (needsResync) {
            _remoteEagleClientLogger.LogLine(
                $"[RESYNC] Requesting full state for Shardok {shardokId}");
        }
    }

    return request;
}

// After successful resync:
public void OnShardokResyncComplete(string shardokGameId) {
    _shardokNeedsResync[shardokGameId] = false;
    _remoteEagleClientLogger.LogLine(
        $"[RESYNC] Completed for Shardok game {shardokGameId}");
}

File: Assets/Eagle/PersistentClientConnection.cs

Mark all active Shardok games for resync on disconnect:

catch (RpcException e) {
    _metrics.LastDisconnect = DateTime.UtcNow;
    _metrics.TotalDisconnects++;

    // Mark all active Shardok games for resync
    lock (this) {
        foreach (var (gameId, subscriber) in _subscribers) {
            foreach (var shardokStatus in subscriber.ShardokViewStatuses) {
                if (subscriber is EagleGameModel model) {
                    model.MarkShardokForResync(shardokStatus.shardokGameId);
                }
            }
        }
    }

    // ... existing error handling ...
}

Server Implementation:

File: src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala

private def streamOneUpdate(
    request: StreamGameRequest,
    responseObserver: SyncResponseObserver
): Unit = {
  lockAndDoWithUserName { userName =>
    request.shardokViewStatuses.foreach { status =>
      if (status.requestFullResync) {
        logger.info(s"Resync requested for Shardok game ${status.shardokGameId} by $userName")
        // Send full state instead of delta by passing filteredResultCount = 0
        // Server logic already handles this case
      }
    }

    // ... existing stream logic ...
  }
}

Testing:

• Force disconnect during Shardok AI turn (mid-update)
• Verify reconnection requests resync
• Verify client state matches server state after resync
• Test with multiple simultaneous Shardok games

Success Metrics:

• Zero state desyncs after reconnection
• Resync completes in <2 seconds
• No duplicate actions displayed after resync

---

2.2 Exponential Backoff for Reconnections

Problem: Immediate reconnection on failure can overwhelm server or waste resources on persistent failures

Solution: Implement exponential backoff with jitter

File: Assets/Eagle/PersistentClientConnection.cs

private int _consecutiveFailures = 0;
private const int MaxBackoffSeconds = 30;
private const double BackoffMultiplier = 2.0;
private const double JitterFactor = 0.2;  // ±20% randomization
private Random _backoffRandom = new Random();

public async Task Connect() {
    // Apply backoff delay if we have consecutive failures
    if (_consecutiveFailures > 0) {
        var baseDelay = Math.Min(
            Math.Pow(BackoffMultiplier, _consecutiveFailures - 1),
            MaxBackoffSeconds
        );

        // Add jitter: delay ± 20%
        var jitter = baseDelay * JitterFactor * (2 * _backoffRandom.NextDouble() - 1);
        var delaySeconds = baseDelay + jitter;

        _remoteEagleClientLogger.LogLine(
            $"[BACKOFF] Waiting {delaySeconds:F1}s before reconnect " +
            $"(failure #{_consecutiveFailures})");

        await Task.Delay((int)(delaySeconds * 1000));
    }

    if (_isConnecting) {
        _remoteEagleClientLogger.LogLine($"Connect() skipped - already connecting");
        return;
    }

    _isConnecting = true;
    try {
        _remoteEagleClientLogger.LogLine($"Connect() called");

        // ... existing connection logic ...

        // Reset consecutive failures on successful connection
        var previousFailures = _consecutiveFailures;
        _consecutiveFailures = 0;

        if (previousFailures > 0) {
            _remoteEagleClientLogger.LogLine(
                $"[BACKOFF] Reset after {previousFailures} consecutive failures");
        }

    } catch (Exception e) {
        _consecutiveFailures++;
        _remoteEagleClientLogger.LogLine(
            $"[BACKOFF] Connection failed, consecutive failures: {_consecutiveFailures}");
        throw;
    } finally {
        _isConnecting = false;
    }
}

// Also increment on RpcException:
catch (RpcException e) {
    _consecutiveFailures++;
    // ... existing handling ...
}

Backoff Schedule:

Failure #	Base Delay	With Jitter Range
1	1s	0.8s - 1.2s
2	2s	1.6s - 2.4s
3	4s	3.2s - 4.8s
4	8s	6.4s - 9.6s
5	16s	12.8s - 19.2s
6+	30s (max)	24s - 36s

Testing:

• Kill server during active gameplay
• Verify backoff delays match expected schedule
• Verify reconnection succeeds after server restart
• Test with rapid on/off server toggling

Success Metrics:

• Server load doesn't spike during outages
• Connection succeeds within 1 minute of server recovery
• Backoff resets properly after successful connection

---

2.3 Connection Health Monitoring & UI Indicators

Problem: Users don't know when connection is degraded until failure

Solution: Monitor connection health and display indicators

File: Assets/Eagle/ConnectionHealthMonitor.cs (new)

public class ConnectionHealthMonitor {
    public enum HealthStatus {
        Excellent,   // <100ms latency, no packet loss
        Good,        // 100-300ms latency, <1% loss
        Fair,        // 300-500ms latency, <5% loss
        Poor,        // >500ms latency or >5% loss
        Disconnected
    }

    private Queue<long> _heartbeatLatencies = new Queue<long>();
    private const int LatencyWindowSize = 10;

    private int _heartbeatsSent = 0;
    private int _heartbeatsReceived = 0;

    public HealthStatus CurrentStatus { get; private set; } = HealthStatus.Disconnected;
    public double AverageLatencyMs { get; private set; }
    public double PacketLossPercent { get; private set; }

    public void RecordHeartbeatSent(long clientTimestamp) {
        _heartbeatsSent++;
    }

    public void RecordHeartbeatReceived(long clientTimestamp, long serverTimestamp) {
        _heartbeatsReceived++;

        var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
        var latency = now - clientTimestamp;  // Round-trip time

        _heartbeatLatencies.Enqueue(latency);
        if (_heartbeatLatencies.Count > LatencyWindowSize) {
            _heartbeatLatencies.Dequeue();
        }

        UpdateHealthStatus();
    }

    public void RecordDisconnect() {
        CurrentStatus = HealthStatus.Disconnected;
        _heartbeatsSent = 0;
        _heartbeatsReceived = 0;
    }

    private void UpdateHealthStatus() {
        // Calculate metrics
        AverageLatencyMs = _heartbeatLatencies.Count > 0
            ? _heartbeatLatencies.Average()
            : 0;

        PacketLossPercent = _heartbeatsSent > 0
            ? 100.0 * (1.0 - (double)_heartbeatsReceived / _heartbeatsSent)
            : 0;

        // Determine status
        if (PacketLossPercent > 5) {
            CurrentStatus = HealthStatus.Poor;
        } else if (AverageLatencyMs > 500) {
            CurrentStatus = HealthStatus.Poor;
        } else if (AverageLatencyMs > 300) {
            CurrentStatus = HealthStatus.Fair;
        } else if (AverageLatencyMs > 100) {
            CurrentStatus = HealthStatus.Good;
        } else {
            CurrentStatus = HealthStatus.Excellent;
        }
    }
}

UI Integration:

Add connection indicator to EagleGameController:

public TextMeshProUGUI connectionStatusText;
public Image connectionStatusIcon;

void Update() {
    // ... existing update logic ...

    // Update connection status display
    var health = ConnectionHealthMonitor.Instance.CurrentStatus;
    connectionStatusText.text = $"{health} ({ConnectionHealthMonitor.Instance.AverageLatencyMs:F0}ms)";

    connectionStatusIcon.color = health switch {
        ConnectionHealthMonitor.HealthStatus.Excellent => Color.green,
        ConnectionHealthMonitor.HealthStatus.Good => Color.yellow,
        ConnectionHealthMonitor.HealthStatus.Fair => new Color(1f, 0.5f, 0f),  // Orange
        ConnectionHealthMonitor.HealthStatus.Poor => Color.red,
        ConnectionHealthMonitor.HealthStatus.Disconnected => Color.gray,
        _ => Color.white
    };
}

Testing:

• Simulate network latency using tc (Linux) or Charles Proxy
• Verify status changes from Excellent → Good → Fair → Poor as latency increases
• Verify packet loss detection

Success Metrics:

• Status accurately reflects connection quality
• Users report issues before complete disconnection
• Helps identify remote player's network problems

---

Priority 3: Architectural Improvements (Week 3-4)

3.1 Consolidate Heartbeat Mechanisms

Problem: Dual-layer heartbeat (HTTP/2 + application) is redundant and inefficient

Current State:

• HTTP/2 keepalive: 15s (after Priority 1.3)
• Application heartbeat: 10s
• Both serve same purpose: detect broken connections

Solution: Remove application-level heartbeat, rely on HTTP/2 PING

Rationale:

• HTTP/2 PING is lower-level, more reliable for detecting transport failures
• gRPC already handles PING/PONG automatically
• Reduces code complexity and message overhead

File: Assets/Eagle/PersistentClientConnection.cs

// REMOVE application-level heartbeat timer logic:
// - Remove HeartbeatTimerSeconds constant
// - Remove SetUpTimer() calls
// - Remove SendHeartbeatRequest() method
// - Remove TimerFired() method

// KEEP timeout detection based on last message received:
private const double MaxIdleSeconds = 30.0;  // 2x HTTP/2 keepalive interval

private void CheckForIdleTimeout() {
    var idleTime = (DateTime.UtcNow - _lastResponseReceived).TotalSeconds;

    if (idleTime > MaxIdleSeconds) {
        _remoteEagleClientLogger.LogLine(
            $"[TIMEOUT] No message for {idleTime:F1}s (max {MaxIdleSeconds}s). " +
            $"HTTP/2 keepalive failed. Reconnecting...");

        // Force reconnection
        var toCancel = _streamingCall;
        if (toCancel != null) {
            toCancel.Dispose();
            lock (this) { _streamingCall = null; }
            Task.Run(() => Connect());
        }
    }
}

// Call CheckForIdleTimeout periodically from a monitoring thread
private Thread _monitoringThread;

private void StartMonitoring() {
    _monitoringThread = new Thread(() => {
        while (!_cancellationToken.IsCancellationRequested) {
            Thread.Sleep(5000);  // Check every 5 seconds
            CheckForIdleTimeout();
        }
    });
    _monitoringThread.IsBackground = true;
    _monitoringThread.Start();
}

Server-Side Enhancement:

File: src/main/scala/net/eagle0/eagle/Main.scala

Enable gRPC keepalive enforcement:

val serverBuilder = ServerBuilder.forPort(grpcPort)
  .keepAliveTime(15, TimeUnit.SECONDS)       // Send PING every 15s if idle
  .keepAliveTimeout(10, TimeUnit.SECONDS)    // Close if no PONG in 10s
  .permitKeepAliveTime(10, TimeUnit.SECONDS) // Allow client PINGs every 10s
  .permitKeepAliveWithoutCalls(true)         // Allow PINGs even when no active calls

serverBuilder.addService(...)

Benefits:

• Simpler code (remove ~100 lines of heartbeat logic)
• Single source of truth for connection liveness
• Faster failure detection (15s vs 20s)
• Lower message overhead (HTTP/2 PING is 8 bytes vs heartbeat protobuf)

Testing:

• Verify HTTP/2 PING frames sent every 15s (Wireshark)
• Kill network connection, verify reconnect within 30s
• Test with multiple simultaneous games

Success Metrics:

• Same or better connection stability than before
• Reduced message count in logs
• Faster detection of dead connections

---

3.2 Circuit Breaker Pattern for Connection Failures

Problem: Repeated connection attempts to failed server waste resources and delay user feedback

Solution: Implement circuit breaker pattern - "fail fast" after repeated failures

States:

1. CLOSED - Normal operation, attempts connections
2. OPEN - Too many failures, reject attempts immediately for timeout period
3. HALF_OPEN - After timeout, allow one test attempt

File: Assets/Eagle/ConnectionCircuitBreaker.cs (new)

public class ConnectionCircuitBreaker {
    public enum State { Closed, Open, HalfOpen }

    private State _state = State.Closed;
    private int _failureCount = 0;
    private DateTime? _openedAt = null;

    private const int FailureThreshold = 5;          // Open after 5 failures
    private const double OpenDurationSeconds = 60;   // Stay open for 60s
    private const double HalfOpenTestTimeout = 10;   // Test attempt timeout

    public State CurrentState => _state;

    public bool ShouldAttemptConnection() {
        lock (this) {
            switch (_state) {
                case State.Closed:
                    return true;  // Normal operation

                case State.Open:
                    // Check if timeout expired
                    if (_openedAt.HasValue &&
                        (DateTime.UtcNow - _openedAt.Value).TotalSeconds > OpenDurationSeconds) {
                        _state = State.HalfOpen;
                        Logger.LogLine("[CIRCUIT] OPEN → HALF_OPEN (timeout expired, testing)");
                        return true;  // Allow test attempt
                    }
                    Logger.LogLine("[CIRCUIT] OPEN - rejecting connection attempt");
                    return false;  // Still in timeout

                case State.HalfOpen:
                    return true;  // Test attempt

                default:
                    return true;
            }
        }
    }

    public void RecordSuccess() {
        lock (this) {
            if (_state != State.Closed) {
                Logger.LogLine($"[CIRCUIT] {_state} → CLOSED (connection succeeded)");
            }
            _state = State.Closed;
            _failureCount = 0;
            _openedAt = null;
        }
    }

    public void RecordFailure() {
        lock (this) {
            _failureCount++;

            if (_state == State.HalfOpen) {
                // Test attempt failed, reopen circuit
                _state = State.Open;
                _openedAt = DateTime.UtcNow;
                Logger.LogLine($"[CIRCUIT] HALF_OPEN → OPEN (test failed)");
            } else if (_failureCount >= FailureThreshold && _state == State.Closed) {
                _state = State.Open;
                _openedAt = DateTime.UtcNow;
                Logger.LogLine($"[CIRCUIT] CLOSED → OPEN (failures={_failureCount})");
            }
        }
    }
}

Integration with PersistentClientConnection:

private ConnectionCircuitBreaker _circuitBreaker = new ConnectionCircuitBreaker();

public async Task Connect() {
    if (!_circuitBreaker.ShouldAttemptConnection()) {
        _remoteEagleClientLogger.LogLine(
            "[CIRCUIT] Connection attempt blocked by circuit breaker");
        return;  // Don't attempt connection
    }

    // ... existing backoff logic ...

    try {
        // ... existing connection logic ...

        _circuitBreaker.RecordSuccess();
    } catch (Exception e) {
        _circuitBreaker.RecordFailure();
        throw;
    }
}

User Feedback:

Show circuit breaker state in UI:

// In EagleGameController or ConnectionUI
if (circuitBreaker.CurrentState == ConnectionCircuitBreaker.State.Open) {
    ShowBanner("Server unavailable. Retrying in 60 seconds...", Color.red);
} else if (circuitBreaker.CurrentState == ConnectionCircuitBreaker.State.HalfOpen) {
    ShowBanner("Testing connection...", Color.yellow);
}

Testing:

• Kill server, verify circuit opens after 5 failures
• Verify no connection attempts during 60s timeout
• Restart server during timeout, verify test attempt after 60s
• Verify circuit closes on successful test

Success Metrics:

• No wasted connection attempts during server outages
• Users see clear "server down" message
• Connection restored within 70s of server recovery (60s timeout + 10s test)

---

3.3 Server-Side Connection Tracking & Metrics

Problem: No visibility into server-side connection health and patterns

Solution: Track connections per user and expose metrics endpoint

File: src/main/scala/net/eagle0/eagle/service/ConnectionTracker.scala (new)

package net.eagle0.eagle.service

import java.time.Instant
import scala.collection.concurrent.TrieMap

case class ConnectionInfo(
  userName: String,
  connectedAt: Instant,
  var lastActivity: Instant,
  var messageCount: Long = 0,
  var activeGameIds: Set[Long] = Set.empty,
  var activeShardokGames: Int = 0
)

object ConnectionTracker {
  private val connections = new TrieMap[String, ConnectionInfo]()

  def recordConnection(userName: String): Unit = {
    val now = Instant.now()
    connections.put(userName, ConnectionInfo(userName, now, now))
    println(s"[CONNECTION] User $userName connected. Total active: ${connections.size}")
  }

  def recordDisconnection(userName: String): Unit = {
    connections.remove(userName) match {
      case Some(info) =>
        val duration = java.time.Duration.between(info.connectedAt, Instant.now())
        println(s"[CONNECTION] User $userName disconnected after ${duration.toMinutes}min. " +
                s"Messages: ${info.messageCount}")
      case None =>
        println(s"[CONNECTION] User $userName disconnected (no info)")
    }
    println(s"[CONNECTION] Total active: ${connections.size}")
  }

  def recordActivity(userName: String, gameId: Option[Long] = None, shardokGames: Int = 0): Unit = {
    connections.get(userName).foreach { info =>
      info.lastActivity = Instant.now()
      info.messageCount += 1
      gameId.foreach(id => info.activeGameIds += id)
      if (shardokGames > 0) info.activeShardokGames = shardokGames
    }
  }

  def getMetrics: Map[String, Any] = Map(
    "total_connections" -> connections.size,
    "connections_by_user" -> connections.map { case (user, info) =>
      Map(
        "user" -> user,
        "connected_minutes" -> java.time.Duration.between(info.connectedAt, Instant.now()).toMinutes,
        "idle_seconds" -> java.time.Duration.between(info.lastActivity, Instant.now()).getSeconds,
        "message_count" -> info.messageCount,
        "active_games" -> info.activeGameIds.size,
        "shardok_games" -> info.activeShardokGames
      )
    }.toList
  )

  // Clean up stale connections (no activity for 10 minutes)
  def cleanupStale(): Unit = {
    val now = Instant.now()
    val staleThreshold = java.time.Duration.ofMinutes(10)

    connections.foreach { case (user, info) =>
      if (java.time.Duration.between(info.lastActivity, now).compareTo(staleThreshold) > 0) {
        println(s"[CONNECTION] Removing stale connection for $user")
        connections.remove(user)
      }
    }
  }
}

Integration with EagleServiceImpl:

private class SyncStreamObserver(responseObserver: SyncResponseObserver)
    extends StreamObserver[UpdateStreamRequest] {

  private val userName = AuthorizationUtils.userName
  ConnectionTracker.recordConnection(userName)

  override def onNext(value: UpdateStreamRequest): Unit = try {
    value.requestDetails match {
      case UpdateStreamRequest.RequestDetails.StreamGameRequest(req) =>
        ConnectionTracker.recordActivity(
          userName,
          Some(req.gameId),
          req.shardokViewStatuses.size
        )
        streamOneUpdate(req, responseObserver)

      // ... other cases ...
    }
  } catch { /* ... */ }

  override def onError(t: Throwable): Unit = {
    ConnectionTracker.recordDisconnection(userName)
    // ... existing error handling ...
  }

  override def onCompleted(): Unit = {
    ConnectionTracker.recordDisconnection(userName)
    // ... existing completion handling ...
  }
}

Metrics Endpoint:

Add to EagleServiceImpl.scala:

// Schedule periodic cleanup
val cleanupScheduler = Executioncontext.global
scala.concurrent.Future {
  while (true) {
    Thread.sleep(60000)  // Every minute
    ConnectionTracker.cleanupStale()
  }
}(cleanupScheduler)

// Expose metrics (could be via separate admin endpoint or logging)
def logConnectionMetrics(): Unit = {
  val metrics = ConnectionTracker.getMetrics
  println(s"[METRICS] ${metrics}")
}

Monitoring Dashboard:

Create simple web endpoint or log-based dashboard:

# Example: Parse metrics from server logs
grep "\\[METRICS\\]" server.log | tail -1 | jq '.connections_by_user'

Testing:

• Connect with multiple users
• Verify metrics show all connections
• Disconnect users, verify metrics update
• Let connections go idle, verify stale cleanup

Success Metrics:

• Can see all active connections in real-time
• Can identify users with connection problems
• Can correlate server-side and client-side connection events

---

Priority 4: Long-Term Enhancements (Week 5+)

4.1 Adaptive Connection Parameters

Problem: Fixed timeouts/keepalives don't adapt to varying network conditions

Solution: Dynamically adjust based on observed latency and packet loss

File: Assets/Eagle/AdaptiveConnectionManager.cs (new)

public class AdaptiveConnectionManager {
    private ConnectionHealthMonitor _healthMonitor;

    // Adaptive parameters
    public double CurrentKeepAliveSeconds { get; private set; } = 15.0;
    public double CurrentTimeoutSeconds { get; private set; } = 300.0;

    // Ranges
    private const double MinKeepAlive = 10.0;
    private const double MaxKeepAlive = 60.0;
    private const double MinTimeout = 120.0;
    private const double MaxTimeout = 600.0;

    public void UpdateParameters() {
        var health = _healthMonitor.CurrentStatus;
        var latency = _healthMonitor.AverageLatencyMs;

        // Adapt keepalive based on connection quality
        switch (health) {
            case ConnectionHealthMonitor.HealthStatus.Excellent:
                // Stable connection, can use longer keepalive
                CurrentKeepAliveSeconds = Math.Min(CurrentKeepAliveSeconds * 1.2, MaxKeepAlive);
                break;

            case ConnectionHealthMonitor.HealthStatus.Poor:
                // Unstable connection, use aggressive keepalive
                CurrentKeepAliveSeconds = Math.Max(CurrentKeepAliveSeconds * 0.8, MinKeepAlive);
                break;
        }

        // Adapt timeout based on latency
        // Timeout should be at least 10x average latency
        var minTimeoutForLatency = Math.Max(10 * latency / 1000.0, MinTimeout);
        CurrentTimeoutSeconds = Math.Clamp(minTimeoutForLatency, MinTimeout, MaxTimeout);
    }
}

Implementation Notes:

• Would require gRPC channel to be recreated when parameters change
• Complex to implement correctly
• Marginal benefit over fixed values
• Recommendation: Defer unless data shows need

---

4.2 WebSocket Fallback Option

Problem: Some networks/proxies may have issues with gRPC/HTTP2

Solution: Implement WebSocket transport as fallback

Complexity: HIGH - requires significant protocol changes

Benefits:

• Better compatibility with restrictive networks
• May work through more proxies/firewalls

Drawbacks:

• Duplicates transport layer code
• Harder to maintain two implementations
• Most modern networks support HTTP/2

Recommendation: Only implement if significant user segment cannot connect via gRPC

---

4.3 Client-Side Prediction for Shardok

Problem: High latency makes Shardok gameplay feel sluggish

Solution: Predict action results locally, reconcile with server

Implementation Overview:

1. Client sends command to server
2. Client immediately predicts result and updates UI
3. Server processes command and sends actual result
4. Client reconciles: if different, apply correction

Complexity: VERY HIGH - requires deterministic simulation on client

Benefits:

• Responsive gameplay even with 300ms+ latency
• Better user experience for remote players

Drawbacks:

• Complex reconciliation logic
• Potential for "rollback" visual artifacts
• Requires shipping Shardok simulation to client

Recommendation: Consider for future major release if player base grows internationally

---

Testing Strategy

Automated Testing

1. Unit Tests:

   • Connection state machine transitions
   • Backoff calculation
   • Circuit breaker state transitions
   • Metrics calculations

2. Integration Tests:

   • Full connect/disconnect/reconnect cycles
   • State resync after forced disconnect
   • Multiple simultaneous Shardok games

3. Load Testing:

   • 100+ concurrent connections
   • Rapid connect/disconnect cycles
   • Large message bursts during Shardok

Manual Testing Scenarios

1. Network Simulation:

   • Use tc (Linux traffic control) or Charles Proxy
   • Test with 50ms, 200ms, 500ms latency
   • Test with 1%, 5%, 10% packet loss
   • Test with bandwidth limits (1Mbps, 512Kbps)

2. Failure Injection:

   • Kill server during Eagle turn
   • Kill server during Shardok combat
   • Kill server during AI processing
   • Restart server multiple times rapidly
   • Disconnect client network for 30s, 60s, 120s

3. Remote Player Simulation:

   • Test from actual remote network (not local)
   • Test through VPN
   • Test from mobile hotspot
   • Test from coffee shop WiFi

Success Criteria

Metric	Current (Baseline)	Target (After Implementation)
Connection uptime during 1hr session	Unknown	>99%
Time to detect disconnect	20s	<30s (with HTTP/2 keepalive)
Time to reconnect after server restart	Variable	<70s (60s circuit + 10s test)
State desyncs after reconnect	Unknown	0%
Connection drops during 30min idle Shardok	Unknown	0%
User-visible connection quality indicator	No	Yes
Server-side connection visibility	No	Yes (metrics dashboard)

---

Implementation Timeline

Week 1: Critical Fixes & Diagnostics

• Day 1-2: Fix Shardok security vulnerability, deploy to production
• Day 3-4: Implement comprehensive connection logging
• Day 5: Reduce HTTP/2 keepalive to 15s, monitor results

Deliverable: Security fix deployed, detailed logs capturing all disconnects

Week 2: State Consistency & Recovery

• Day 1-2: Implement Shardok state resync mechanism (client + server)
• Day 3: Implement exponential backoff
• Day 4-5: Implement connection health monitoring and UI indicators

Deliverable: Zero state desyncs after reconnection, visible connection quality

Week 3-4: Architectural Improvements

• Day 1-2: Consolidate heartbeat mechanisms (remove application-level)
• Day 3: Implement circuit breaker pattern
• Day 4-5: Server-side connection tracking and metrics

Deliverable: Simplified codebase, circuit breaker prevents wasted retries

Week 5+: Long-Term Enhancements

• As needed: Adaptive connection parameters
• Future release: WebSocket fallback
• Future release: Client-side prediction

Deliverable: Production-ready resilient connection system

---

Monitoring & Observability

Key Metrics to Track

1. Connection Stability:

   • Mean time between disconnects (MTBD)
   • Connection success rate on first attempt
   • Average connection duration

2. Network Quality:

   • Average latency (p50, p95, p99)
   • Packet loss percentage
   • Jitter (latency variance)

3. Failure Recovery:

   • Time to detect disconnect
   • Time to reconnect
   • Circuit breaker activation frequency

4. State Consistency:

   • Resync request frequency
   • Resync success rate
   • Average resync duration

5. User Experience:

   • Gameplay sessions interrupted by disconnects
   • User complaints about connection
   • Average connection health status

Logging Strategy

Structured Logging Format:

[TIMESTAMP] [LEVEL] [COMPONENT] [EVENT] key1=value1 key2=value2 ...

Example:

[2024-11-30 14:23:15.123] [INFO] [CONNECT] SUCCESS duration=2.3s attempts=1 backoff=0s
[2024-11-30 14:25:30.456] [WARN] [DISCONNECT] DETECTED reason=StatusCode.Cancelled duration=135.3s idle=45.2s shardok=true
[2024-11-30 14:25:32.789] [INFO] [BACKOFF] WAITING delay=1.2s failures=1
[2024-11-30 14:25:34.012] [INFO] [RESYNC] REQUESTED game=shardok_12345

Log Aggregation:

Use ELK stack or similar:

• Elasticsearch: Store all logs
• Logstash: Parse and index
• Kibana: Visualize and alert

Alerts:

1. Critical: Circuit breaker opened (server down)
2. Warning: Connection health drops below "Good" for >5 minutes
3. Warning: >10% of connection attempts fail
4. Info: Resync requested (possible state issue)

---

Rollback Plan

Each change should be independently deployable and revertible:

1. Feature Flags:

   public static class FeatureFlags {
       public static bool UseAggressiveKeepAlive => GetFlag("aggressive_keepalive", true);
       public static bool UseCircuitBreaker => GetFlag("circuit_breaker", true);
       public static bool UseShardokResync => GetFlag("shardok_resync", true);
   }
   

2. Gradual Rollout:

   • Deploy to 10% of users first
   • Monitor metrics for 24 hours
   • If stable, roll out to 50%, then 100%

3. Quick Revert:

   • Keep previous nginx config backed up
   • Feature flags can disable changes without redeployment
   • Database/state changes are additive only (no breaking schema changes)

---

Security Considerations

Authentication & Authorization

1. Shardok Internal Interface:

   • MUST remove public access or add authentication
   • Consider mutual TLS for internal Eagle ↔ Shardok communication

2. Rate Limiting:

   # Per-user rate limits
   limit_req_zone $remote_user zone=per_user:10m rate=100r/s;
   limit_req zone=per_user burst=20 nodelay;
   
   # Global rate limit
   limit_req_zone $server_name zone=global:10m rate=1000r/s;
   limit_req zone=global burst=100 nodelay;
   

3. DDoS Protection:

   • nginx rate limiting (above)
   • Connection limit per IP: limit_conn_zone $binary_remote_addr zone=addr:10m; limit_conn addr 5;
   • Consider Cloudflare or similar CDN/DDoS protection

Input Validation

1. Command Validation:

   • Server MUST validate all commands from client
   • Check token/state before executing
   • Sanitize all user input

2. State Validation:

   • Verify filtered_result_count is within valid range
   • Reject obviously corrupted state

Audit Logging

Log all security-relevant events:

• Failed authentication attempts
• Rate limit violations
• Invalid command attempts
• Unusual connection patterns

---

Performance Considerations

Client-Side

1. Memory Usage:

   • Connection metrics: ~1KB per connection
   • Log buffering: limit to last 1000 entries, ~100KB
   • State resync: temporary spike during full state load

2. CPU Usage:

   • Heartbeat consolidation: reduces CPU by ~5-10%
   • Metrics calculation: negligible (<1% CPU)
   • Backoff jitter calculation: negligible

3. Network Bandwidth:

   • HTTP/2 keepalive at 15s: ~32 bytes/min (negligible)
   • Removing application heartbeat: saves ~40 bytes/min
   • State resync: one-time spike of full game state (~50-200KB)

Server-Side

1. Memory Usage:

   • Connection tracking: ~2KB per connection
   • 1000 concurrent users = 2MB (negligible)

2. CPU Usage:

   • Connection cleanup: runs every 60s, <1ms
   • Metrics calculation: on-demand, <10ms

3. Database:

   • No new database requirements
   • All metrics in-memory only

---

Conclusion

This comprehensive resilience plan addresses all identified connection issues through a prioritized, incremental approach:

Immediate Impact (Week 1):

• Security vulnerability fixed
• Can measure and diagnose connection issues
• NAT timeout mitigation deployed

Short-Term Gains (Week 2):

• Zero state desyncs after reconnection
• Graceful failure recovery
• User visibility into connection quality

Long-Term Benefits (Week 3-4):

• Simpler, more maintainable codebase
• Better resource utilization during failures
• Comprehensive server-side observability

Success will be measured by:

• 99% connection uptime during gameplay

• Zero user-reported state inconsistencies
• <1 minute to recover from server outages
• Clear visibility into connection health for both users and operators

The plan is designed for incremental deployment, easy rollback, and continuous monitoring to ensure each change improves the system without introducing new issues