eagle0/docs/CONNECTION_ARCHITECTURE.md

# Eagle0 Client/Server Connection Architecture

## Overview

Eagle0 uses a bidirectional streaming gRPC connection between the Unity C# client and the Scala Eagle server. The Shardok tactical combat server (C++) is not directly exposed to clients; instead, Eagle server acts as a proxy, communicating with Shardok via internal gRPC and forwarding updates to clients.

## Architecture Diagram

```
Unity Client (C#) <--[gRPC/HTTP2/TLS]--> nginx <--[HTTP]--> Eagle Server (Scala) <--[gRPC]--> Shardok Server (C++)
```

## Protocol Definition

**Service:** `src/main/protobuf/net/eagle0/eagle/api/eagle.proto`

```protobuf
service Eagle {
    rpc StreamUpdates(stream UpdateStreamRequest) returns (stream UpdateStreamResponse) {}
}
```

This is a bidirectional streaming RPC where:
- **Client can send:** StreamGameRequest, HeartbeatRequest, EnterLobbyRequest, PostCommandRequest, ErrorRequest, etc.
- **Server can send:** GameUpdate, HeartbeatResponse, LobbyResponse, etc.

### Key Message Types

**UpdateStreamRequest:**
- `StreamGameRequest`: Subscribe to game updates, includes:
  - `game_id`: Which game to watch
  - `unfiltered_result_count`: Last known Eagle action result count (for delta updates)
  - `shardok_view_statuses[]`: Array of Shardok game states with their known result counts
  - `streaming_text_statuses[]`: LLM text streaming progress
- `HeartbeatRequest`: Keepalive ping with client timestamp
- `PostCommandRequest`: Send Eagle or Shardok commands

**UpdateStreamResponse:**
- `GameUpdate`: Contains either Eagle action results or Shardok action results
- `HeartbeatResponse`: Keepalive pong with server timestamp

## Client Implementation (C#/Unity)

### Connection Setup

**File:** `Assets/EagleConnection.cs`

```csharp
private const double KeepAliveSeconds = 45.0;

_channel = GrpcChannel.ForAddress("https://" + url,
    new GrpcChannelOptions {
        HttpHandler = new YetAnotherHttpHandler {
            Http2Only = true,
            Http2KeepAliveInterval = TimeSpan.FromSeconds(KeepAliveSeconds)  // 45 seconds
        },
        // ...
    });
```

**Key Settings:**
- **HTTP/2 KeepAlive Interval:** 45 seconds
- **Connection URL:** `https://<url>` (TLS required via nginx)
- **HTTP Handler:** YetAnotherHttpHandler (Unity-compatible HTTP/2 implementation)

### Persistent Connection Management

**File:** `Assets/Eagle/PersistentClientConnection.cs`

**Constants:**
```csharp
private const double TimeoutSeconds = 300.0;           // 5 minutes
private const double HeartbeatTimerSeconds = 10.0;     // 10 seconds
```

**Connection Lifecycle:**

1. **Connect()** (line 91)
   - Creates bidirectional streaming call with 300-second deadline
   - Starts background thread `HandleStreamingCall()` to process responses
   - Sends `StreamGameRequest` for each subscribed game
   - Prevents concurrent connection attempts with `_isConnecting` flag

2. **HandleStreamingCall()** (line 466)
   - Runs in dedicated thread
   - Continuously reads from `ResponseStream`
   - Updates `_lastResponseReceived` timestamp on every message
   - Sets up heartbeat timer via `SetUpTimer()` (line 475, 482)
   - On exception: attempts reconnection based on status code

3. **HeartbeatTimer Logic** (line 648)
   - Timer fires every 10 seconds
   - **If no response > 20 seconds:** Forces reconnection by disposing stream and calling `Connect()`
   - **If no response 10-20 seconds:** Sends heartbeat and creates new timer
   - **If response < 10 seconds ago:** Heartbeat only sent when stream has new data

4. **Reconnection Handling** (line 579)
   - **StatusCode.Cancelled:** Check if intentional cancellation, otherwise reconnect
   - **StatusCode.Internal:** Immediate reconnection
   - **StatusCode.DeadlineExceeded:** Immediate reconnection (300-second deadline hit)
   - **StatusCode.Unavailable:** Wait 5 seconds, then reconnect
   - **ObjectDisposedException:** Immediate reconnection

### Subscription Model

The client maintains a `Dictionary<EagleGameId, IClientConnectionSubscriber>` of active game subscriptions. When a game is subscribed:

1. Client sends `StreamGameRequest` with game ID and last known state
2. Server sends incremental updates (only new action results since `unfiltered_result_count`)
3. For Shardok games: Client sends `shardok_view_statuses[]` array with each Shardok game's last known result count

### Shardok Update Flow

**For Shardok tactical battles:**

1. Client tracks multiple concurrent Shardok games via `ShardokViewStatuses` array
2. Each `StreamGameRequest` includes:
   ```csharp
   ShardokViewStatus {
       string shardok_game_id
       int32 filtered_result_count  // Last known action count for this Shardok game
   }
   ```
3. Server responds with `ShardokActionResultResponse` containing:
   ```protobuf
   message SingleShardokGameResultResponse {
       string shardok_game_id
       repeated ActionResultView action_result_views  // Only new actions since filtered_result_count
       int32 new_result_view_count
       AvailableCommands available_commands
   }
   ```

This delta update mechanism reduces bandwidth during Shardok gameplay.

## Server Implementation (Scala)

**File:** `src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala`

### StreamUpdates Method

```scala
def streamUpdates(responseObserver: StreamObserver[UpdateStreamResponse]):
    StreamObserver[UpdateStreamRequest] =
      new SyncStreamObserver(new SyncResponseObserver(AuthorizationUtils.userName, responseObserver))
```

Creates a bidirectional stream handler that:
1. Wraps the response observer with `SyncResponseObserver` (username-tagged)
2. Returns `SyncStreamObserver` to handle client requests

### Request Handling

**SyncStreamObserver.onNext()** dispatches based on request type:
- `StreamGameRequest` → `streamOneUpdate()` → delegates to GamesManager or CustomBattleManager
- `HeartbeatRequest` → `handleHeartbeat()` → sends HeartbeatResponse immediately
- `PostCommandRequest` → `postCommand()` → processes Eagle/Shardok commands
- `EnterLobbyRequest` → adds user to lobby subscribers, sends lobby state

### Heartbeat Handling

```scala
private def handleHeartbeat(clientTimestamp: Long, responseObserver: SyncResponseObserver): Unit = {
  println(s"got heartbeat from $userName. Client timestamp $clientTimestamp,
          server timestamp ${System.currentTimeMillis()}")
  responseObserver.onNext(
    UpdateStreamResponse(
      responseDetails = HeartbeatResponse(serverTimestamp = System.currentTimeMillis())
    )
  )
}
```

Server immediately responds to heartbeats with server timestamp. No server-side timeout enforcement visible in this layer.

### Eagle ↔ Shardok Communication

When client sends `ShardokCommand` or `ShardokPlacementCommands`:

1. Eagle server checks if game is in `gamesManager` or `customBattleManager`
2. Routes to appropriate manager's `postShardokCommand()` or `postPlacementCommands()`
3. Manager communicates with Shardok server via internal gRPC
4. Shardok action results flow back through Eagle server to client via `ShardokActionResultResponse`

## Network Layer: nginx Proxy

**File:** `/Users/dancrosby/CodingProjects/github/eagle0/eagle0.net` (sibling repo)

**Configuration:**

```nginx
location /net.eagle0.eagle.api.Eagle {
    auth_basic "User Area";
    auth_basic_user_file /etc/nginx/.htpasswd;
    grpc_pass grpc://192.168.123.100:40032;
    proxy_connect_timeout 1200s;  # 20 minutes
    proxy_send_timeout 1200s;     # 20 minutes
    proxy_read_timeout 1200s;     # 20 minutes
    grpc_send_timeout 1200s;      # 20 minutes
    grpc_read_timeout 1200s;      # 20 minutes
    grpc_socket_keepalive on;
}

location /net.eagle0.common.ShardokInternalInterface {
    grpc_pass grpc://192.168.123.100:40042;
}
```

**Key Settings:**
- **Protocol:** HTTP/2 over TLS (line 6: `listen 443 ssl http2`)
- **Eagle Server:** `grpc://192.168.123.100:40032`
- **Shardok Server:** `grpc://192.168.123.100:40042` (⚠️ EXPOSED TO CLIENTS!)
- **Timeouts:** 1200 seconds (20 minutes) - very generous
- **Socket Keepalive:** Enabled for Eagle endpoint
- **TLS:** Let's Encrypt certificate for eagle0.net

**Analysis:**
- nginx timeouts are NOT the problem (20 minutes is very generous)
- HTTP/2 is properly configured
- Socket keepalive is enabled
- ⚠️ **CRITICAL:** Shardok internal interface is exposed at `/net.eagle0.common.ShardokInternalInterface` without auth or timeouts!

## Server Implementation Details

**File:** `src/main/scala/net/eagle0/eagle/Main.scala`

```scala
val serverBuilder = ServerBuilder.forPort(grpcPort)  // Default port: 40032
```

**Server Configuration:**
- Uses default `ServerBuilder.forPort()` with NO custom timeout settings
- gRPC Java defaults:
  - `maxConnectionIdle`: infinity (no automatic closure of idle connections)
  - `maxConnectionAge`: infinity (connections never expire)
  - `keepAliveTime`: 2 hours (but NOT enforced by default)
  - `keepAliveTimeout`: 20 seconds

**Key Finding:** Server does NOT enforce any timeouts that would cause 2-minute drops.

## Complete Timeout Settings Summary

| Layer | Component | Setting | Value | Notes |
|-------|-----------|---------|-------|-------|
| **Client** | HTTP/2 Handler | KeepAlive Interval | 45s | HTTP/2 PING frames (YetAnotherHttpHandler) |
| **Client** | gRPC Call | Deadline | 300s (5 min) | Per-call timeout for StreamUpdates() |
| **Client** | Connection | Heartbeat Timer | 10s | Application-level ping when idle |
| **Client** | Connection | Heartbeat Timeout | 20s | Force reconnect if no response |
| **Client** | Reconnection | Unavailable Retry Delay | 5s | Backoff when server unavailable |
| **nginx** | Proxy | connect_timeout | 1200s (20 min) | Connection establishment |
| **nginx** | Proxy | send_timeout | 1200s (20 min) | Time to send request |
| **nginx** | Proxy | read_timeout | 1200s (20 min) | Time to read response |
| **nginx** | gRPC | send_timeout | 1200s (20 min) | gRPC-specific send |
| **nginx** | gRPC | read_timeout | 1200s (20 min) | gRPC-specific read |
| **Server** | gRPC | maxConnectionIdle | ∞ | No forced idle timeout |
| **Server** | gRPC | maxConnectionAge | ∞ | No forced age limit |
| **Server** | gRPC | keepAliveTime | 2 hours (unused) | Not enforced |

**Historical Note:** Client deadline was increased from 55s to 300s in commit 7f5b246d59 (Aug 2024) to "extend the timeout".

## Identified Issues

### 1. **MYSTERY: Where is the "2 minute timeout"?**

**User reports:** "Connection drops every 2min or so" and "I (think I) put it on a timeout slightly under 2min"

**Current settings in code:**
- Client gRPC deadline: **300 seconds (5 minutes)**
- nginx timeouts: **1200 seconds (20 minutes)**
- Server timeouts: **infinity (none)**

**FINDINGS:**
- ✅ nginx has very generous 20-minute timeouts - NOT the problem
- ✅ Server has no forced timeouts - NOT the problem
- ✅ Client deadline is 5 minutes - NOT 2 minutes

**HYPOTHESIS:**
1. **User may be misremembering** - there is NO 2-minute timeout in the code
2. **OR: External network infrastructure** - ISP/router/firewall between remote player and nginx may have 2-minute idle timeout
3. **OR: The "2 minutes" observation is approximate** - actual drops may be at 5 minutes (deadline) or caused by other issues

**RECOMMENDATION:** Add detailed logging with precise timestamps to measure actual time between drops.

### 2. **🔴 CRITICAL BUG: Shardok Internal Interface Exposed Without Auth!**

**nginx configuration (lines 53-57):**
```nginx
location /net.eagle0.common.ShardokInternalInterface {
    grpc_pass grpc://192.168.123.100:40042;
}
```

**PROBLEMS:**
- ❌ No `auth_basic` directive (Eagle endpoint has auth on line 37-38)
- ❌ No timeout settings (Eagle endpoint has timeouts on lines 44-49)
- ❌ No socket keepalive (Eagle endpoint has it on line 49)
- ❌ Exposes internal Shardok interface directly to internet

**IMPACT:**
- Anyone can bypass Eagle server and call Shardok directly
- No authentication or authorization checks
- Contradicts architecture claim that "Shardok server is not exposed directly to the client"
- Potential security vulnerability

**RECOMMENDATION:** Either:
1. Remove this location block entirely (Shardok should only be called by Eagle server internally)
2. OR add same auth and timeout settings as Eagle endpoint

### 3. **HTTP/2 KeepAlive vs Application Heartbeat Mismatch**

**Current settings:**
- **HTTP/2 KeepAlive:** 45 seconds (YetAnotherHttpHandler)
- **Application Heartbeat:** 10 seconds (PersistentClientConnection)
- **Heartbeat Timeout:** 20 seconds (force reconnect)

**ISSUES:**
- HTTP/2 layer sends PING every 45 seconds
- Application layer sends heartbeat every 10 seconds (if no updates)
- **Redundant and wasteful** - two separate keepalive mechanisms
- Application heartbeat fires 4-5 times per HTTP/2 PING

**RECOMMENDATION:**
- Align HTTP/2 keepalive with application heartbeat (both 10s)
- OR disable application heartbeat and rely on HTTP/2 PING
- Current setup works but is inefficient

### 4. **Shardok-Specific Connection Issues**

**User reports:** "Issues only for one player who is not on the local network, and apparently only while Shardok game is going"

**Why Shardok is Different:**
1. **Higher message frequency** - Tactical combat has more frequent action/reaction cycles than strategic Eagle gameplay
2. **Larger message payloads** - Shardok sends full unit positions, stats, available actions per turn
3. **Multiple simultaneous games** - Client tracks array of `ShardokViewStatuses[]`, each with own result count
4. **Bidirectional traffic** - Player commands and AI responses flowing simultaneously

**Possible Root Causes:**

**A) Network Infrastructure NAT/Firewall Timeout**
- Remote player likely behind home router/NAT
- Many consumer routers have 60-120 second idle timeouts for UDP/TCP connections
- During active Shardok gameplay, connection may appear "idle" at transport layer if:
  - Application data is buffered
  - HTTP/2 multiplexing makes connection look inactive
  - Router doesn't recognize gRPC traffic as "active"

**B) HTTP/2 Flow Control Window Exhaustion**
- Shardok sends bursts of updates during AI turns
- Client may not consume messages fast enough (UI thread blocking?)
- HTTP/2 flow control window fills up
- Server blocks sending, connection appears stalled
- Eventually timeout or client-side buffer overflow

**C) nginx Buffer Issues**
- nginx may buffer gRPC responses before sending to client
- Default nginx buffers might be too small for Shardok update bursts
- Missing `grpc_buffer_size` or `proxy_buffer_size` configuration
- Buffering could add latency that appears as "connection issues"

**D) Remote Player Packet Loss/Latency**
- High latency + packet loss = TCP retransmission delays
- gRPC is sensitive to head-of-line blocking in HTTP/2
- One lost packet can stall entire stream
- Appears as timeout or "dropped connection"

**E) Client-Side Processing Bottleneck**
- Shardok updates processed on Unity main thread
- If UI freezes during update processing, background thread can't read from stream
- Server-side buffers fill up
- Connection appears stalled

### 5. **Delta Update State Synchronization Risk**

**For Shardok**, client must track multiple game states:
```csharp
ShardokViewStatus {
    string shardok_game_id
    int32 filtered_result_count  // Last action index seen
}
```

**Problem:** If connection drops mid-update:
1. Client may have partially processed updates
2. `filtered_result_count` may not match server's expectation
3. Reconnection sends stale count
4. Server either:
   - Sends duplicate updates (client sees repeated actions)
   - Skips updates (client misses actions)

**State recovery mechanism (partial):**
- Client can request resync by setting `filteredResultCount = 0` in `ShardokViewStatus`
- Server will send all results from the beginning when `filteredResultCount = 0`
- ⚠️ **NOTE:** The `request_full_resync` field exists in the proto (`eagle.proto:88`) but is **NOT read by the server**. It's effectively dead code. The resync works via `filteredResultCount = 0` instead.
- No checksum/hash to verify client and server are in sync

**TODO:** Either implement server-side handling of `request_full_resync`, or remove the field from the proto.

**Impact:** After connection drop during Shardok, client can recover via resync, but relies on `filteredResultCount = 0` workaround.

## Recommendations

### Critical Security Issue

**🔴 PRIORITY 1: Fix Shardok Internal Interface Exposure**

Update nginx config `/Users/dancrosby/CodingProjects/github/eagle0/eagle0.net`:

```nginx
location /net.eagle0.common.ShardokInternalInterface {
    # This endpoint should NOT be publicly accessible
    # Option 1: Remove this block entirely (recommended)
    # Option 2: Add authentication and timeouts like Eagle endpoint:
    auth_basic "User Area";
    auth_basic_user_file /etc/nginx/.htpasswd;
    grpc_pass grpc://192.168.123.100:40042;
    grpc_send_timeout 1200s;
    grpc_read_timeout 1200s;
    grpc_socket_keepalive on;
}
```

**Recommended:** Remove the location block entirely. Shardok should only be accessible to Eagle server on internal network.

### Immediate Diagnostic Actions

**1. Add Precise Connection Drop Logging**

Modify `PersistentClientConnection.cs` to log:
- Exact timestamp of every connection attempt
- Exact timestamp of every disconnection
- StatusCode of every RpcException
- Time delta between last successful message and disconnect
- Whether disconnect happened during Shardok gameplay

**Example log format:**
```
[2024-11-30 03:27:31.026] DISCONNECT after 127.3s idle, StatusCode=Cancelled, InShardok=true
```

This will answer:
- Is it ACTUALLY ~2 minutes, or something else?
- What is the exact error code?
- Does it only happen during Shardok?

**2. Add Remote Player Network Diagnostics**

For the affected remote player, collect:
- Ping time to eagle0.net (measure TCP latency, not ICMP)
- Traceroute to eagle0.net (identify network hops)
- MTU discovery (check for fragmentation issues)
- Connection stability test (long-running TCP connection)

**Tools:**
```bash
# On remote player's machine
mtr eagle0.net  # Continuous ping + traceroute
curl -w "@curl-format.txt" -o /dev/null -s https://eagle0.net/
```

**3. Enable nginx Logging for gRPC Endpoint**

Add to nginx config:
```nginx
location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...
    access_log /var/log/nginx/eagle-grpc-access.log;
    error_log /var/log/nginx/eagle-grpc-error.log debug;
}
```

Look for patterns in nginx logs when remote player disconnects.

### Targeted Fixes for Shardok Issues

**If Root Cause is NAT/Firewall Timeout:**

**Solution A: Reduce HTTP/2 KeepAlive to be more aggressive**

In `EagleConnection.cs`, change:
```csharp
private const double KeepAliveSeconds = 15.0;  // Was 45.0
```

Sending HTTP/2 PING every 15 seconds should keep NAT sessions alive.

**Solution B: Enable TCP keepalive at socket level**

Add to nginx config:
```nginx
location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...
    proxy_socket_keepalive on;  # Add this
}
```

**If Root Cause is HTTP/2 Flow Control:**

**Solution: Increase nginx HTTP/2 buffer sizes**

Add to nginx `http` block:
```nginx
http {
    http2_max_field_size 16k;
    http2_max_header_size 32k;
    client_body_buffer_size 128k;
    client_max_body_size 10m;
}
```

**If Root Cause is nginx Buffering:**

**Solution: Disable nginx buffering for gRPC**

Add to nginx gRPC location:
```nginx
location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...
    grpc_buffering off;  # Add this - disable response buffering
}
```

**If Root Cause is Client-Side Processing Delay:**

**Solution: Move Shardok update processing off Unity main thread**

Profile Unity to check if:
- `HandleStreamingCall()` thread is blocked waiting to write to queue
- Main thread is slow processing Shardok updates
- GC pauses are causing delays

### General Connection Robustness Improvements

**1. Add State Resync Mechanism**

When reconnecting after drop during Shardok:
- Send `StreamGameRequest` with `filtered_result_count = 0`
- Server sends full current state instead of delta
- Client discards partial state and resyncs

**2. Align Heartbeat Timings**

Current setup is inefficient with two layers:
- HTTP/2 keepalive: 45s
- Application heartbeat: 10s

**Option A: Single-layer approach**
```csharp
// In EagleConnection.cs
private const double KeepAliveSeconds = 10.0;  // Align with application heartbeat

// In PersistentClientConnection.cs
// Remove application-level heartbeat entirely, rely on HTTP/2
```

**Option B: Keep both but optimize**
```csharp
// In EagleConnection.cs
private const double KeepAliveSeconds = 30.0;  // Longer HTTP/2 interval

// In PersistentClientConnection.cs
private const double HeartbeatTimerSeconds = 30.0;  // Match HTTP/2
```

**3. Implement Exponential Backoff for Reconnections**

Currently, client reconnects immediately on error. For transient network issues:
```csharp
private int _reconnectAttempts = 0;
private const int MaxReconnectDelay = 30; // seconds

public async Task Connect() {
    if (_reconnectAttempts > 0) {
        int delay = Math.Min(Math.Pow(2, _reconnectAttempts), MaxReconnectDelay);
        await Task.Delay(delay * 1000);
    }
    // ... existing connect logic ...
    _reconnectAttempts++; // Increment on failure
}

// Reset on successful connection:
// _reconnectAttempts = 0;
```

## Summary of Key Findings

### ✅ What's Working

1. **nginx Configuration:** Very generous 20-minute timeouts, properly configured for gRPC
2. **Client Reconnection Logic:** Now fixed with recent PRs (#4596, #4597) to handle heartbeat timeout and race conditions
3. **Server Configuration:** No forced timeouts that would cause premature disconnections
4. **TLS/HTTPS:** Properly configured with Let's Encrypt, HTTP/2 enabled

### 🔴 Critical Issues Found

1. **Shardok Internal Interface Exposed Without Auth** (Security vulnerability)
2. **Mystery "2-minute timeout"** - No 120-second timeout found in code
3. **No state resync mechanism** after connection drops during Shardok
4. **Inefficient dual-layer heartbeat** (HTTP/2 + application level)

### 🟡 Hypotheses for Remote Player Disconnects

**Most Likely:**
1. **NAT/Firewall timeout** at remote player's router/ISP (60-120s idle timeout)
2. **Network packet loss/latency** causing HTTP/2 head-of-line blocking
3. **nginx buffering** adding latency during Shardok update bursts

**Less Likely:**
4. HTTP/2 flow control window exhaustion
5. Client-side processing bottleneck

**Needs Measurement:**
- Exact time between connect and disconnect
- Correlation with Shardok gameplay phases (player turn vs AI turn)
- Remote player's network characteristics

## Next Steps (Prioritized)

1. **🔴 SECURITY:** Fix Shardok internal interface exposure in nginx config
2. **📊 DIAGNOSTICS:** Add precise connection drop logging with timestamps
3. **🔍 INVESTIGATE:** Get network diagnostics from affected remote player
4. **🐛 FIX:** Implement most likely solution (reduce HTTP/2 keepalive to 15s)
5. **🔄 IMPROVE:** Add state resync mechanism for Shardok
6. **⚡ OPTIMIZE:** Align heartbeat timings to reduce redundancy

## Files Examined

- ✅ `src/main/protobuf/net/eagle0/eagle/api/eagle.proto` - gRPC service definition
- ✅ `src/main/csharp/net/eagle0/clients/unity/eagle0/Assets/EagleConnection.cs` - Channel setup
- ✅ `src/main/csharp/net/eagle0/clients/unity/eagle0/Assets/Eagle/PersistentClientConnection.cs` - Connection management
- ✅ `src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala` - Server-side handler
- ✅ `src/main/scala/net/eagle0/eagle/Main.scala` - Server startup configuration
- ✅ `/Users/dancrosby/CodingProjects/github/eagle0/eagle0.net` - nginx configuration

## Files Not Yet Examined

- YetAnotherHttpHandler source code (client HTTP/2 implementation)
- Shardok ↔ Eagle internal gRPC interface implementation
- GamesManager/CustomBattleManager Shardok integration
- Client-side Shardok update processing (Unity main thread impact)
- Server-side Shardok result caching/delta calculation

---

**Document Created:** 2024-11-30
**Last Updated:** 2024-11-30
**Author:** Claude Code (automated audit)
**Status:** Includes comprehensive resilience implementation plan
**Critical Finding:** Shardok internal interface security vulnerability
**Recommended First Action:** Add precise connection drop logging to measure actual timeout duration

---

# Connection Resilience Implementation Plan

This section outlines a comprehensive, prioritized plan to make the Eagle0 connection system more resilient, stable, and observable.

## Design Principles

1. **Fail Fast, Recover Fast** - Detect failures quickly and reconnect automatically
2. **Observability First** - Cannot fix what cannot measure
3. **Graceful Degradation** - System should remain partially functional during issues
4. **State Consistency** - Game state must be correct or obviously broken (no silent corruption)
5. **Security by Default** - All endpoints authenticated and rate-limited
6. **Progressive Enhancement** - Implement improvements incrementally without breaking existing functionality

## Priority 1: Critical Fixes & Diagnostics (Week 1)

### 1.1 Fix Shardok Security Vulnerability

**Problem:** Shardok internal interface exposed to internet without authentication

**Impact:** CRITICAL - Anyone can bypass Eagle server and manipulate Shardok games

**Implementation:**

**Option A: Remove Public Access (RECOMMENDED)**
```nginx
# In eagle0.net, REMOVE this entire location block:
# location /net.eagle0.common.ShardokInternalInterface {
#     grpc_pass grpc://192.168.123.100:40042;
# }
```

Shardok should only be accessible on internal network (192.168.123.x) from Eagle server.

**Option B: Add Authentication & Timeouts (if public access needed)**
```nginx
location /net.eagle0.common.ShardokInternalInterface {
    auth_basic "Shardok Internal - Authorized Use Only";
    auth_basic_user_file /etc/nginx/.htpasswd_shardok;  # Separate from Eagle

    # Add same timeout protections as Eagle
    grpc_pass grpc://192.168.123.100:40042;
    grpc_send_timeout 1200s;
    grpc_read_timeout 1200s;
    grpc_socket_keepalive on;

    # Rate limiting
    limit_req zone=shardok_internal burst=10 nodelay;

    # IP whitelist (optional)
    allow 192.168.123.0/24;  # Internal network
    deny all;
}

# In http block, add rate limit zone:
limit_req_zone $binary_remote_addr zone=shardok_internal:10m rate=100r/s;
```

**Testing:**
- Verify Shardok calls fail without auth
- Verify Eagle server can still call Shardok internally
- Load test to ensure rate limits work

**Rollback Plan:** Keep old config file backup, can revert nginx config instantly

**Metrics:** Monitor nginx 401/403 errors on Shardok endpoint

---

### 1.2 Add Comprehensive Connection Logging

**Problem:** Cannot diagnose connection issues without precise timing data

**Implementation:**

**File:** `Assets/Eagle/PersistentClientConnection.cs`

Add connection metrics tracking:

```csharp
// New class for connection metrics
public class ConnectionMetrics {
    public DateTime? LastConnectAttempt { get; set; }
    public DateTime? LastSuccessfulConnect { get; set; }
    public DateTime? LastDisconnect { get; set; }
    public DateTime? LastMessageReceived { get; set; }
    public DateTime? LastMessageSent { get; set; }

    public int TotalConnectAttempts { get; set; }
    public int TotalDisconnects { get; set; }
    public int TotalReconnects { get; set; }

    public StatusCode? LastDisconnectReason { get; set; }
    public bool IsInShardokGame { get; set; }
    public int ActiveShardokGamesCount { get; set; }

    public double? LastConnectionDurationSeconds =>
        (LastDisconnect.HasValue && LastSuccessfulConnect.HasValue)
            ? (LastDisconnect.Value - LastSuccessfulConnect.Value).TotalSeconds
            : null;

    public double? SecondsSinceLastMessage =>
        LastMessageReceived.HasValue
            ? (DateTime.UtcNow - LastMessageReceived.Value).TotalSeconds
            : null;
}

private ConnectionMetrics _metrics = new ConnectionMetrics();
```

Enhanced logging in key methods:

```csharp
public async Task Connect() {
    _metrics.LastConnectAttempt = DateTime.UtcNow;
    _metrics.TotalConnectAttempts++;

    if (_isConnecting) {
        _remoteEagleClientLogger.LogLine(
            $"[CONNECT] Skipped - already connecting. Attempts={_metrics.TotalConnectAttempts}");
        return;
    }

    _remoteEagleClientLogger.LogLine(
        $"[CONNECT] Starting connection attempt #{_metrics.TotalConnectAttempts}. " +
        $"Last disconnect: {FormatTimeDelta(_metrics.LastDisconnect)} ago, " +
        $"Reason: {_metrics.LastDisconnectReason}");

    // ... existing connect logic ...

    _metrics.LastSuccessfulConnect = DateTime.UtcNow;
    var connectDuration = (DateTime.UtcNow - _metrics.LastConnectAttempt.Value).TotalSeconds;
    _remoteEagleClientLogger.LogLine(
        $"[CONNECT] SUCCESS in {connectDuration:F2}s. Stream started.");
}

// In HandleStreamingCall response handling:
private async Task HandleStreamingCall() {
    // ... existing setup ...

    try {
        while (await sc.ResponseStream.MoveNext(_currentThreadToken)) {
            _metrics.LastMessageReceived = DateTime.UtcNow;

            var response = sc.ResponseStream.Current;

            _timingsLogger.LogLine(
                $"[MESSAGE] Received {response.ResponseDetailsCase}, " +
                $"ActiveShardok={_metrics.ActiveShardokGamesCount}, " +
                $"Idle={_metrics.SecondsSinceLastMessage:F1}s");

            // ... existing message handling ...
        }
    } catch (RpcException e) {
        _metrics.LastDisconnect = DateTime.UtcNow;
        _metrics.TotalDisconnects++;
        _metrics.LastDisconnectReason = e.StatusCode;

        var connectionDuration = _metrics.LastConnectionDurationSeconds ?? 0;

        _remoteEagleClientLogger.LogLine(
            $"[DISCONNECT] StatusCode={e.StatusCode}, " +
            $"Duration={connectionDuration:F1}s, " +
            $"InShardok={_metrics.IsInShardokGame}, " +
            $"ShardokGames={_metrics.ActiveShardokGamesCount}, " +
            $"IdleTime={_metrics.SecondsSinceLastMessage:F1}s, " +
            $"TotalDisconnects={_metrics.TotalDisconnects}");

        // ... existing error handling ...
    }
}

// Helper method
private string FormatTimeDelta(DateTime? time) {
    if (!time.HasValue) return "never";
    var delta = (DateTime.UtcNow - time.Value).TotalSeconds;
    if (delta < 60) return $"{delta:F1}s";
    if (delta < 3600) return $"{delta/60:F1}m";
    return $"{delta/3600:F1}h";
}

// Update when entering/leaving Shardok
private void UpdateShardokStatus(IClientConnectionSubscriber subscriber) {
    _metrics.ActiveShardokGamesCount = subscriber.ShardokViewStatuses.Count;
    _metrics.IsInShardokGame = _metrics.ActiveShardokGamesCount > 0;
}
```

**Log Analysis Script:**

Create `scripts/analyze_connection_logs.py`:

```python
#!/usr/bin/env python3
import re
from datetime import datetime
from collections import defaultdict

# Parse connection logs and generate statistics
def analyze_logs(log_file):
    disconnects = []

    with open(log_file) as f:
        for line in f:
            if '[DISCONNECT]' in line:
                # Parse: Duration=X.Xs, InShardok=true/false, IdleTime=X.Xs
                match = re.search(r'Duration=([\d.]+)s.*InShardok=(\w+).*IdleTime=([\d.]+)s', line)
                if match:
                    disconnects.append({
                        'duration': float(match.group(1)),
                        'in_shardok': match.group(2) == 'true',
                        'idle_time': float(match.group(3))
                    })

    # Statistics
    print(f"Total disconnects: {len(disconnects)}")

    durations = [d['duration'] for d in disconnects]
    if durations:
        print(f"Connection duration - Avg: {sum(durations)/len(durations):.1f}s, " +
              f"Min: {min(durations):.1f}s, Max: {max(durations):.1f}s")

    shardok_disconnects = [d for d in disconnects if d['in_shardok']]
    print(f"Shardok disconnects: {len(shardok_disconnects)} " +
          f"({100*len(shardok_disconnects)/len(disconnects):.1f}%)")

    idle_at_disconnect = [d['idle_time'] for d in disconnects]
    if idle_at_disconnect:
        print(f"Idle time at disconnect - Avg: {sum(idle_at_disconnect)/len(idle_at_disconnect):.1f}s")

if __name__ == '__main__':
    import sys
    analyze_logs(sys.argv[1] if len(sys.argv) > 1 else 'connection.log')
```

**Testing:**
- Run game for 1 hour, force several disconnects
- Verify logs contain all expected fields
- Run analysis script to validate metrics

**Success Metrics:**
- 100% of disconnects logged with duration and reason
- Can identify "2-minute pattern" if it exists
- Can correlate disconnects with Shardok gameplay

---

### 1.3 Aggressive HTTP/2 Keepalive for NAT Traversal

**Problem:** HTTP/2 keepalive (45s) may not prevent NAT/firewall timeouts (60-120s)

**Solution:** Reduce keepalive to 15s to ensure consistent NAT session refresh

**File:** `Assets/EagleConnection.cs`

```csharp
private const double KeepAliveSeconds = 15.0;  // Was 45.0

// Reason: Many consumer routers have 60-120s idle timeout for NAT sessions.
// 15-second HTTP/2 PING ensures NAT sees connection as "active"
```

**Tradeoff Analysis:**
- ✅ **Benefit:** Keeps NAT sessions alive, prevents premature disconnects
- ✅ **Benefit:** Faster detection of broken connections
- ⚠️ **Cost:** Slightly more network traffic (~8 bytes every 15s = 0.5 KB/min)
- ⚠️ **Cost:** Slightly more CPU for PING/PONG processing (negligible)

**nginx Configuration Enhancement:**

Add to Eagle endpoint in `eagle0.net`:

```nginx
location /net.eagle0.eagle.api.Eagle {
    # ... existing config ...

    # Ensure nginx doesn't buffer HTTP/2 frames
    proxy_buffering off;
    proxy_request_buffering off;

    # HTTP/2-specific settings
    http2_push_preload on;  # Allow server push if needed
}
```

**Testing:**
- Remote player plays for 30+ minutes with no input during Shardok
- Monitor connection logs for disconnects during idle periods
- Verify HTTP/2 PING frames occur every 15s (Wireshark capture)

**Success Metrics:**
- Connection stays alive for 10+ minutes during Shardok idle periods
- No disconnects with <60s connection duration
- Remote player reports no connection issues

---

## Priority 2: State Consistency & Recovery (Week 2)

### 2.1 Implement State Resync Mechanism for Shardok

**Problem:** After connection drop during Shardok, client may have inconsistent state due to delta updates

**Current Behavior:**
- Client sends `filtered_result_count` for each Shardok game
- Server sends only actions since that count
- If connection drops mid-update, counts can desync

**Solution:** Add resync request mechanism

**Protocol Extension:**

Add to `eagle.proto`:

```protobuf
message StreamGameRequest {
    int64 game_id = 1;
    int32 unfiltered_result_count = 2;

    message ShardokViewStatus {
        string shardok_game_id = 1;
        int32 filtered_result_count = 2;
        bool request_full_resync = 3;  // NEW: Request full state instead of delta
    }
    repeated ShardokViewStatus shardok_view_statuses = 3;
    // ...
}
```

**Client Implementation:**

File: `Assets/Eagle/EagleGameModel.cs`

```csharp
// Track if we need resync after reconnection
private Dictionary<string, bool> _shardokNeedsResync = new Dictionary<string, bool>();

// After connection drop:
public void MarkShardokForResync(string shardokGameId) {
    _shardokNeedsResync[shardokGameId] = true;
    _remoteEagleClientLogger.LogLine(
        $"[RESYNC] Marked Shardok game {shardokGameId} for full resync");
}

// When building StreamGameRequest:
private StreamGameRequest BuildStreamRequest() {
    var request = new StreamGameRequest {
        GameId = this.GameId,
        UnfilteredResultCount = _lastUnfilteredResultCount
    };

    foreach (var (shardokId, model) in _currentModel.ShardokGameModels) {
        var needsResync = _shardokNeedsResync.GetValueOrDefault(shardokId, false);

        request.ShardokViewStatuses.Add(new StreamGameRequest.Types.ShardokViewStatus {
            ShardokGameId = shardokId,
            FilteredResultCount = needsResync ? 0 : model.LastResultCount,
            RequestFullResync = needsResync
        });

        if (needsResync) {
            _remoteEagleClientLogger.LogLine(
                $"[RESYNC] Requesting full state for Shardok {shardokId}");
        }
    }

    return request;
}

// After successful resync:
public void OnShardokResyncComplete(string shardokGameId) {
    _shardokNeedsResync[shardokGameId] = false;
    _remoteEagleClientLogger.LogLine(
        $"[RESYNC] Completed for Shardok game {shardokGameId}");
}
```

**File:** `Assets/Eagle/PersistentClientConnection.cs`

Mark all active Shardok games for resync on disconnect:

```csharp
catch (RpcException e) {
    _metrics.LastDisconnect = DateTime.UtcNow;
    _metrics.TotalDisconnects++;

    // Mark all active Shardok games for resync
    lock (this) {
        foreach (var (gameId, subscriber) in _subscribers) {
            foreach (var shardokStatus in subscriber.ShardokViewStatuses) {
                if (subscriber is EagleGameModel model) {
                    model.MarkShardokForResync(shardokStatus.shardokGameId);
                }
            }
        }
    }

    // ... existing error handling ...
}
```

**Server Implementation:**

File: `src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala`

```scala
private def streamOneUpdate(
    request: StreamGameRequest,
    responseObserver: SyncResponseObserver
): Unit = {
  lockAndDoWithUserName { userName =>
    request.shardokViewStatuses.foreach { status =>
      if (status.requestFullResync) {
        logger.info(s"Resync requested for Shardok game ${status.shardokGameId} by $userName")
        // Send full state instead of delta by passing filteredResultCount = 0
        // Server logic already handles this case
      }
    }

    // ... existing stream logic ...
  }
}
```

**Testing:**
- Force disconnect during Shardok AI turn (mid-update)
- Verify reconnection requests resync
- Verify client state matches server state after resync
- Test with multiple simultaneous Shardok games

**Success Metrics:**
- Zero state desyncs after reconnection
- Resync completes in <2 seconds
- No duplicate actions displayed after resync

---

### 2.2 Exponential Backoff for Reconnections

**Problem:** Immediate reconnection on failure can overwhelm server or waste resources on persistent failures

**Solution:** Implement exponential backoff with jitter

**File:** `Assets/Eagle/PersistentClientConnection.cs`

```csharp
private int _consecutiveFailures = 0;
private const int MaxBackoffSeconds = 30;
private const double BackoffMultiplier = 2.0;
private const double JitterFactor = 0.2;  // ±20% randomization
private Random _backoffRandom = new Random();

public async Task Connect() {
    // Apply backoff delay if we have consecutive failures
    if (_consecutiveFailures > 0) {
        var baseDelay = Math.Min(
            Math.Pow(BackoffMultiplier, _consecutiveFailures - 1),
            MaxBackoffSeconds
        );

        // Add jitter: delay ± 20%
        var jitter = baseDelay * JitterFactor * (2 * _backoffRandom.NextDouble() - 1);
        var delaySeconds = baseDelay + jitter;

        _remoteEagleClientLogger.LogLine(
            $"[BACKOFF] Waiting {delaySeconds:F1}s before reconnect " +
            $"(failure #{_consecutiveFailures})");

        await Task.Delay((int)(delaySeconds * 1000));
    }

    if (_isConnecting) {
        _remoteEagleClientLogger.LogLine($"Connect() skipped - already connecting");
        return;
    }

    _isConnecting = true;
    try {
        _remoteEagleClientLogger.LogLine($"Connect() called");

        // ... existing connection logic ...

        // Reset consecutive failures on successful connection
        var previousFailures = _consecutiveFailures;
        _consecutiveFailures = 0;

        if (previousFailures > 0) {
            _remoteEagleClientLogger.LogLine(
                $"[BACKOFF] Reset after {previousFailures} consecutive failures");
        }

    } catch (Exception e) {
        _consecutiveFailures++;
        _remoteEagleClientLogger.LogLine(
            $"[BACKOFF] Connection failed, consecutive failures: {_consecutiveFailures}");
        throw;
    } finally {
        _isConnecting = false;
    }
}

// Also increment on RpcException:
catch (RpcException e) {
    _consecutiveFailures++;
    // ... existing handling ...
}
```

**Backoff Schedule:**

| Failure # | Base Delay | With Jitter Range |
|-----------|------------|-------------------|
| 1 | 1s | 0.8s - 1.2s |
| 2 | 2s | 1.6s - 2.4s |
| 3 | 4s | 3.2s - 4.8s |
| 4 | 8s | 6.4s - 9.6s |
| 5 | 16s | 12.8s - 19.2s |
| 6+ | 30s (max) | 24s - 36s |

**Testing:**
- Kill server during active gameplay
- Verify backoff delays match expected schedule
- Verify reconnection succeeds after server restart
- Test with rapid on/off server toggling

**Success Metrics:**
- Server load doesn't spike during outages
- Connection succeeds within 1 minute of server recovery
- Backoff resets properly after successful connection

---

### 2.3 Connection Health Monitoring & UI Indicators

**Problem:** Users don't know when connection is degraded until failure

**Solution:** Monitor connection health and display indicators

**File:** `Assets/Eagle/ConnectionHealthMonitor.cs` (new)

```csharp
public class ConnectionHealthMonitor {
    public enum HealthStatus {
        Excellent,   // <100ms latency, no packet loss
        Good,        // 100-300ms latency, <1% loss
        Fair,        // 300-500ms latency, <5% loss
        Poor,        // >500ms latency or >5% loss
        Disconnected
    }

    private Queue<long> _heartbeatLatencies = new Queue<long>();
    private const int LatencyWindowSize = 10;

    private int _heartbeatsSent = 0;
    private int _heartbeatsReceived = 0;

    public HealthStatus CurrentStatus { get; private set; } = HealthStatus.Disconnected;
    public double AverageLatencyMs { get; private set; }
    public double PacketLossPercent { get; private set; }

    public void RecordHeartbeatSent(long clientTimestamp) {
        _heartbeatsSent++;
    }

    public void RecordHeartbeatReceived(long clientTimestamp, long serverTimestamp) {
        _heartbeatsReceived++;

        var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
        var latency = now - clientTimestamp;  // Round-trip time

        _heartbeatLatencies.Enqueue(latency);
        if (_heartbeatLatencies.Count > LatencyWindowSize) {
            _heartbeatLatencies.Dequeue();
        }

        UpdateHealthStatus();
    }

    public void RecordDisconnect() {
        CurrentStatus = HealthStatus.Disconnected;
        _heartbeatsSent = 0;
        _heartbeatsReceived = 0;
    }

    private void UpdateHealthStatus() {
        // Calculate metrics
        AverageLatencyMs = _heartbeatLatencies.Count > 0
            ? _heartbeatLatencies.Average()
            : 0;

        PacketLossPercent = _heartbeatsSent > 0
            ? 100.0 * (1.0 - (double)_heartbeatsReceived / _heartbeatsSent)
            : 0;

        // Determine status
        if (PacketLossPercent > 5) {
            CurrentStatus = HealthStatus.Poor;
        } else if (AverageLatencyMs > 500) {
            CurrentStatus = HealthStatus.Poor;
        } else if (AverageLatencyMs > 300) {
            CurrentStatus = HealthStatus.Fair;
        } else if (AverageLatencyMs > 100) {
            CurrentStatus = HealthStatus.Good;
        } else {
            CurrentStatus = HealthStatus.Excellent;
        }
    }
}
```

**UI Integration:**

Add connection indicator to `EagleGameController`:

```csharp
public TextMeshProUGUI connectionStatusText;
public Image connectionStatusIcon;

void Update() {
    // ... existing update logic ...

    // Update connection status display
    var health = ConnectionHealthMonitor.Instance.CurrentStatus;
    connectionStatusText.text = $"{health} ({ConnectionHealthMonitor.Instance.AverageLatencyMs:F0}ms)";

    connectionStatusIcon.color = health switch {
        ConnectionHealthMonitor.HealthStatus.Excellent => Color.green,
        ConnectionHealthMonitor.HealthStatus.Good => Color.yellow,
        ConnectionHealthMonitor.HealthStatus.Fair => new Color(1f, 0.5f, 0f),  // Orange
        ConnectionHealthMonitor.HealthStatus.Poor => Color.red,
        ConnectionHealthMonitor.HealthStatus.Disconnected => Color.gray,
        _ => Color.white
    };
}
```

**Testing:**
- Simulate network latency using tc (Linux) or Charles Proxy
- Verify status changes from Excellent → Good → Fair → Poor as latency increases
- Verify packet loss detection

**Success Metrics:**
- Status accurately reflects connection quality
- Users report issues before complete disconnection
- Helps identify remote player's network problems

---

## Priority 3: Architectural Improvements (Week 3-4)

### 3.1 Consolidate Heartbeat Mechanisms

**Problem:** Dual-layer heartbeat (HTTP/2 + application) is redundant and inefficient

**Current State:**
- HTTP/2 keepalive: 15s (after Priority 1.3)
- Application heartbeat: 10s
- Both serve same purpose: detect broken connections

**Solution:** Remove application-level heartbeat, rely on HTTP/2 PING

**Rationale:**
- HTTP/2 PING is lower-level, more reliable for detecting transport failures
- gRPC already handles PING/PONG automatically
- Reduces code complexity and message overhead

**File:** `Assets/Eagle/PersistentClientConnection.cs`

```csharp
// REMOVE application-level heartbeat timer logic:
// - Remove HeartbeatTimerSeconds constant
// - Remove SetUpTimer() calls
// - Remove SendHeartbeatRequest() method
// - Remove TimerFired() method

// KEEP timeout detection based on last message received:
private const double MaxIdleSeconds = 30.0;  // 2x HTTP/2 keepalive interval

private void CheckForIdleTimeout() {
    var idleTime = (DateTime.UtcNow - _lastResponseReceived).TotalSeconds;

    if (idleTime > MaxIdleSeconds) {
        _remoteEagleClientLogger.LogLine(
            $"[TIMEOUT] No message for {idleTime:F1}s (max {MaxIdleSeconds}s). " +
            $"HTTP/2 keepalive failed. Reconnecting...");

        // Force reconnection
        var toCancel = _streamingCall;
        if (toCancel != null) {
            toCancel.Dispose();
            lock (this) { _streamingCall = null; }
            Task.Run(() => Connect());
        }
    }
}

// Call CheckForIdleTimeout periodically from a monitoring thread
private Thread _monitoringThread;

private void StartMonitoring() {
    _monitoringThread = new Thread(() => {
        while (!_cancellationToken.IsCancellationRequested) {
            Thread.Sleep(5000);  // Check every 5 seconds
            CheckForIdleTimeout();
        }
    });
    _monitoringThread.IsBackground = true;
    _monitoringThread.Start();
}
```

**Server-Side Enhancement:**

File: `src/main/scala/net/eagle0/eagle/Main.scala`

Enable gRPC keepalive enforcement:

```scala
val serverBuilder = ServerBuilder.forPort(grpcPort)
  .keepAliveTime(15, TimeUnit.SECONDS)       // Send PING every 15s if idle
  .keepAliveTimeout(10, TimeUnit.SECONDS)    // Close if no PONG in 10s
  .permitKeepAliveTime(10, TimeUnit.SECONDS) // Allow client PINGs every 10s
  .permitKeepAliveWithoutCalls(true)         // Allow PINGs even when no active calls

serverBuilder.addService(...)
```

**Benefits:**
- Simpler code (remove ~100 lines of heartbeat logic)
- Single source of truth for connection liveness
- Faster failure detection (15s vs 20s)
- Lower message overhead (HTTP/2 PING is 8 bytes vs heartbeat protobuf)

**Testing:**
- Verify HTTP/2 PING frames sent every 15s (Wireshark)
- Kill network connection, verify reconnect within 30s
- Test with multiple simultaneous games

**Success Metrics:**
- Same or better connection stability than before
- Reduced message count in logs
- Faster detection of dead connections

---

### 3.2 Circuit Breaker Pattern for Connection Failures

**Problem:** Repeated connection attempts to failed server waste resources and delay user feedback

**Solution:** Implement circuit breaker pattern - "fail fast" after repeated failures

**States:**
1. **CLOSED** - Normal operation, attempts connections
2. **OPEN** - Too many failures, reject attempts immediately for timeout period
3. **HALF_OPEN** - After timeout, allow one test attempt

**File:** `Assets/Eagle/ConnectionCircuitBreaker.cs` (new)

```csharp
public class ConnectionCircuitBreaker {
    public enum State { Closed, Open, HalfOpen }

    private State _state = State.Closed;
    private int _failureCount = 0;
    private DateTime? _openedAt = null;

    private const int FailureThreshold = 5;          // Open after 5 failures
    private const double OpenDurationSeconds = 60;   // Stay open for 60s
    private const double HalfOpenTestTimeout = 10;   // Test attempt timeout

    public State CurrentState => _state;

    public bool ShouldAttemptConnection() {
        lock (this) {
            switch (_state) {
                case State.Closed:
                    return true;  // Normal operation

                case State.Open:
                    // Check if timeout expired
                    if (_openedAt.HasValue &&
                        (DateTime.UtcNow - _openedAt.Value).TotalSeconds > OpenDurationSeconds) {
                        _state = State.HalfOpen;
                        Logger.LogLine("[CIRCUIT] OPEN → HALF_OPEN (timeout expired, testing)");
                        return true;  // Allow test attempt
                    }
                    Logger.LogLine("[CIRCUIT] OPEN - rejecting connection attempt");
                    return false;  // Still in timeout

                case State.HalfOpen:
                    return true;  // Test attempt

                default:
                    return true;
            }
        }
    }

    public void RecordSuccess() {
        lock (this) {
            if (_state != State.Closed) {
                Logger.LogLine($"[CIRCUIT] {_state} → CLOSED (connection succeeded)");
            }
            _state = State.Closed;
            _failureCount = 0;
            _openedAt = null;
        }
    }

    public void RecordFailure() {
        lock (this) {
            _failureCount++;

            if (_state == State.HalfOpen) {
                // Test attempt failed, reopen circuit
                _state = State.Open;
                _openedAt = DateTime.UtcNow;
                Logger.LogLine($"[CIRCUIT] HALF_OPEN → OPEN (test failed)");
            } else if (_failureCount >= FailureThreshold && _state == State.Closed) {
                _state = State.Open;
                _openedAt = DateTime.UtcNow;
                Logger.LogLine($"[CIRCUIT] CLOSED → OPEN (failures={_failureCount})");
            }
        }
    }
}
```

**Integration with PersistentClientConnection:**

```csharp
private ConnectionCircuitBreaker _circuitBreaker = new ConnectionCircuitBreaker();

public async Task Connect() {
    if (!_circuitBreaker.ShouldAttemptConnection()) {
        _remoteEagleClientLogger.LogLine(
            "[CIRCUIT] Connection attempt blocked by circuit breaker");
        return;  // Don't attempt connection
    }

    // ... existing backoff logic ...

    try {
        // ... existing connection logic ...

        _circuitBreaker.RecordSuccess();
    } catch (Exception e) {
        _circuitBreaker.RecordFailure();
        throw;
    }
}
```

**User Feedback:**

Show circuit breaker state in UI:

```csharp
// In EagleGameController or ConnectionUI
if (circuitBreaker.CurrentState == ConnectionCircuitBreaker.State.Open) {
    ShowBanner("Server unavailable. Retrying in 60 seconds...", Color.red);
} else if (circuitBreaker.CurrentState == ConnectionCircuitBreaker.State.HalfOpen) {
    ShowBanner("Testing connection...", Color.yellow);
}
```

**Testing:**
- Kill server, verify circuit opens after 5 failures
- Verify no connection attempts during 60s timeout
- Restart server during timeout, verify test attempt after 60s
- Verify circuit closes on successful test

**Success Metrics:**
- No wasted connection attempts during server outages
- Users see clear "server down" message
- Connection restored within 70s of server recovery (60s timeout + 10s test)

---

### 3.3 Server-Side Connection Tracking & Metrics

**Problem:** No visibility into server-side connection health and patterns

**Solution:** Track connections per user and expose metrics endpoint

**File:** `src/main/scala/net/eagle0/eagle/service/ConnectionTracker.scala` (new)

```scala
package net.eagle0.eagle.service

import java.time.Instant
import scala.collection.concurrent.TrieMap

case class ConnectionInfo(
  userName: String,
  connectedAt: Instant,
  var lastActivity: Instant,
  var messageCount: Long = 0,
  var activeGameIds: Set[Long] = Set.empty,
  var activeShardokGames: Int = 0
)

object ConnectionTracker {
  private val connections = new TrieMap[String, ConnectionInfo]()

  def recordConnection(userName: String): Unit = {
    val now = Instant.now()
    connections.put(userName, ConnectionInfo(userName, now, now))
    println(s"[CONNECTION] User $userName connected. Total active: ${connections.size}")
  }

  def recordDisconnection(userName: String): Unit = {
    connections.remove(userName) match {
      case Some(info) =>
        val duration = java.time.Duration.between(info.connectedAt, Instant.now())
        println(s"[CONNECTION] User $userName disconnected after ${duration.toMinutes}min. " +
                s"Messages: ${info.messageCount}")
      case None =>
        println(s"[CONNECTION] User $userName disconnected (no info)")
    }
    println(s"[CONNECTION] Total active: ${connections.size}")
  }

  def recordActivity(userName: String, gameId: Option[Long] = None, shardokGames: Int = 0): Unit = {
    connections.get(userName).foreach { info =>
      info.lastActivity = Instant.now()
      info.messageCount += 1
      gameId.foreach(id => info.activeGameIds += id)
      if (shardokGames > 0) info.activeShardokGames = shardokGames
    }
  }

  def getMetrics: Map[String, Any] = Map(
    "total_connections" -> connections.size,
    "connections_by_user" -> connections.map { case (user, info) =>
      Map(
        "user" -> user,
        "connected_minutes" -> java.time.Duration.between(info.connectedAt, Instant.now()).toMinutes,
        "idle_seconds" -> java.time.Duration.between(info.lastActivity, Instant.now()).getSeconds,
        "message_count" -> info.messageCount,
        "active_games" -> info.activeGameIds.size,
        "shardok_games" -> info.activeShardokGames
      )
    }.toList
  )

  // Clean up stale connections (no activity for 10 minutes)
  def cleanupStale(): Unit = {
    val now = Instant.now()
    val staleThreshold = java.time.Duration.ofMinutes(10)

    connections.foreach { case (user, info) =>
      if (java.time.Duration.between(info.lastActivity, now).compareTo(staleThreshold) > 0) {
        println(s"[CONNECTION] Removing stale connection for $user")
        connections.remove(user)
      }
    }
  }
}
```

**Integration with EagleServiceImpl:**

```scala
private class SyncStreamObserver(responseObserver: SyncResponseObserver)
    extends StreamObserver[UpdateStreamRequest] {

  private val userName = AuthorizationUtils.userName
  ConnectionTracker.recordConnection(userName)

  override def onNext(value: UpdateStreamRequest): Unit = try {
    value.requestDetails match {
      case UpdateStreamRequest.RequestDetails.StreamGameRequest(req) =>
        ConnectionTracker.recordActivity(
          userName,
          Some(req.gameId),
          req.shardokViewStatuses.size
        )
        streamOneUpdate(req, responseObserver)

      // ... other cases ...
    }
  } catch { /* ... */ }

  override def onError(t: Throwable): Unit = {
    ConnectionTracker.recordDisconnection(userName)
    // ... existing error handling ...
  }

  override def onCompleted(): Unit = {
    ConnectionTracker.recordDisconnection(userName)
    // ... existing completion handling ...
  }
}
```

**Metrics Endpoint:**

Add to `EagleServiceImpl.scala`:

```scala
// Schedule periodic cleanup
val cleanupScheduler = Executioncontext.global
scala.concurrent.Future {
  while (true) {
    Thread.sleep(60000)  // Every minute
    ConnectionTracker.cleanupStale()
  }
}(cleanupScheduler)

// Expose metrics (could be via separate admin endpoint or logging)
def logConnectionMetrics(): Unit = {
  val metrics = ConnectionTracker.getMetrics
  println(s"[METRICS] ${metrics}")
}
```

**Monitoring Dashboard:**

Create simple web endpoint or log-based dashboard:

```bash
# Example: Parse metrics from server logs
grep "\\[METRICS\\]" server.log | tail -1 | jq '.connections_by_user'
```

**Testing:**
- Connect with multiple users
- Verify metrics show all connections
- Disconnect users, verify metrics update
- Let connections go idle, verify stale cleanup

**Success Metrics:**
- Can see all active connections in real-time
- Can identify users with connection problems
- Can correlate server-side and client-side connection events

---

## Priority 4: Long-Term Enhancements (Week 5+)

### 4.1 Adaptive Connection Parameters

**Problem:** Fixed timeouts/keepalives don't adapt to varying network conditions

**Solution:** Dynamically adjust based on observed latency and packet loss

**File:** `Assets/Eagle/AdaptiveConnectionManager.cs` (new)

```csharp
public class AdaptiveConnectionManager {
    private ConnectionHealthMonitor _healthMonitor;

    // Adaptive parameters
    public double CurrentKeepAliveSeconds { get; private set; } = 15.0;
    public double CurrentTimeoutSeconds { get; private set; } = 300.0;

    // Ranges
    private const double MinKeepAlive = 10.0;
    private const double MaxKeepAlive = 60.0;
    private const double MinTimeout = 120.0;
    private const double MaxTimeout = 600.0;

    public void UpdateParameters() {
        var health = _healthMonitor.CurrentStatus;
        var latency = _healthMonitor.AverageLatencyMs;

        // Adapt keepalive based on connection quality
        switch (health) {
            case ConnectionHealthMonitor.HealthStatus.Excellent:
                // Stable connection, can use longer keepalive
                CurrentKeepAliveSeconds = Math.Min(CurrentKeepAliveSeconds * 1.2, MaxKeepAlive);
                break;

            case ConnectionHealthMonitor.HealthStatus.Poor:
                // Unstable connection, use aggressive keepalive
                CurrentKeepAliveSeconds = Math.Max(CurrentKeepAliveSeconds * 0.8, MinKeepAlive);
                break;
        }

        // Adapt timeout based on latency
        // Timeout should be at least 10x average latency
        var minTimeoutForLatency = Math.Max(10 * latency / 1000.0, MinTimeout);
        CurrentTimeoutSeconds = Math.Clamp(minTimeoutForLatency, MinTimeout, MaxTimeout);
    }
}
```

**Implementation Notes:**
- Would require gRPC channel to be recreated when parameters change
- Complex to implement correctly
- Marginal benefit over fixed values
- **Recommendation:** Defer unless data shows need

---

### 4.2 WebSocket Fallback Option

**Problem:** Some networks/proxies may have issues with gRPC/HTTP2

**Solution:** Implement WebSocket transport as fallback

**Complexity:** HIGH - requires significant protocol changes

**Benefits:**
- Better compatibility with restrictive networks
- May work through more proxies/firewalls

**Drawbacks:**
- Duplicates transport layer code
- Harder to maintain two implementations
- Most modern networks support HTTP/2

**Recommendation:** Only implement if significant user segment cannot connect via gRPC

---

### 4.3 Client-Side Prediction for Shardok

**Problem:** High latency makes Shardok gameplay feel sluggish

**Solution:** Predict action results locally, reconcile with server

**Implementation Overview:**
1. Client sends command to server
2. Client immediately predicts result and updates UI
3. Server processes command and sends actual result
4. Client reconciles: if different, apply correction

**Complexity:** VERY HIGH - requires deterministic simulation on client

**Benefits:**
- Responsive gameplay even with 300ms+ latency
- Better user experience for remote players

**Drawbacks:**
- Complex reconciliation logic
- Potential for "rollback" visual artifacts
- Requires shipping Shardok simulation to client

**Recommendation:** Consider for future major release if player base grows internationally

---

## Testing Strategy

### Automated Testing

1. **Unit Tests:**
   - Connection state machine transitions
   - Backoff calculation
   - Circuit breaker state transitions
   - Metrics calculations

2. **Integration Tests:**
   - Full connect/disconnect/reconnect cycles
   - State resync after forced disconnect
   - Multiple simultaneous Shardok games

3. **Load Testing:**
   - 100+ concurrent connections
   - Rapid connect/disconnect cycles
   - Large message bursts during Shardok

### Manual Testing Scenarios

1. **Network Simulation:**
   - Use `tc` (Linux traffic control) or Charles Proxy
   - Test with 50ms, 200ms, 500ms latency
   - Test with 1%, 5%, 10% packet loss
   - Test with bandwidth limits (1Mbps, 512Kbps)

2. **Failure Injection:**
   - Kill server during Eagle turn
   - Kill server during Shardok combat
   - Kill server during AI processing
   - Restart server multiple times rapidly
   - Disconnect client network for 30s, 60s, 120s

3. **Remote Player Simulation:**
   - Test from actual remote network (not local)
   - Test through VPN
   - Test from mobile hotspot
   - Test from coffee shop WiFi

### Success Criteria

| Metric | Current (Baseline) | Target (After Implementation) |
|--------|-------------------|-------------------------------|
| Connection uptime during 1hr session | Unknown | >99% |
| Time to detect disconnect | 20s | <30s (with HTTP/2 keepalive) |
| Time to reconnect after server restart | Variable | <70s (60s circuit + 10s test) |
| State desyncs after reconnect | Unknown | 0% |
| Connection drops during 30min idle Shardok | Unknown | 0% |
| User-visible connection quality indicator | No | Yes |
| Server-side connection visibility | No | Yes (metrics dashboard) |

---

## Implementation Timeline

### Week 1: Critical Fixes & Diagnostics
- **Day 1-2:** Fix Shardok security vulnerability, deploy to production
- **Day 3-4:** Implement comprehensive connection logging
- **Day 5:** Reduce HTTP/2 keepalive to 15s, monitor results

**Deliverable:** Security fix deployed, detailed logs capturing all disconnects

### Week 2: State Consistency & Recovery
- **Day 1-2:** Implement Shardok state resync mechanism (client + server)
- **Day 3:** Implement exponential backoff
- **Day 4-5:** Implement connection health monitoring and UI indicators

**Deliverable:** Zero state desyncs after reconnection, visible connection quality

### Week 3-4: Architectural Improvements
- **Day 1-2:** Consolidate heartbeat mechanisms (remove application-level)
- **Day 3:** Implement circuit breaker pattern
- **Day 4-5:** Server-side connection tracking and metrics

**Deliverable:** Simplified codebase, circuit breaker prevents wasted retries

### Week 5+: Long-Term Enhancements
- **As needed:** Adaptive connection parameters
- **Future release:** WebSocket fallback
- **Future release:** Client-side prediction

**Deliverable:** Production-ready resilient connection system

---

## Monitoring & Observability

### Key Metrics to Track

1. **Connection Stability:**
   - Mean time between disconnects (MTBD)
   - Connection success rate on first attempt
   - Average connection duration

2. **Network Quality:**
   - Average latency (p50, p95, p99)
   - Packet loss percentage
   - Jitter (latency variance)

3. **Failure Recovery:**
   - Time to detect disconnect
   - Time to reconnect
   - Circuit breaker activation frequency

4. **State Consistency:**
   - Resync request frequency
   - Resync success rate
   - Average resync duration

5. **User Experience:**
   - Gameplay sessions interrupted by disconnects
   - User complaints about connection
   - Average connection health status

### Logging Strategy

**Structured Logging Format:**

```
[TIMESTAMP] [LEVEL] [COMPONENT] [EVENT] key1=value1 key2=value2 ...
```

**Example:**
```
[2024-11-30 14:23:15.123] [INFO] [CONNECT] SUCCESS duration=2.3s attempts=1 backoff=0s
[2024-11-30 14:25:30.456] [WARN] [DISCONNECT] DETECTED reason=StatusCode.Cancelled duration=135.3s idle=45.2s shardok=true
[2024-11-30 14:25:32.789] [INFO] [BACKOFF] WAITING delay=1.2s failures=1
[2024-11-30 14:25:34.012] [INFO] [RESYNC] REQUESTED game=shardok_12345
```

**Log Aggregation:**

Use ELK stack or similar:
- **Elasticsearch:** Store all logs
- **Logstash:** Parse and index
- **Kibana:** Visualize and alert

**Alerts:**

1. **Critical:** Circuit breaker opened (server down)
2. **Warning:** Connection health drops below "Good" for >5 minutes
3. **Warning:** >10% of connection attempts fail
4. **Info:** Resync requested (possible state issue)

---

## Rollback Plan

Each change should be independently deployable and revertible:

1. **Feature Flags:**
   ```csharp
   public static class FeatureFlags {
       public static bool UseAggressiveKeepAlive => GetFlag("aggressive_keepalive", true);
       public static bool UseCircuitBreaker => GetFlag("circuit_breaker", true);
       public static bool UseShardokResync => GetFlag("shardok_resync", true);
   }
   ```

2. **Gradual Rollout:**
   - Deploy to 10% of users first
   - Monitor metrics for 24 hours
   - If stable, roll out to 50%, then 100%

3. **Quick Revert:**
   - Keep previous nginx config backed up
   - Feature flags can disable changes without redeployment
   - Database/state changes are additive only (no breaking schema changes)

---

## Security Considerations

### Authentication & Authorization

1. **Shardok Internal Interface:**
   - MUST remove public access or add authentication
   - Consider mutual TLS for internal Eagle ↔ Shardok communication

2. **Rate Limiting:**
   ```nginx
   # Per-user rate limits
   limit_req_zone $remote_user zone=per_user:10m rate=100r/s;
   limit_req zone=per_user burst=20 nodelay;

   # Global rate limit
   limit_req_zone $server_name zone=global:10m rate=1000r/s;
   limit_req zone=global burst=100 nodelay;
   ```

3. **DDoS Protection:**
   - nginx rate limiting (above)
   - Connection limit per IP: `limit_conn_zone $binary_remote_addr zone=addr:10m; limit_conn addr 5;`
   - Consider Cloudflare or similar CDN/DDoS protection

### Input Validation

1. **Command Validation:**
   - Server MUST validate all commands from client
   - Check token/state before executing
   - Sanitize all user input

2. **State Validation:**
   - Verify `filtered_result_count` is within valid range
   - Reject obviously corrupted state

### Audit Logging

Log all security-relevant events:
- Failed authentication attempts
- Rate limit violations
- Invalid command attempts
- Unusual connection patterns

---

## Performance Considerations

### Client-Side

1. **Memory Usage:**
   - Connection metrics: ~1KB per connection
   - Log buffering: limit to last 1000 entries, ~100KB
   - State resync: temporary spike during full state load

2. **CPU Usage:**
   - Heartbeat consolidation: reduces CPU by ~5-10%
   - Metrics calculation: negligible (<1% CPU)
   - Backoff jitter calculation: negligible

3. **Network Bandwidth:**
   - HTTP/2 keepalive at 15s: ~32 bytes/min (negligible)
   - Removing application heartbeat: saves ~40 bytes/min
   - State resync: one-time spike of full game state (~50-200KB)

### Server-Side

1. **Memory Usage:**
   - Connection tracking: ~2KB per connection
   - 1000 concurrent users = 2MB (negligible)

2. **CPU Usage:**
   - Connection cleanup: runs every 60s, <1ms
   - Metrics calculation: on-demand, <10ms

3. **Database:**
   - No new database requirements
   - All metrics in-memory only

---

## Conclusion

This comprehensive resilience plan addresses all identified connection issues through a prioritized, incremental approach:

**Immediate Impact (Week 1):**
- Security vulnerability fixed
- Can measure and diagnose connection issues
- NAT timeout mitigation deployed

**Short-Term Gains (Week 2):**
- Zero state desyncs after reconnection
- Graceful failure recovery
- User visibility into connection quality

**Long-Term Benefits (Week 3-4):**
- Simpler, more maintainable codebase
- Better resource utilization during failures
- Comprehensive server-side observability

**Success will be measured by:**
- >99% connection uptime during gameplay
- Zero user-reported state inconsistencies
- <1 minute to recover from server outages
- Clear visibility into connection health for both users and operators

The plan is designed for incremental deployment, easy rollback, and continuous monitoring to ensure each change improves the system without introducing new issues