mirror of
https://github.com/nolen777/eagle0.git
synced 2026-07-28 20:55:44 +00:00
Organize documentation into docs/ folder (#4598)
Create docs/ folder at repo root and move documentation files: - CONNECTION_ARCHITECTURE.md (new comprehensive connection docs) - COMMAND_PROTO_USAGE_ANALYSIS.md - DEPROTO_PLAN.md - SCALA3_MODERNIZATION.md - actions-model-usage-analysis.md - occupants-optimization-report.md - scala3-reflection-issues.md CLAUDE.md remains at root (project instructions for Claude Code). Connection architecture documentation includes: - gRPC bidirectional streaming protocol details - Client-side connection management (PersistentClientConnection) - Server-side implementation (EagleServiceImpl) - nginx proxy configuration and timeouts - Timeout settings across all layers (client, nginx, server) - Eagle ↔ Shardok communication flow Critical findings: - 🔴 SECURITY: Shardok internal interface exposed without auth in nginx config - Mystery "2-minute timeout" doesn't exist in code (all timeouts are 5-20 minutes) - No state resync mechanism after connection drops during Shardok - Inefficient dual-layer heartbeat (HTTP/2 + application level) Hypotheses for remote player connection issues: - Most likely: NAT/firewall timeout at player's router/ISP (60-120s) - HTTP/2 keepalive (45s) may not be frequent enough to keep NAT alive - Shardok's bursty traffic pattern may appear "idle" at transport layer Recommendations: 1. Fix Shardok internal interface security vulnerability 2. Add precise connection drop logging with timestamps 3. Reduce HTTP/2 keepalive from 45s to 15s 4. Get network diagnostics from affected remote player 5. Implement state resync mechanism for Shardok 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,645 @@
|
||||
# Eagle0 Client/Server Connection Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
Eagle0 uses a bidirectional streaming gRPC connection between the Unity C# client and the Scala Eagle server. The Shardok tactical combat server (C++) is not directly exposed to clients; instead, Eagle server acts as a proxy, communicating with Shardok via internal gRPC and forwarding updates to clients.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
Unity Client (C#) <--[gRPC/HTTP2/TLS]--> nginx <--[HTTP]--> Eagle Server (Scala) <--[gRPC]--> Shardok Server (C++)
|
||||
```
|
||||
|
||||
## Protocol Definition
|
||||
|
||||
**Service:** `src/main/protobuf/net/eagle0/eagle/api/eagle.proto`
|
||||
|
||||
```protobuf
|
||||
service Eagle {
|
||||
rpc StreamUpdates(stream UpdateStreamRequest) returns (stream UpdateStreamResponse) {}
|
||||
}
|
||||
```
|
||||
|
||||
This is a bidirectional streaming RPC where:
|
||||
- **Client can send:** StreamGameRequest, HeartbeatRequest, EnterLobbyRequest, PostCommandRequest, ErrorRequest, etc.
|
||||
- **Server can send:** GameUpdate, HeartbeatResponse, LobbyResponse, etc.
|
||||
|
||||
### Key Message Types
|
||||
|
||||
**UpdateStreamRequest:**
|
||||
- `StreamGameRequest`: Subscribe to game updates, includes:
|
||||
- `game_id`: Which game to watch
|
||||
- `unfiltered_result_count`: Last known Eagle action result count (for delta updates)
|
||||
- `shardok_view_statuses[]`: Array of Shardok game states with their known result counts
|
||||
- `streaming_text_statuses[]`: LLM text streaming progress
|
||||
- `HeartbeatRequest`: Keepalive ping with client timestamp
|
||||
- `PostCommandRequest`: Send Eagle or Shardok commands
|
||||
|
||||
**UpdateStreamResponse:**
|
||||
- `GameUpdate`: Contains either Eagle action results or Shardok action results
|
||||
- `HeartbeatResponse`: Keepalive pong with server timestamp
|
||||
|
||||
## Client Implementation (C#/Unity)
|
||||
|
||||
### Connection Setup
|
||||
|
||||
**File:** `Assets/EagleConnection.cs`
|
||||
|
||||
```csharp
|
||||
private const double KeepAliveSeconds = 45.0;
|
||||
|
||||
_channel = GrpcChannel.ForAddress("https://" + url,
|
||||
new GrpcChannelOptions {
|
||||
HttpHandler = new YetAnotherHttpHandler {
|
||||
Http2Only = true,
|
||||
Http2KeepAliveInterval = TimeSpan.FromSeconds(KeepAliveSeconds) // 45 seconds
|
||||
},
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
**Key Settings:**
|
||||
- **HTTP/2 KeepAlive Interval:** 45 seconds
|
||||
- **Connection URL:** `https://<url>` (TLS required via nginx)
|
||||
- **HTTP Handler:** YetAnotherHttpHandler (Unity-compatible HTTP/2 implementation)
|
||||
|
||||
### Persistent Connection Management
|
||||
|
||||
**File:** `Assets/Eagle/PersistentClientConnection.cs`
|
||||
|
||||
**Constants:**
|
||||
```csharp
|
||||
private const double TimeoutSeconds = 300.0; // 5 minutes
|
||||
private const double HeartbeatTimerSeconds = 10.0; // 10 seconds
|
||||
```
|
||||
|
||||
**Connection Lifecycle:**
|
||||
|
||||
1. **Connect()** (line 91)
|
||||
- Creates bidirectional streaming call with 300-second deadline
|
||||
- Starts background thread `HandleStreamingCall()` to process responses
|
||||
- Sends `StreamGameRequest` for each subscribed game
|
||||
- Prevents concurrent connection attempts with `_isConnecting` flag
|
||||
|
||||
2. **HandleStreamingCall()** (line 466)
|
||||
- Runs in dedicated thread
|
||||
- Continuously reads from `ResponseStream`
|
||||
- Updates `_lastResponseReceived` timestamp on every message
|
||||
- Sets up heartbeat timer via `SetUpTimer()` (line 475, 482)
|
||||
- On exception: attempts reconnection based on status code
|
||||
|
||||
3. **HeartbeatTimer Logic** (line 648)
|
||||
- Timer fires every 10 seconds
|
||||
- **If no response > 20 seconds:** Forces reconnection by disposing stream and calling `Connect()`
|
||||
- **If no response 10-20 seconds:** Sends heartbeat and creates new timer
|
||||
- **If response < 10 seconds ago:** Heartbeat only sent when stream has new data
|
||||
|
||||
4. **Reconnection Handling** (line 579)
|
||||
- **StatusCode.Cancelled:** Check if intentional cancellation, otherwise reconnect
|
||||
- **StatusCode.Internal:** Immediate reconnection
|
||||
- **StatusCode.DeadlineExceeded:** Immediate reconnection (300-second deadline hit)
|
||||
- **StatusCode.Unavailable:** Wait 5 seconds, then reconnect
|
||||
- **ObjectDisposedException:** Immediate reconnection
|
||||
|
||||
### Subscription Model
|
||||
|
||||
The client maintains a `Dictionary<EagleGameId, IClientConnectionSubscriber>` of active game subscriptions. When a game is subscribed:
|
||||
|
||||
1. Client sends `StreamGameRequest` with game ID and last known state
|
||||
2. Server sends incremental updates (only new action results since `unfiltered_result_count`)
|
||||
3. For Shardok games: Client sends `shardok_view_statuses[]` array with each Shardok game's last known result count
|
||||
|
||||
### Shardok Update Flow
|
||||
|
||||
**For Shardok tactical battles:**
|
||||
|
||||
1. Client tracks multiple concurrent Shardok games via `ShardokViewStatuses` array
|
||||
2. Each `StreamGameRequest` includes:
|
||||
```csharp
|
||||
ShardokViewStatus {
|
||||
string shardok_game_id
|
||||
int32 filtered_result_count // Last known action count for this Shardok game
|
||||
}
|
||||
```
|
||||
3. Server responds with `ShardokActionResultResponse` containing:
|
||||
```protobuf
|
||||
message SingleShardokGameResultResponse {
|
||||
string shardok_game_id
|
||||
repeated ActionResultView action_result_views // Only new actions since filtered_result_count
|
||||
int32 new_result_view_count
|
||||
AvailableCommands available_commands
|
||||
}
|
||||
```
|
||||
|
||||
This delta update mechanism reduces bandwidth during Shardok gameplay.
|
||||
|
||||
## Server Implementation (Scala)
|
||||
|
||||
**File:** `src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala`
|
||||
|
||||
### StreamUpdates Method
|
||||
|
||||
```scala
|
||||
def streamUpdates(responseObserver: StreamObserver[UpdateStreamResponse]):
|
||||
StreamObserver[UpdateStreamRequest] =
|
||||
new SyncStreamObserver(new SyncResponseObserver(AuthorizationUtils.userName, responseObserver))
|
||||
```
|
||||
|
||||
Creates a bidirectional stream handler that:
|
||||
1. Wraps the response observer with `SyncResponseObserver` (username-tagged)
|
||||
2. Returns `SyncStreamObserver` to handle client requests
|
||||
|
||||
### Request Handling
|
||||
|
||||
**SyncStreamObserver.onNext()** dispatches based on request type:
|
||||
- `StreamGameRequest` → `streamOneUpdate()` → delegates to GamesManager or CustomBattleManager
|
||||
- `HeartbeatRequest` → `handleHeartbeat()` → sends HeartbeatResponse immediately
|
||||
- `PostCommandRequest` → `postCommand()` → processes Eagle/Shardok commands
|
||||
- `EnterLobbyRequest` → adds user to lobby subscribers, sends lobby state
|
||||
|
||||
### Heartbeat Handling
|
||||
|
||||
```scala
|
||||
private def handleHeartbeat(clientTimestamp: Long, responseObserver: SyncResponseObserver): Unit = {
|
||||
println(s"got heartbeat from $userName. Client timestamp $clientTimestamp,
|
||||
server timestamp ${System.currentTimeMillis()}")
|
||||
responseObserver.onNext(
|
||||
UpdateStreamResponse(
|
||||
responseDetails = HeartbeatResponse(serverTimestamp = System.currentTimeMillis())
|
||||
)
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Server immediately responds to heartbeats with server timestamp. No server-side timeout enforcement visible in this layer.
|
||||
|
||||
### Eagle ↔ Shardok Communication
|
||||
|
||||
When client sends `ShardokCommand` or `ShardokPlacementCommands`:
|
||||
|
||||
1. Eagle server checks if game is in `gamesManager` or `customBattleManager`
|
||||
2. Routes to appropriate manager's `postShardokCommand()` or `postPlacementCommands()`
|
||||
3. Manager communicates with Shardok server via internal gRPC
|
||||
4. Shardok action results flow back through Eagle server to client via `ShardokActionResultResponse`
|
||||
|
||||
## Network Layer: nginx Proxy
|
||||
|
||||
**File:** `/Users/dancrosby/CodingProjects/github/eagle0/eagle0.net` (sibling repo)
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```nginx
|
||||
location /net.eagle0.eagle.api.Eagle {
|
||||
auth_basic "User Area";
|
||||
auth_basic_user_file /etc/nginx/.htpasswd;
|
||||
grpc_pass grpc://192.168.123.100:40032;
|
||||
proxy_connect_timeout 1200s; # 20 minutes
|
||||
proxy_send_timeout 1200s; # 20 minutes
|
||||
proxy_read_timeout 1200s; # 20 minutes
|
||||
grpc_send_timeout 1200s; # 20 minutes
|
||||
grpc_read_timeout 1200s; # 20 minutes
|
||||
grpc_socket_keepalive on;
|
||||
}
|
||||
|
||||
location /net.eagle0.common.ShardokInternalInterface {
|
||||
grpc_pass grpc://192.168.123.100:40042;
|
||||
}
|
||||
```
|
||||
|
||||
**Key Settings:**
|
||||
- **Protocol:** HTTP/2 over TLS (line 6: `listen 443 ssl http2`)
|
||||
- **Eagle Server:** `grpc://192.168.123.100:40032`
|
||||
- **Shardok Server:** `grpc://192.168.123.100:40042` (⚠️ EXPOSED TO CLIENTS!)
|
||||
- **Timeouts:** 1200 seconds (20 minutes) - very generous
|
||||
- **Socket Keepalive:** Enabled for Eagle endpoint
|
||||
- **TLS:** Let's Encrypt certificate for eagle0.net
|
||||
|
||||
**Analysis:**
|
||||
- nginx timeouts are NOT the problem (20 minutes is very generous)
|
||||
- HTTP/2 is properly configured
|
||||
- Socket keepalive is enabled
|
||||
- ⚠️ **CRITICAL:** Shardok internal interface is exposed at `/net.eagle0.common.ShardokInternalInterface` without auth or timeouts!
|
||||
|
||||
## Server Implementation Details
|
||||
|
||||
**File:** `src/main/scala/net/eagle0/eagle/Main.scala`
|
||||
|
||||
```scala
|
||||
val serverBuilder = ServerBuilder.forPort(grpcPort) // Default port: 40032
|
||||
```
|
||||
|
||||
**Server Configuration:**
|
||||
- Uses default `ServerBuilder.forPort()` with NO custom timeout settings
|
||||
- gRPC Java defaults:
|
||||
- `maxConnectionIdle`: infinity (no automatic closure of idle connections)
|
||||
- `maxConnectionAge`: infinity (connections never expire)
|
||||
- `keepAliveTime`: 2 hours (but NOT enforced by default)
|
||||
- `keepAliveTimeout`: 20 seconds
|
||||
|
||||
**Key Finding:** Server does NOT enforce any timeouts that would cause 2-minute drops.
|
||||
|
||||
## Complete Timeout Settings Summary
|
||||
|
||||
| Layer | Component | Setting | Value | Notes |
|
||||
|-------|-----------|---------|-------|-------|
|
||||
| **Client** | HTTP/2 Handler | KeepAlive Interval | 45s | HTTP/2 PING frames (YetAnotherHttpHandler) |
|
||||
| **Client** | gRPC Call | Deadline | 300s (5 min) | Per-call timeout for StreamUpdates() |
|
||||
| **Client** | Connection | Heartbeat Timer | 10s | Application-level ping when idle |
|
||||
| **Client** | Connection | Heartbeat Timeout | 20s | Force reconnect if no response |
|
||||
| **Client** | Reconnection | Unavailable Retry Delay | 5s | Backoff when server unavailable |
|
||||
| **nginx** | Proxy | connect_timeout | 1200s (20 min) | Connection establishment |
|
||||
| **nginx** | Proxy | send_timeout | 1200s (20 min) | Time to send request |
|
||||
| **nginx** | Proxy | read_timeout | 1200s (20 min) | Time to read response |
|
||||
| **nginx** | gRPC | send_timeout | 1200s (20 min) | gRPC-specific send |
|
||||
| **nginx** | gRPC | read_timeout | 1200s (20 min) | gRPC-specific read |
|
||||
| **Server** | gRPC | maxConnectionIdle | ∞ | No forced idle timeout |
|
||||
| **Server** | gRPC | maxConnectionAge | ∞ | No forced age limit |
|
||||
| **Server** | gRPC | keepAliveTime | 2 hours (unused) | Not enforced |
|
||||
|
||||
**Historical Note:** Client deadline was increased from 55s to 300s in commit 7f5b246d59 (Aug 2024) to "extend the timeout".
|
||||
|
||||
## Identified Issues
|
||||
|
||||
### 1. **MYSTERY: Where is the "2 minute timeout"?**
|
||||
|
||||
**User reports:** "Connection drops every 2min or so" and "I (think I) put it on a timeout slightly under 2min"
|
||||
|
||||
**Current settings in code:**
|
||||
- Client gRPC deadline: **300 seconds (5 minutes)**
|
||||
- nginx timeouts: **1200 seconds (20 minutes)**
|
||||
- Server timeouts: **infinity (none)**
|
||||
|
||||
**FINDINGS:**
|
||||
- ✅ nginx has very generous 20-minute timeouts - NOT the problem
|
||||
- ✅ Server has no forced timeouts - NOT the problem
|
||||
- ✅ Client deadline is 5 minutes - NOT 2 minutes
|
||||
|
||||
**HYPOTHESIS:**
|
||||
1. **User may be misremembering** - there is NO 2-minute timeout in the code
|
||||
2. **OR: External network infrastructure** - ISP/router/firewall between remote player and nginx may have 2-minute idle timeout
|
||||
3. **OR: The "2 minutes" observation is approximate** - actual drops may be at 5 minutes (deadline) or caused by other issues
|
||||
|
||||
**RECOMMENDATION:** Add detailed logging with precise timestamps to measure actual time between drops.
|
||||
|
||||
### 2. **🔴 CRITICAL BUG: Shardok Internal Interface Exposed Without Auth!**
|
||||
|
||||
**nginx configuration (lines 53-57):**
|
||||
```nginx
|
||||
location /net.eagle0.common.ShardokInternalInterface {
|
||||
grpc_pass grpc://192.168.123.100:40042;
|
||||
}
|
||||
```
|
||||
|
||||
**PROBLEMS:**
|
||||
- ❌ No `auth_basic` directive (Eagle endpoint has auth on line 37-38)
|
||||
- ❌ No timeout settings (Eagle endpoint has timeouts on lines 44-49)
|
||||
- ❌ No socket keepalive (Eagle endpoint has it on line 49)
|
||||
- ❌ Exposes internal Shardok interface directly to internet
|
||||
|
||||
**IMPACT:**
|
||||
- Anyone can bypass Eagle server and call Shardok directly
|
||||
- No authentication or authorization checks
|
||||
- Contradicts architecture claim that "Shardok server is not exposed directly to the client"
|
||||
- Potential security vulnerability
|
||||
|
||||
**RECOMMENDATION:** Either:
|
||||
1. Remove this location block entirely (Shardok should only be called by Eagle server internally)
|
||||
2. OR add same auth and timeout settings as Eagle endpoint
|
||||
|
||||
### 3. **HTTP/2 KeepAlive vs Application Heartbeat Mismatch**
|
||||
|
||||
**Current settings:**
|
||||
- **HTTP/2 KeepAlive:** 45 seconds (YetAnotherHttpHandler)
|
||||
- **Application Heartbeat:** 10 seconds (PersistentClientConnection)
|
||||
- **Heartbeat Timeout:** 20 seconds (force reconnect)
|
||||
|
||||
**ISSUES:**
|
||||
- HTTP/2 layer sends PING every 45 seconds
|
||||
- Application layer sends heartbeat every 10 seconds (if no updates)
|
||||
- **Redundant and wasteful** - two separate keepalive mechanisms
|
||||
- Application heartbeat fires 4-5 times per HTTP/2 PING
|
||||
|
||||
**RECOMMENDATION:**
|
||||
- Align HTTP/2 keepalive with application heartbeat (both 10s)
|
||||
- OR disable application heartbeat and rely on HTTP/2 PING
|
||||
- Current setup works but is inefficient
|
||||
|
||||
### 4. **Shardok-Specific Connection Issues**
|
||||
|
||||
**User reports:** "Issues only for one player who is not on the local network, and apparently only while Shardok game is going"
|
||||
|
||||
**Why Shardok is Different:**
|
||||
1. **Higher message frequency** - Tactical combat has more frequent action/reaction cycles than strategic Eagle gameplay
|
||||
2. **Larger message payloads** - Shardok sends full unit positions, stats, available actions per turn
|
||||
3. **Multiple simultaneous games** - Client tracks array of `ShardokViewStatuses[]`, each with own result count
|
||||
4. **Bidirectional traffic** - Player commands and AI responses flowing simultaneously
|
||||
|
||||
**Possible Root Causes:**
|
||||
|
||||
**A) Network Infrastructure NAT/Firewall Timeout**
|
||||
- Remote player likely behind home router/NAT
|
||||
- Many consumer routers have 60-120 second idle timeouts for UDP/TCP connections
|
||||
- During active Shardok gameplay, connection may appear "idle" at transport layer if:
|
||||
- Application data is buffered
|
||||
- HTTP/2 multiplexing makes connection look inactive
|
||||
- Router doesn't recognize gRPC traffic as "active"
|
||||
|
||||
**B) HTTP/2 Flow Control Window Exhaustion**
|
||||
- Shardok sends bursts of updates during AI turns
|
||||
- Client may not consume messages fast enough (UI thread blocking?)
|
||||
- HTTP/2 flow control window fills up
|
||||
- Server blocks sending, connection appears stalled
|
||||
- Eventually timeout or client-side buffer overflow
|
||||
|
||||
**C) nginx Buffer Issues**
|
||||
- nginx may buffer gRPC responses before sending to client
|
||||
- Default nginx buffers might be too small for Shardok update bursts
|
||||
- Missing `grpc_buffer_size` or `proxy_buffer_size` configuration
|
||||
- Buffering could add latency that appears as "connection issues"
|
||||
|
||||
**D) Remote Player Packet Loss/Latency**
|
||||
- High latency + packet loss = TCP retransmission delays
|
||||
- gRPC is sensitive to head-of-line blocking in HTTP/2
|
||||
- One lost packet can stall entire stream
|
||||
- Appears as timeout or "dropped connection"
|
||||
|
||||
**E) Client-Side Processing Bottleneck**
|
||||
- Shardok updates processed on Unity main thread
|
||||
- If UI freezes during update processing, background thread can't read from stream
|
||||
- Server-side buffers fill up
|
||||
- Connection appears stalled
|
||||
|
||||
### 5. **Delta Update State Synchronization Risk**
|
||||
|
||||
**For Shardok**, client must track multiple game states:
|
||||
```csharp
|
||||
ShardokViewStatus {
|
||||
string shardok_game_id
|
||||
int32 filtered_result_count // Last action index seen
|
||||
}
|
||||
```
|
||||
|
||||
**Problem:** If connection drops mid-update:
|
||||
1. Client may have partially processed updates
|
||||
2. `filtered_result_count` may not match server's expectation
|
||||
3. Reconnection sends stale count
|
||||
4. Server either:
|
||||
- Sends duplicate updates (client sees repeated actions)
|
||||
- Skips updates (client misses actions)
|
||||
|
||||
**No visible state recovery mechanism** in the code:
|
||||
- No "resync" request to get full game state
|
||||
- No checksum/hash to verify client and server are in sync
|
||||
- No explicit handling of "I missed updates" scenario
|
||||
|
||||
**Impact:** After connection drop during Shardok, game state may be inconsistent.
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Critical Security Issue
|
||||
|
||||
**🔴 PRIORITY 1: Fix Shardok Internal Interface Exposure**
|
||||
|
||||
Update nginx config `/Users/dancrosby/CodingProjects/github/eagle0/eagle0.net`:
|
||||
|
||||
```nginx
|
||||
location /net.eagle0.common.ShardokInternalInterface {
|
||||
# This endpoint should NOT be publicly accessible
|
||||
# Option 1: Remove this block entirely (recommended)
|
||||
# Option 2: Add authentication and timeouts like Eagle endpoint:
|
||||
auth_basic "User Area";
|
||||
auth_basic_user_file /etc/nginx/.htpasswd;
|
||||
grpc_pass grpc://192.168.123.100:40042;
|
||||
grpc_send_timeout 1200s;
|
||||
grpc_read_timeout 1200s;
|
||||
grpc_socket_keepalive on;
|
||||
}
|
||||
```
|
||||
|
||||
**Recommended:** Remove the location block entirely. Shardok should only be accessible to Eagle server on internal network.
|
||||
|
||||
### Immediate Diagnostic Actions
|
||||
|
||||
**1. Add Precise Connection Drop Logging**
|
||||
|
||||
Modify `PersistentClientConnection.cs` to log:
|
||||
- Exact timestamp of every connection attempt
|
||||
- Exact timestamp of every disconnection
|
||||
- StatusCode of every RpcException
|
||||
- Time delta between last successful message and disconnect
|
||||
- Whether disconnect happened during Shardok gameplay
|
||||
|
||||
**Example log format:**
|
||||
```
|
||||
[2024-11-30 03:27:31.026] DISCONNECT after 127.3s idle, StatusCode=Cancelled, InShardok=true
|
||||
```
|
||||
|
||||
This will answer:
|
||||
- Is it ACTUALLY ~2 minutes, or something else?
|
||||
- What is the exact error code?
|
||||
- Does it only happen during Shardok?
|
||||
|
||||
**2. Add Remote Player Network Diagnostics**
|
||||
|
||||
For the affected remote player, collect:
|
||||
- Ping time to eagle0.net (measure TCP latency, not ICMP)
|
||||
- Traceroute to eagle0.net (identify network hops)
|
||||
- MTU discovery (check for fragmentation issues)
|
||||
- Connection stability test (long-running TCP connection)
|
||||
|
||||
**Tools:**
|
||||
```bash
|
||||
# On remote player's machine
|
||||
mtr eagle0.net # Continuous ping + traceroute
|
||||
curl -w "@curl-format.txt" -o /dev/null -s https://eagle0.net/
|
||||
```
|
||||
|
||||
**3. Enable nginx Logging for gRPC Endpoint**
|
||||
|
||||
Add to nginx config:
|
||||
```nginx
|
||||
location /net.eagle0.eagle.api.Eagle {
|
||||
# ... existing config ...
|
||||
access_log /var/log/nginx/eagle-grpc-access.log;
|
||||
error_log /var/log/nginx/eagle-grpc-error.log debug;
|
||||
}
|
||||
```
|
||||
|
||||
Look for patterns in nginx logs when remote player disconnects.
|
||||
|
||||
### Targeted Fixes for Shardok Issues
|
||||
|
||||
**If Root Cause is NAT/Firewall Timeout:**
|
||||
|
||||
**Solution A: Reduce HTTP/2 KeepAlive to be more aggressive**
|
||||
|
||||
In `EagleConnection.cs`, change:
|
||||
```csharp
|
||||
private const double KeepAliveSeconds = 15.0; // Was 45.0
|
||||
```
|
||||
|
||||
Sending HTTP/2 PING every 15 seconds should keep NAT sessions alive.
|
||||
|
||||
**Solution B: Enable TCP keepalive at socket level**
|
||||
|
||||
Add to nginx config:
|
||||
```nginx
|
||||
location /net.eagle0.eagle.api.Eagle {
|
||||
# ... existing config ...
|
||||
proxy_socket_keepalive on; # Add this
|
||||
}
|
||||
```
|
||||
|
||||
**If Root Cause is HTTP/2 Flow Control:**
|
||||
|
||||
**Solution: Increase nginx HTTP/2 buffer sizes**
|
||||
|
||||
Add to nginx `http` block:
|
||||
```nginx
|
||||
http {
|
||||
http2_max_field_size 16k;
|
||||
http2_max_header_size 32k;
|
||||
client_body_buffer_size 128k;
|
||||
client_max_body_size 10m;
|
||||
}
|
||||
```
|
||||
|
||||
**If Root Cause is nginx Buffering:**
|
||||
|
||||
**Solution: Disable nginx buffering for gRPC**
|
||||
|
||||
Add to nginx gRPC location:
|
||||
```nginx
|
||||
location /net.eagle0.eagle.api.Eagle {
|
||||
# ... existing config ...
|
||||
grpc_buffering off; # Add this - disable response buffering
|
||||
}
|
||||
```
|
||||
|
||||
**If Root Cause is Client-Side Processing Delay:**
|
||||
|
||||
**Solution: Move Shardok update processing off Unity main thread**
|
||||
|
||||
Profile Unity to check if:
|
||||
- `HandleStreamingCall()` thread is blocked waiting to write to queue
|
||||
- Main thread is slow processing Shardok updates
|
||||
- GC pauses are causing delays
|
||||
|
||||
### General Connection Robustness Improvements
|
||||
|
||||
**1. Add State Resync Mechanism**
|
||||
|
||||
When reconnecting after drop during Shardok:
|
||||
- Send `StreamGameRequest` with `filtered_result_count = 0`
|
||||
- Server sends full current state instead of delta
|
||||
- Client discards partial state and resyncs
|
||||
|
||||
**2. Align Heartbeat Timings**
|
||||
|
||||
Current setup is inefficient with two layers:
|
||||
- HTTP/2 keepalive: 45s
|
||||
- Application heartbeat: 10s
|
||||
|
||||
**Option A: Single-layer approach**
|
||||
```csharp
|
||||
// In EagleConnection.cs
|
||||
private const double KeepAliveSeconds = 10.0; // Align with application heartbeat
|
||||
|
||||
// In PersistentClientConnection.cs
|
||||
// Remove application-level heartbeat entirely, rely on HTTP/2
|
||||
```
|
||||
|
||||
**Option B: Keep both but optimize**
|
||||
```csharp
|
||||
// In EagleConnection.cs
|
||||
private const double KeepAliveSeconds = 30.0; // Longer HTTP/2 interval
|
||||
|
||||
// In PersistentClientConnection.cs
|
||||
private const double HeartbeatTimerSeconds = 30.0; // Match HTTP/2
|
||||
```
|
||||
|
||||
**3. Implement Exponential Backoff for Reconnections**
|
||||
|
||||
Currently, client reconnects immediately on error. For transient network issues:
|
||||
```csharp
|
||||
private int _reconnectAttempts = 0;
|
||||
private const int MaxReconnectDelay = 30; // seconds
|
||||
|
||||
public async Task Connect() {
|
||||
if (_reconnectAttempts > 0) {
|
||||
int delay = Math.Min(Math.Pow(2, _reconnectAttempts), MaxReconnectDelay);
|
||||
await Task.Delay(delay * 1000);
|
||||
}
|
||||
// ... existing connect logic ...
|
||||
_reconnectAttempts++; // Increment on failure
|
||||
}
|
||||
|
||||
// Reset on successful connection:
|
||||
// _reconnectAttempts = 0;
|
||||
```
|
||||
|
||||
## Summary of Key Findings
|
||||
|
||||
### ✅ What's Working
|
||||
|
||||
1. **nginx Configuration:** Very generous 20-minute timeouts, properly configured for gRPC
|
||||
2. **Client Reconnection Logic:** Now fixed with recent PRs (#4596, #4597) to handle heartbeat timeout and race conditions
|
||||
3. **Server Configuration:** No forced timeouts that would cause premature disconnections
|
||||
4. **TLS/HTTPS:** Properly configured with Let's Encrypt, HTTP/2 enabled
|
||||
|
||||
### 🔴 Critical Issues Found
|
||||
|
||||
1. **Shardok Internal Interface Exposed Without Auth** (Security vulnerability)
|
||||
2. **Mystery "2-minute timeout"** - No 120-second timeout found in code
|
||||
3. **No state resync mechanism** after connection drops during Shardok
|
||||
4. **Inefficient dual-layer heartbeat** (HTTP/2 + application level)
|
||||
|
||||
### 🟡 Hypotheses for Remote Player Disconnects
|
||||
|
||||
**Most Likely:**
|
||||
1. **NAT/Firewall timeout** at remote player's router/ISP (60-120s idle timeout)
|
||||
2. **Network packet loss/latency** causing HTTP/2 head-of-line blocking
|
||||
3. **nginx buffering** adding latency during Shardok update bursts
|
||||
|
||||
**Less Likely:**
|
||||
4. HTTP/2 flow control window exhaustion
|
||||
5. Client-side processing bottleneck
|
||||
|
||||
**Needs Measurement:**
|
||||
- Exact time between connect and disconnect
|
||||
- Correlation with Shardok gameplay phases (player turn vs AI turn)
|
||||
- Remote player's network characteristics
|
||||
|
||||
## Next Steps (Prioritized)
|
||||
|
||||
1. **🔴 SECURITY:** Fix Shardok internal interface exposure in nginx config
|
||||
2. **📊 DIAGNOSTICS:** Add precise connection drop logging with timestamps
|
||||
3. **🔍 INVESTIGATE:** Get network diagnostics from affected remote player
|
||||
4. **🐛 FIX:** Implement most likely solution (reduce HTTP/2 keepalive to 15s)
|
||||
5. **🔄 IMPROVE:** Add state resync mechanism for Shardok
|
||||
6. **⚡ OPTIMIZE:** Align heartbeat timings to reduce redundancy
|
||||
|
||||
## Files Examined
|
||||
|
||||
- ✅ `src/main/protobuf/net/eagle0/eagle/api/eagle.proto` - gRPC service definition
|
||||
- ✅ `src/main/csharp/net/eagle0/clients/unity/eagle0/Assets/EagleConnection.cs` - Channel setup
|
||||
- ✅ `src/main/csharp/net/eagle0/clients/unity/eagle0/Assets/Eagle/PersistentClientConnection.cs` - Connection management
|
||||
- ✅ `src/main/scala/net/eagle0/eagle/service/EagleServiceImpl.scala` - Server-side handler
|
||||
- ✅ `src/main/scala/net/eagle0/eagle/Main.scala` - Server startup configuration
|
||||
- ✅ `/Users/dancrosby/CodingProjects/github/eagle0/eagle0.net` - nginx configuration
|
||||
|
||||
## Files Not Yet Examined
|
||||
|
||||
- YetAnotherHttpHandler source code (client HTTP/2 implementation)
|
||||
- Shardok ↔ Eagle internal gRPC interface implementation
|
||||
- GamesManager/CustomBattleManager Shardok integration
|
||||
- Client-side Shardok update processing (Unity main thread impact)
|
||||
- Server-side Shardok result caching/delta calculation
|
||||
|
||||
---
|
||||
|
||||
**Document Created:** 2024-11-30
|
||||
**Author:** Claude Code (automated audit)
|
||||
**Status:** Comprehensive initial audit complete - awaiting user feedback and diagnostic data
|
||||
**Critical Finding:** Shardok internal interface security vulnerability
|
||||
**Recommended First Action:** Add precise connection drop logging to measure actual timeout duration
|
||||
Reference in New Issue
Block a user