mirror of
https://github.com/nolen777/eagle0.git
synced 2026-07-28 20:55:44 +00:00
Replace Hetzner plan doc with latency hiding strategies (#5053)
The Hetzner on-demand compute implementation is complete: - ARM64 builds working in CI - Shardok running on Hetzner CAX41 in Helsinki - Production Eagle connected via TLS + token auth - IPv6/NAT64 networking configured Delete the completed planning doc and add a new doc covering future latency optimization strategies: 1. Client-side animation masking (recommended first step) 2. Split Shardok architecture (future if needed) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,216 @@
|
||||
# Shardok Latency Hiding Strategies
|
||||
|
||||
## Problem Statement
|
||||
|
||||
With Shardok running on Hetzner (Helsinki) and Eagle on DigitalOcean (US), the round-trip latency for human commands is ~200-400ms:
|
||||
|
||||
```
|
||||
Human posts command:
|
||||
Unity → Eagle (DO) → Shardok (Hetzner) → Eagle (DO) → Unity
|
||||
~10ms ~100ms ~100ms ~10ms
|
||||
Total: ~220ms round-trip
|
||||
```
|
||||
|
||||
This latency is acceptable for AI turns (users watch animations anyway), but creates noticeable lag when humans post commands.
|
||||
|
||||
---
|
||||
|
||||
## Strategy 1: Client-Side Animation Masking
|
||||
|
||||
### Concept
|
||||
|
||||
Start animations immediately when the user clicks, before server confirmation arrives. The animation duration masks the network latency.
|
||||
|
||||
### Implementation by Command Type
|
||||
|
||||
**Movement Commands** (deterministic):
|
||||
- Client knows the destination hex and movement path
|
||||
- Start movement animation immediately on click
|
||||
- Server confirms the move (should always match)
|
||||
- If server rejects (invalid state), snap unit back to origin
|
||||
|
||||
**Attack Commands** (RNG-dependent):
|
||||
- Show attack animation immediately (unit swings sword, fires arrow)
|
||||
- Wait for server to return dice roll result
|
||||
- Show damage numbers / hit effects when server responds
|
||||
- Animation typically takes 300-500ms, masking most of the latency
|
||||
|
||||
**End Turn**:
|
||||
- Latency not noticeable (user expects transition delay)
|
||||
|
||||
### Unity Implementation Sketch
|
||||
|
||||
```csharp
|
||||
// In CommandHandler.cs
|
||||
public void OnCommandSelected(Command command) {
|
||||
// Start animation immediately
|
||||
if (command.Type == CommandType.Move) {
|
||||
unitController.StartMoveAnimation(command.TargetHex);
|
||||
} else if (command.Type == CommandType.Attack) {
|
||||
unitController.StartAttackAnimation(command.TargetUnit);
|
||||
}
|
||||
|
||||
// Send to server in parallel
|
||||
connection.SendCommand(command, (response) => {
|
||||
if (response.Success) {
|
||||
// Animation continues, apply result
|
||||
ApplyCommandResult(response);
|
||||
} else {
|
||||
// Rollback animation
|
||||
unitController.CancelAnimation();
|
||||
ShowError(response.ErrorMessage);
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Pros
|
||||
- Simple implementation
|
||||
- No server-side changes
|
||||
- Works with existing architecture
|
||||
|
||||
### Cons
|
||||
- Doesn't eliminate latency for attacks with RNG (must wait for dice roll)
|
||||
- Rollback needed if server rejects command (rare but possible)
|
||||
|
||||
### Estimated Improvement
|
||||
- Movement: ~200ms latency hidden (feels instant)
|
||||
- Attacks: ~100-200ms hidden by animation, ~100ms visible wait for dice result
|
||||
|
||||
---
|
||||
|
||||
## Strategy 2: Split Shardok Architecture
|
||||
|
||||
### Concept
|
||||
|
||||
Run two Shardok instances:
|
||||
- **Shardok-Primary (DigitalOcean)**: Handles command processing, source of truth
|
||||
- **Shardok-AI (Hetzner)**: AI computation only
|
||||
|
||||
Human commands go to the nearby Primary for low latency. AI computation uses the powerful Hetzner instance.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Human commands (low latency ~20ms)
|
||||
Unity ←→ Eagle ←→ Shardok-Primary (DigitalOcean)
|
||||
↓ state sync (when AI turn starts)
|
||||
Shardok-AI (Hetzner)
|
||||
↑ AI command response
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
**Human Turn:**
|
||||
1. Human posts command → Eagle → Shardok-Primary (DO)
|
||||
2. Primary processes command immediately (~10ms local)
|
||||
3. Primary streams result to client via Eagle (~10ms)
|
||||
4. **Total latency: ~20ms** (vs ~220ms current)
|
||||
|
||||
**AI Turn:**
|
||||
1. When AI's turn starts, Primary sends game state snapshot to Hetzner
|
||||
2. Shardok-AI computes best command using full CPU power
|
||||
3. Shardok-AI returns command index to Primary
|
||||
4. Primary executes command locally and streams to client
|
||||
5. Repeat until AI turn ends
|
||||
|
||||
### Protocol Changes
|
||||
|
||||
```protobuf
|
||||
// New service for AI-only computation
|
||||
service ShardokAIService {
|
||||
// Send game state, receive AI's chosen command
|
||||
rpc GetAICommand(AICommandRequest) returns (AICommandResponse);
|
||||
}
|
||||
|
||||
message AICommandRequest {
|
||||
bytes game_state = 1; // Serialized game state
|
||||
int32 player_id = 2; // Which AI player
|
||||
repeated bytes available_commands = 3; // Available command descriptors
|
||||
}
|
||||
|
||||
message AICommandResponse {
|
||||
int32 command_index = 1; // Index into available_commands
|
||||
int32 search_depth = 2; // For debugging
|
||||
double best_score = 3; // For debugging
|
||||
}
|
||||
```
|
||||
|
||||
### Shardok-Primary Requirements
|
||||
|
||||
Shardok-Primary on DigitalOcean needs to:
|
||||
- Process all commands (human and AI)
|
||||
- Maintain authoritative game state
|
||||
- Serialize/deserialize game state for AI requests
|
||||
- Run on minimal CPU (command processing is fast)
|
||||
|
||||
This is essentially the current Shardok, but without running the AI search.
|
||||
|
||||
### Shardok-AI Requirements
|
||||
|
||||
Shardok-AI on Hetzner needs to:
|
||||
- Receive game state snapshots
|
||||
- Run AI evaluation (IterativeDeepeningAI or MCTS)
|
||||
- Return best command index
|
||||
- No persistent state (stateless worker)
|
||||
|
||||
### AI Turn Latency
|
||||
|
||||
Each AI command has ~200ms network latency. This is acceptable because:
|
||||
1. User is watching animations anyway
|
||||
2. Natural pacing lets user observe AI decisions
|
||||
3. AI computation is fast on Hetzner's 16 cores
|
||||
|
||||
For a typical AI turn with 5 commands: 5 × 200ms = 1 second network overhead, plus AI thinking time. With animations, this feels natural.
|
||||
|
||||
### Implementation Phases
|
||||
|
||||
**Phase 1: Add Shardok-Primary (minimal)**
|
||||
- Deploy existing Shardok container to DigitalOcean
|
||||
- Configure Eagle to use local Shardok for all commands
|
||||
- Human latency immediately improves
|
||||
|
||||
**Phase 2: Add AI offload**
|
||||
- Implement `ShardokAIService` RPC
|
||||
- Shardok-Primary calls Hetzner for AI commands
|
||||
- Shardok-AI processes requests statelessly
|
||||
|
||||
**Phase 3: Optimize**
|
||||
- Batch multiple AI actions if possible
|
||||
- Pre-warm Shardok-AI connection
|
||||
- Add fallback if Hetzner unavailable
|
||||
|
||||
### Pros
|
||||
- Human command latency drops from ~220ms to ~20ms
|
||||
- AI still gets Hetzner's CPU power
|
||||
- Clear separation of concerns
|
||||
- Shardok-Primary can fall back to local AI if Hetzner unavailable
|
||||
|
||||
### Cons
|
||||
- Two Shardok instances to maintain
|
||||
- State serialization overhead for AI requests
|
||||
- Each AI action has network round-trip (acceptable with animations)
|
||||
|
||||
---
|
||||
|
||||
## Comparison
|
||||
|
||||
| Approach | Human Latency | AI Throughput | Complexity | Changes Required |
|
||||
|----------|---------------|---------------|------------|------------------|
|
||||
| Current | ~220ms | High | - | - |
|
||||
| Animation masking | ~220ms (perceived ~50ms) | High | Low | Unity only |
|
||||
| Split architecture | ~20ms | High | Medium | New RPC, two deployments |
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Phase 1 (now)**: Implement animation masking in Unity client
|
||||
- Quick win, no server changes
|
||||
- Improves perceived latency significantly for movement
|
||||
- Attacks still show dice animation while waiting
|
||||
|
||||
**Phase 2 (future)**: Split architecture if animation masking insufficient
|
||||
- Only needed if users complain about attack latency
|
||||
- More complex but provides true low latency
|
||||
- Natural evolution of current architecture
|
||||
@@ -1,769 +0,0 @@
|
||||
# Shardok On-Demand Compute Infrastructure Analysis
|
||||
|
||||
## Overview
|
||||
|
||||
This document analyzes options for running Shardok (the tactical combat AI server) on powerful on-demand compute, separate from the low-power droplet that hosts Eagle.
|
||||
|
||||
**Current State**: Shardok runs on the same DigitalOcean droplet as Eagle, limiting AI performance.
|
||||
|
||||
**Goal**: Spin up powerful compute only when battles occur, keeping costs low while improving AI quality.
|
||||
|
||||
---
|
||||
|
||||
## Cloud Provider Pricing Comparison
|
||||
|
||||
### High-Performance CPU Options (8-16 cores)
|
||||
|
||||
| Provider | Instance Type | vCPUs | RAM | Hourly Cost | Monthly Est.* |
|
||||
|----------|--------------|-------|-----|-------------|---------------|
|
||||
| **Hetzner** | CCX23 | 8 | 32GB | €0.078 (~$0.085) | ~$6-8 |
|
||||
| **Hetzner** | CCX33 | 16 | 64GB | €0.155 (~$0.17) | ~$12-16 |
|
||||
| **Vultr** | High Frequency 8-core | 8 | 32GB | $0.238 | ~$20-25 |
|
||||
| **Vultr** | High Frequency 16-core | 16 | 64GB | $0.476 | ~$40-50 |
|
||||
| **DigitalOcean** | Premium CPU 8-core | 8 | 32GB | $0.250 | ~$22-28 |
|
||||
| **DigitalOcean** | Premium CPU 16-core | 16 | 64GB | $0.500 | ~$44-55 |
|
||||
|
||||
*Monthly estimates assume ~4 hours/week of battle usage (~17 hours/month)
|
||||
|
||||
### Key Finding: Hetzner is 3-5x Cheaper
|
||||
|
||||
Hetzner's dedicated vCPU instances offer the best price-performance ratio by a significant margin. For equivalent 16-core compute:
|
||||
- Hetzner: ~$0.17/hr
|
||||
- DigitalOcean: ~$0.50/hr (2.9x more expensive)
|
||||
- Vultr: ~$0.48/hr (2.8x more expensive)
|
||||
|
||||
---
|
||||
|
||||
## Instance Startup Time Analysis
|
||||
|
||||
### Current Startup Times by Provider
|
||||
|
||||
| Provider | Cold Start | Warm Image | Container Registry |
|
||||
|----------|-----------|------------|-------------------|
|
||||
| Hetzner | 30-60s | 20-40s | Native support |
|
||||
| Vultr | 30-45s | 20-30s | Native support |
|
||||
| DigitalOcean | 30-60s | 25-45s | Already in use |
|
||||
|
||||
### 10-Second Max Wait Time Constraint
|
||||
|
||||
A 10-second startup constraint eliminates traditional VM spin-up as an option. Alternatives:
|
||||
|
||||
#### Option 1: Pre-warmed Instance Pool (Recommended)
|
||||
- Keep one small instance running 24/7 as a "hot standby"
|
||||
- Cost: ~$5-10/month for a minimal Hetzner instance
|
||||
- Startup time: **< 1 second** (already running)
|
||||
- Upgrade on demand when battle starts (resize takes 2-3 min, but battle can start immediately on small instance)
|
||||
|
||||
#### Option 2: Predictive Spin-up
|
||||
- Start spinning up when user enters battle setup
|
||||
- Battle setup typically takes 30-60 seconds (map selection, army placement)
|
||||
- Could achieve "instant" availability if spin-up completes during setup
|
||||
- Risk: User might cancel, wasting spin-up cost (minimal at ~$0.01-0.02)
|
||||
|
||||
#### Option 3: Container-Based Quick Start
|
||||
- Use pre-pulled Docker images on a warm container host
|
||||
- Providers like Fly.io or Railway can start containers in 1-5 seconds
|
||||
- Trade-off: Container platforms typically cost more than raw VMs
|
||||
|
||||
#### Option 4: Serverless/Function Compute
|
||||
- Not viable for Shardok due to:
|
||||
- Long-running connections (battles last minutes)
|
||||
- High memory requirements
|
||||
- Stateful game state
|
||||
|
||||
### Recommendation for 10s Constraint
|
||||
|
||||
**Hybrid approach**: Keep a minimal "always-on" instance ($5-10/month) that can handle battles at reduced AI quality. When a battle is anticipated (entering battle setup), start spinning up a powerful instance. If the powerful instance is ready when the battle starts, use it; otherwise, fall back to the always-on instance.
|
||||
|
||||
---
|
||||
|
||||
## ARM Options Outside AWS
|
||||
|
||||
### Available ARM Compute
|
||||
|
||||
| Provider | ARM Offering | vCPUs | RAM | Hourly Cost | Notes |
|
||||
|----------|-------------|-------|-----|-------------|-------|
|
||||
| **Hetzner** | CAX21 | 4 | 8GB | €0.009 | Ampere Altra |
|
||||
| **Hetzner** | CAX31 | 8 | 16GB | €0.018 | Ampere Altra |
|
||||
| **Hetzner** | CAX41 | 16 | 32GB | €0.035 | Ampere Altra |
|
||||
| **Oracle Cloud** | A1 Flex | 4 | 24GB | Free tier | Always Free eligible |
|
||||
| **Oracle Cloud** | A1 Flex | 24 | 144GB | ~$0.10 | Ampere Altra |
|
||||
| **Scaleway** | AMP2-C8 | 8 | 16GB | €0.104 | Ampere Altra |
|
||||
|
||||
### Key Finding: Hetzner ARM is Extremely Cheap
|
||||
|
||||
Hetzner's ARM instances are 4-5x cheaper than their x86 equivalents:
|
||||
- CAX41 (16 ARM cores): €0.035/hr (~$0.04)
|
||||
- CCX33 (16 x86 cores): €0.155/hr (~$0.17)
|
||||
|
||||
### ARM Considerations for Shardok
|
||||
|
||||
**Pros:**
|
||||
- Significant cost savings (4-5x)
|
||||
- Good single-threaded performance on Ampere Altra
|
||||
- Native ARM builds already supported (macOS development)
|
||||
|
||||
**Cons:**
|
||||
- Requires ARM64 Linux build (currently only building x86_64 Linux)
|
||||
- Some performance tuning may be needed
|
||||
- Less mature ecosystem than x86
|
||||
|
||||
**Build Effort**: Medium. Bazel supports cross-compilation, and the codebase is portable C++. Main work is adding ARM64 Linux to the build matrix.
|
||||
|
||||
---
|
||||
|
||||
## Hetzner Geographic Presence
|
||||
|
||||
### Current Locations
|
||||
|
||||
Hetzner operates datacenters in:
|
||||
- **Germany**: Nuremberg (NBG1), Falkenstein (FSN1)
|
||||
- **Finland**: Helsinki (HEL1)
|
||||
- **USA**: Ashburn, Virginia (ASH) - **opened 2023**
|
||||
- **Singapore**: (coming 2024/2025)
|
||||
|
||||
### Latency Implications
|
||||
|
||||
For users primarily in North America:
|
||||
- **Hetzner US (Ashburn)**: 20-40ms to East Coast, 60-80ms to West Coast
|
||||
- **Hetzner EU**: 80-120ms from US East Coast, 140-180ms from US West Coast
|
||||
|
||||
Since Shardok handles AI computation (not real-time player input), latency is less critical. The Eagle-Shardok communication adds some overhead, but AI "thinking time" dominates.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Use **Hetzner US (Ashburn)** for North American users. The US datacenter has the same pricing as EU datacenters and provides better latency.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Options Summary
|
||||
|
||||
### Option A: Simple On-Demand (30-60s startup)
|
||||
- Spin up Hetzner CCX33 when battle starts
|
||||
- Cost: ~$0.17/hr × 4 hrs/week = **~$3/month**
|
||||
- Latency: 30-60s wait at battle start
|
||||
- Complexity: Low
|
||||
|
||||
### Option B: Predictive Spin-up (0-30s startup)
|
||||
- Start spinning up during battle setup phase
|
||||
- Cost: Same as Option A, plus occasional wasted spin-ups (~$1/month)
|
||||
- Latency: 0-30s depending on setup duration
|
||||
- Complexity: Medium
|
||||
|
||||
### Option C: Hybrid Always-On + On-Demand (< 1s startup)
|
||||
- Keep minimal instance always running ($5-10/month)
|
||||
- Spin up powerful instance during battle setup
|
||||
- Use powerful if ready, fall back to minimal
|
||||
- Cost: **~$8-15/month**
|
||||
- Latency: < 1s (always-on fallback)
|
||||
- Complexity: Medium-High
|
||||
|
||||
### Option D: ARM Compute (Cheapest)
|
||||
- Build ARM64 Linux target
|
||||
- Use Hetzner CAX41 (16 ARM cores) at €0.035/hr
|
||||
- Cost: **~$0.60/month** for compute (+ always-on if needed)
|
||||
- Latency: Same as x86 options
|
||||
- Complexity: Medium (build system changes)
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Phase 1 (Quick Win)**: Implement Option B (Predictive Spin-up) with Hetzner CCX33 in Ashburn.
|
||||
- Minimal code changes (add spin-up trigger in Eagle when battle setup begins)
|
||||
- Good user experience (usually ready by battle start)
|
||||
- Low cost (~$3-5/month)
|
||||
|
||||
**Phase 2 (Future)**: Add ARM64 Linux build and switch to ARM instances.
|
||||
- 4-5x cost reduction
|
||||
- Same or better performance
|
||||
- Requires build system work
|
||||
|
||||
**Phase 3 (If Needed)**: Add always-on fallback for guaranteed instant start.
|
||||
- Only if Phase 1 startup times prove problematic
|
||||
- Adds ~$5-10/month fixed cost
|
||||
|
||||
---
|
||||
|
||||
# Implementation Plan
|
||||
|
||||
## Chosen Architecture
|
||||
|
||||
- **Provider**: Hetzner Cloud
|
||||
- **Instance**: CAX41 (16 ARM cores, 32GB RAM)
|
||||
- **Location**: Ashburn, Virginia (ash)
|
||||
- **Cost**: €0.035/hr (~$0.04/hr)
|
||||
- **Strategy**: On-demand with predictive spin-up and idle keep-alive
|
||||
|
||||
---
|
||||
|
||||
## System Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ DigitalOcean │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Eagle Server │ │
|
||||
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
|
||||
│ │ │ ShardokManager │ │ ActivityTracker │ │ │
|
||||
│ │ │ - spin up/down │ │ - player active │ │ │
|
||||
│ │ │ - health check │ │ - trigger start │ │ │
|
||||
│ │ └────────┬────────┘ └────────┬────────┘ │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ └────────┬───────────┘ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ ┌────────────────┐ │ │
|
||||
│ │ │ ShardokClient │◄─── TLS + Token Auth │ │
|
||||
│ │ └────────┬───────┘ │ │
|
||||
│ └────────────────────┼─────────────────────────────────────────┘ │
|
||||
└───────────────────────┼──────────────────────────────────────────────┘
|
||||
│ gRPC over TLS (Let's Encrypt)
|
||||
▼
|
||||
┌───────────────────────────────────────────────────────────────────────┐
|
||||
│ Hetzner Cloud (Ashburn) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ CAX41 (On-Demand) │ │
|
||||
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Shardok Server │ │ │
|
||||
│ │ │ - 16 ARM cores for AI computation │ │ │
|
||||
│ │ │ - TLS cert from Let's Encrypt (auto-renewed) │ │ │
|
||||
│ │ │ - Validates auth token on each request │ │ │
|
||||
│ │ └─────────────────────────────────────────────────────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Instance Lifecycle Management
|
||||
|
||||
### State Machine
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ STOPPED │◄────────────────────┐
|
||||
└──────┬───────┘ │
|
||||
│ human takes turn │ idle timeout
|
||||
▼ │ (10 minutes)
|
||||
┌──────────────┐ │
|
||||
│ STARTING │ │
|
||||
└──────┬───────┘ │
|
||||
│ health check passes │
|
||||
▼ │
|
||||
┌──────────────┐ │
|
||||
┌─────────►│ READY │─────────────────────┤
|
||||
│ └──────┬───────┘ │
|
||||
│ │ battle starts │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ IN_USE │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ battle ends │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
Note: The IDLE state was removed - we now simply track time since last human turn and shut down after 10 minutes of inactivity.
|
||||
|
||||
### Lifecycle Rules
|
||||
|
||||
1. **Spin-up Trigger**: Human player takes a turn
|
||||
- Any command submitted by a human player to Eagle
|
||||
- Simple and reliable - no prediction needed
|
||||
|
||||
2. **Keep-Alive**: Reset timer on each human turn
|
||||
- Covers thinking time between turns
|
||||
- Natural pauses (snacks, calls) under 10 min don't cause shutdown
|
||||
|
||||
3. **Shutdown**: 10 minutes since last human turn
|
||||
- Hetzner API: `DELETE /servers/{id}` to stop billing
|
||||
- Short timeout keeps costs low during idle periods
|
||||
|
||||
4. **Health Checks**: Every 10 seconds while STARTING
|
||||
- gRPC connectivity check to Shardok
|
||||
- Mark READY only when Shardok responds successfully
|
||||
|
||||
---
|
||||
|
||||
## Spin-up Strategy
|
||||
|
||||
### Simple Approach: Human Turn-Based
|
||||
|
||||
Spin up Shardok whenever a human player takes a turn. Shut down after 10 minutes of inactivity.
|
||||
|
||||
This is simpler than trying to predict battles. Battles happen regularly during active play, so keeping Shardok running while humans are playing ensures it's ready when needed.
|
||||
|
||||
**Spin-up trigger:**
|
||||
- Any human player takes a turn (submits commands to Eagle)
|
||||
|
||||
**Shutdown trigger:**
|
||||
- No human player has taken a turn in the last 10 minutes
|
||||
|
||||
**Why this works:**
|
||||
- Battles are common during active play sessions
|
||||
- 10-minute idle timeout is long enough to cover thinking time between turns
|
||||
- Cost is so low (~$0.04/hr) that running during entire play sessions is fine
|
||||
- No complex prediction logic needed
|
||||
|
||||
### Cost Impact
|
||||
|
||||
| Scenario | Hours/Week | Cost/Month |
|
||||
|----------|------------|------------|
|
||||
| Human turn-based (10 min idle) | ~15-20 hrs | ~$2.50-3.50 |
|
||||
| Always-on 24/7 | 168 hrs | ~$29 |
|
||||
|
||||
Human turn-based is simple and cheap. The 10-minute timeout catches natural breaks (getting a snack, answering a call) while shutting down during longer idle periods.
|
||||
|
||||
### Implementation in Eagle
|
||||
|
||||
```scala
|
||||
class ShardokInstanceManager {
|
||||
private var instanceState: InstanceState = STOPPED
|
||||
private var lastHumanTurnTime: Instant = Instant.MIN
|
||||
private val idleTimeoutMinutes = 10
|
||||
|
||||
// Called when any human player submits a turn
|
||||
def onHumanTurn(): Unit = {
|
||||
lastHumanTurnTime = Instant.now()
|
||||
if (instanceState == STOPPED) {
|
||||
startInstance()
|
||||
}
|
||||
}
|
||||
|
||||
// Run every minute
|
||||
def periodicShutdownCheck(): Unit = {
|
||||
val idleTime = Duration.between(lastHumanTurnTime, Instant.now())
|
||||
if (instanceState != STOPPED && idleTime > Duration.ofMinutes(idleTimeoutMinutes)) {
|
||||
stopInstance()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration Point
|
||||
|
||||
Wire `onHumanTurn()` into Eagle's command processing. The cleanest place is where Eagle receives player commands from the Unity client - likely in `EagleServiceImpl` when handling the gRPC streaming commands.
|
||||
|
||||
---
|
||||
|
||||
## Security: Eagle ↔ Shardok Authentication
|
||||
|
||||
Since Eagle (DigitalOcean) and Shardok (Hetzner) are on different cloud providers, they cannot share a private network. We need secure authentication over the public internet.
|
||||
|
||||
### Approach: TLS + Token Authentication
|
||||
|
||||
This is the same pattern used by Stripe, GitHub, and most cloud APIs.
|
||||
|
||||
**Layer 1: TLS Encryption (Let's Encrypt)**
|
||||
- Server certificate from Let's Encrypt, auto-renewed via certbot
|
||||
- All gRPC traffic encrypted with TLS 1.3
|
||||
- Prevents eavesdropping and MITM attacks
|
||||
|
||||
**Layer 2: Token Authentication**
|
||||
- 256-bit random shared secret in gRPC metadata
|
||||
- Only requests with valid token are processed
|
||||
- Easy to rotate without any certificate changes
|
||||
|
||||
### How Token Auth Works
|
||||
|
||||
**Setup (once):**
|
||||
```bash
|
||||
# Generate a 256-bit random token
|
||||
openssl rand -hex 32
|
||||
# Output: a3f8b2c9d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1
|
||||
|
||||
# Store in GitHub Secrets as SHARDOK_AUTH_TOKEN
|
||||
# Eagle reads from environment, Shardok receives via cloud-init
|
||||
```
|
||||
|
||||
**Every Request:**
|
||||
```
|
||||
Eagle Shardok
|
||||
────── ───────
|
||||
gRPC call with metadata: → Validate token:
|
||||
"authorization: Bearer a3f8b2c9..." if token != expected:
|
||||
return UNAUTHENTICATED
|
||||
else:
|
||||
process request
|
||||
```
|
||||
|
||||
### Server Certificate (Let's Encrypt)
|
||||
|
||||
Shardok needs a domain name for Let's Encrypt. Options:
|
||||
|
||||
1. **Dynamic DNS**: Assign `shardok.prod.eagle0.net` to instance IP on spin-up
|
||||
2. **Elastic IP**: Reserve a Hetzner floating IP, always points to active instance
|
||||
|
||||
Cloud-init runs certbot on first boot:
|
||||
|
||||
```yaml
|
||||
runcmd:
|
||||
# Install certbot
|
||||
- apt-get install -y certbot
|
||||
|
||||
# Get certificate (DNS must already point to this IP)
|
||||
- certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net
|
||||
|
||||
# Certificate auto-renews via systemd timer (certbot installs this)
|
||||
# Certs at: /etc/letsencrypt/live/shardok.prod.eagle0.net/
|
||||
```
|
||||
|
||||
### Implementation
|
||||
|
||||
**Shardok gRPC Server (C++):**
|
||||
```cpp
|
||||
// Load Let's Encrypt certificates
|
||||
std::string cert = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem");
|
||||
std::string key = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem");
|
||||
|
||||
grpc::SslServerCredentialsOptions ssl_opts;
|
||||
ssl_opts.pem_key_cert_pairs.push_back({key, cert});
|
||||
auto creds = grpc::SslServerCredentials(ssl_opts);
|
||||
builder.AddListeningPort("0.0.0.0:50051", creds);
|
||||
|
||||
// Token validation - check on every RPC
|
||||
Status PostCommand(ServerContext* context, const Request* req, Response* resp) {
|
||||
auto auth = context->client_metadata().find("authorization");
|
||||
std::string expected = "Bearer " + auth_token_;
|
||||
if (auth == context->client_metadata().end() ||
|
||||
auth->second != expected) {
|
||||
return Status(grpc::UNAUTHENTICATED, "Invalid token");
|
||||
}
|
||||
// ... handle request
|
||||
}
|
||||
```
|
||||
|
||||
**Eagle gRPC Client (Scala):**
|
||||
```scala
|
||||
// Standard TLS (trusts Let's Encrypt via system CA store)
|
||||
val channel = NettyChannelBuilder
|
||||
.forAddress("shardok.prod.eagle0.net", 50051)
|
||||
.useTransportSecurity() // TLS with system trust store
|
||||
.intercept(new ClientInterceptor {
|
||||
override def interceptCall[Req, Resp](
|
||||
method: MethodDescriptor[Req, Resp],
|
||||
callOptions: CallOptions,
|
||||
next: Channel): ClientCall[Req, Resp] = {
|
||||
new ForwardingClientCall.SimpleForwardingClientCall(
|
||||
next.newCall(method, callOptions)) {
|
||||
override def start(listener: Listener[Resp], headers: Metadata): Unit = {
|
||||
headers.put(
|
||||
Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER),
|
||||
s"Bearer $authToken"
|
||||
)
|
||||
super.start(listener, headers)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
.build()
|
||||
```
|
||||
|
||||
### Token Rotation
|
||||
|
||||
When rotating tokens (e.g., if compromised or as routine hygiene):
|
||||
|
||||
1. Generate new token
|
||||
2. Update Shardok to accept **both** old and new tokens temporarily
|
||||
3. Update Eagle to use new token
|
||||
4. Remove old token from Shardok
|
||||
|
||||
No downtime, no certificate regeneration.
|
||||
|
||||
### Firewall Rules
|
||||
|
||||
```bash
|
||||
# Hetzner firewall via cloud-init
|
||||
ufw default deny incoming
|
||||
ufw allow 80/tcp # Let's Encrypt HTTP-01 challenge (certbot)
|
||||
ufw allow 50051/tcp # gRPC (protected by TLS + token)
|
||||
ufw allow from <admin-ip>/32 to any port 22 # SSH for debugging
|
||||
ufw enable
|
||||
```
|
||||
|
||||
### Security Summary
|
||||
|
||||
| Threat | Mitigation |
|
||||
|--------|------------|
|
||||
| Eavesdropping | TLS 1.3 encryption |
|
||||
| MITM attack | Let's Encrypt certificate validates server identity |
|
||||
| Unauthorized access | 256-bit token required on every request |
|
||||
| Token theft | TLS prevents sniffing; rotate if compromised |
|
||||
|
||||
This is the industry-standard approach for API security.
|
||||
|
||||
---
|
||||
|
||||
## Hetzner API Integration
|
||||
|
||||
### API Authentication
|
||||
|
||||
```bash
|
||||
# Store in secrets manager
|
||||
HETZNER_API_TOKEN=<token from Hetzner Cloud Console>
|
||||
```
|
||||
|
||||
### Key API Operations
|
||||
|
||||
**Create Instance:**
|
||||
```bash
|
||||
curl -X POST "https://api.hetzner.cloud/v1/servers" \
|
||||
-H "Authorization: Bearer $HETZNER_API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "shardok-prod",
|
||||
"server_type": "cax41",
|
||||
"location": "ash",
|
||||
"image": "docker-ce",
|
||||
"ssh_keys": ["eagle-deploy"],
|
||||
"user_data": "<cloud-init script>",
|
||||
"labels": {"service": "shardok", "env": "prod"}
|
||||
}'
|
||||
```
|
||||
|
||||
**Check Status:**
|
||||
```bash
|
||||
curl "https://api.hetzner.cloud/v1/servers/{id}" \
|
||||
-H "Authorization: Bearer $HETZNER_API_TOKEN"
|
||||
# Response includes: status (running/starting/off), public IP
|
||||
```
|
||||
|
||||
**Delete Instance (Stop Billing):**
|
||||
```bash
|
||||
curl -X DELETE "https://api.hetzner.cloud/v1/servers/{id}" \
|
||||
-H "Authorization: Bearer $HETZNER_API_TOKEN"
|
||||
```
|
||||
|
||||
### Cloud-Init Script
|
||||
|
||||
```yaml
|
||||
#cloud-config
|
||||
package_update: true
|
||||
packages:
|
||||
- docker.io
|
||||
- certbot
|
||||
|
||||
write_files:
|
||||
- path: /etc/shardok/auth_token
|
||||
permissions: '0600'
|
||||
content: |
|
||||
${AUTH_TOKEN}
|
||||
|
||||
runcmd:
|
||||
# Get Let's Encrypt certificate (DNS must point to this IP first)
|
||||
- certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net
|
||||
|
||||
# Pull and run Shardok container
|
||||
- echo ${DO_REGISTRY_TOKEN} | docker login registry.digitalocean.com -u ${DO_REGISTRY_TOKEN} --password-stdin
|
||||
- docker pull registry.digitalocean.com/eagle0/shardok-server:arm64-latest
|
||||
- docker run -d --name shardok \
|
||||
-p 50051:50051 \
|
||||
-v /etc/letsencrypt:/etc/letsencrypt:ro \
|
||||
-v /etc/shardok:/etc/shardok:ro \
|
||||
registry.digitalocean.com/eagle0/shardok-server:arm64-latest \
|
||||
--cert=/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem \
|
||||
--key=/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem \
|
||||
--token-file=/etc/shardok/auth_token
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Container Registry
|
||||
|
||||
Using DigitalOcean Container Registry for all images (consistent with Eagle and other services).
|
||||
|
||||
Both architectures use the same repository with different tag prefixes:
|
||||
- ARM64 image: `registry.digitalocean.com/eagle0/shardok-server:arm64-latest`
|
||||
- x86 image: `registry.digitalocean.com/eagle0/shardok-server:latest`
|
||||
|
||||
CI automatically builds and pushes ARM64 images when C++ code changes.
|
||||
|
||||
---
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
### If Shardok Instance Not Ready
|
||||
|
||||
When a battle starts but the Hetzner instance isn't ready:
|
||||
|
||||
1. **Option A: Wait with Progress Indicator**
|
||||
- Show "Preparing battle arena..." with spinner
|
||||
- Most users won't notice 20-30s wait if UI is engaging
|
||||
- Simplest implementation
|
||||
|
||||
2. **Option B: Local Fallback** (Current behavior)
|
||||
- Run Shardok on Eagle's droplet temporarily
|
||||
- Lower AI quality but instant start
|
||||
- Migrate battle to Hetzner when ready (complex)
|
||||
|
||||
3. **Option C: Reduced AI Settings**
|
||||
- Start battle immediately with very short time budgets
|
||||
- Improve AI quality when Hetzner ready
|
||||
- Seamless to user but requires AI hot-swap
|
||||
|
||||
**Recommendation**: Start with Option A. If user feedback indicates wait time is problematic, implement Option B.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### Metrics to Track
|
||||
|
||||
| Metric | Purpose |
|
||||
|--------|---------|
|
||||
| `shardok_instance_state` | Current lifecycle state |
|
||||
| `shardok_startup_duration_seconds` | Time from spin-up to ready |
|
||||
| `shardok_battles_per_instance` | Efficiency of keep-alive |
|
||||
| `shardok_wasted_spinups` | Spin-ups without battles |
|
||||
| `shardok_hourly_cost` | Running cost tracking |
|
||||
|
||||
### Alerting
|
||||
|
||||
- Alert if startup takes > 90 seconds
|
||||
- Alert if instance stuck in STARTING for > 3 minutes
|
||||
- Alert if Hetzner API errors persist
|
||||
|
||||
---
|
||||
|
||||
## Cost Estimate (Activity-Based, ARM)
|
||||
|
||||
With activity-based spin-up, Shardok runs during active play sessions:
|
||||
|
||||
| Play Pattern | Hours/Week | Compute/Month |
|
||||
|--------------|------------|---------------|
|
||||
| Casual (few sessions) | ~10 hrs | **~$1.75** |
|
||||
| Regular (daily play) | ~20 hrs | **~$3.50** |
|
||||
| Heavy (multiple daily) | ~40 hrs | **~$7.00** |
|
||||
|
||||
Plus fixed costs:
|
||||
- Container registry: $0 (GitHub Container Registry)
|
||||
- Secrets management: $0 (GitHub Secrets)
|
||||
|
||||
**Total estimated cost: $2-7/month** for typical usage.
|
||||
|
||||
Compare to current DigitalOcean droplet upgrade (~$20-40/month for equivalent CPU power).
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Basic On-Demand (1-2 weeks)
|
||||
|
||||
1. ✅ Add ARM64 Linux to CI build matrix (PR #4990, merged 2026-01-02)
|
||||
- Added LLVM toolchain for ARM64 cross-compilation
|
||||
- Created ARM64 sysroot build workflow
|
||||
- Added Ubuntu 24.04 ARM64 base image for containers
|
||||
2. ✅ Push ARM64 container to DigitalOcean Container Registry (PR #4990)
|
||||
- Created shardok_arm64_build.yml workflow
|
||||
- Pushes to registry.digitalocean.com/eagle0/shardok-server:arm64-latest
|
||||
3. ✅ Implement Hetzner API client in Eagle (Scala) (PR #4996, merged 2026-01-02)
|
||||
- HetznerApiClient with create/delete/list/power operations
|
||||
- Async HTTP via OkHttp with Future-based API
|
||||
4. ✅ Add ShardokInstanceManager with spin-up/shutdown logic (PR #4998, merged 2026-01-02)
|
||||
- Activity-based spin-up when players connect
|
||||
- Idle timeout shutdown (default 60 minutes)
|
||||
- State machine: Stopped → Starting → Ready → InUse → Stopping
|
||||
5. ✅ Add TLS + token authentication to Shardok server (PR #5001, merged 2026-01-02)
|
||||
- C++ server: TLS via Let's Encrypt, Bearer token validation
|
||||
- Scala client: TLS and auth token support in ServerSetupHelpers
|
||||
6. 🔄 Test end-to-end with manual triggers ← **CURRENT**
|
||||
- ✅ Hetzner server created (CAX41 ARM64 in Helsinki)
|
||||
- ✅ Floating IPv6 configured and persisted via netplan
|
||||
- ✅ DNS configured (shardok.prod.eagle0.net → floating IP)
|
||||
- ✅ Let's Encrypt certificate obtained
|
||||
- ✅ NAT64 configured for IPv6-only server to reach IPv4 registries
|
||||
- ✅ Fixed runfiles error (missing SHARDOK_RESOURCES_PATH env var in docker run)
|
||||
- 🔄 Re-test container startup on Hetzner
|
||||
|
||||
### Known Issues
|
||||
|
||||
**IPv6-Only Servers Need NAT64**
|
||||
|
||||
Hetzner ARM instances are IPv6-only by default. DigitalOcean Container Registry's blob storage is IPv4-only. Solution: configure NAT64 DNS:
|
||||
|
||||
```bash
|
||||
mkdir -p /etc/systemd/resolved.conf.d
|
||||
cat > /etc/systemd/resolved.conf.d/nat64.conf << 'EOF'
|
||||
[Resolve]
|
||||
DNS=2a00:1098:2b::1 2a00:1098:2c::1
|
||||
EOF
|
||||
systemctl restart systemd-resolved
|
||||
```
|
||||
|
||||
**Floating IP Requires Manual Config**
|
||||
|
||||
Hetzner floating IPs must be added to netplan:
|
||||
|
||||
```yaml
|
||||
# In /etc/netplan/50-cloud-init.yaml, add to addresses list:
|
||||
addresses:
|
||||
- "2a01:4f9:xxxx:xxxx::1/64" # Server's own IP
|
||||
- "2a01:4f9:yyyy:yyyy::1/64" # Floating IP
|
||||
```
|
||||
|
||||
Then run `netplan apply`.
|
||||
|
||||
### Phase 2: Wire into Eagle
|
||||
|
||||
1. Call `onHumanTurn()` when human players submit commands
|
||||
2. Wire ShardokInstanceManager into Eagle's startup
|
||||
3. Add environment variables for Hetzner API token, floating IP
|
||||
4. Add "Preparing battle..." UI state for when Shardok is starting
|
||||
|
||||
### Phase 3: Monitoring & Polish
|
||||
|
||||
1. Add metrics and dashboards
|
||||
2. Implement alerting
|
||||
3. Cost tracking
|
||||
4. Documentation and runbooks
|
||||
|
||||
---
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Certificate Management
|
||||
|
||||
**Fully automated via Let's Encrypt + certbot.**
|
||||
|
||||
- Certbot runs on instance startup to obtain certificate
|
||||
- Systemd timer auto-renews every 60 days (certs valid 90 days)
|
||||
- No manual intervention required
|
||||
- Token rotation is manual but simple (generate new token, update both sides)
|
||||
|
||||
### Concurrent Battles
|
||||
|
||||
**Decision**: Single shared Shardok instance handles all concurrent battles.
|
||||
|
||||
Shardok already supports multiple simultaneous games. With 16 cores and activity-based spin-up, one instance is sufficient for the foreseeable future.
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
These are not needed now but documented for future consideration:
|
||||
|
||||
### Multi-Region Deployment
|
||||
|
||||
If player base expands globally:
|
||||
- Deploy Shardok instances in multiple Hetzner regions (Ashburn, Helsinki, Singapore)
|
||||
- Route players to nearest region based on their Eagle connection
|
||||
- Each region independent (no cross-region state sharing needed)
|
||||
|
||||
**Trigger**: Consistent complaints about battle latency from non-US players.
|
||||
|
||||
### Dynamic Instance Sizing
|
||||
|
||||
Could adjust instance size based on battle complexity:
|
||||
- Small battles (< 10 units): CAX21 (4 cores)
|
||||
- Medium battles: CAX31 (8 cores)
|
||||
- Large battles: CAX41 (16 cores)
|
||||
|
||||
**Current decision**: Start with CAX41 for all battles. The cost difference is minimal (~$0.02/hr between sizes) and simplicity is valuable.
|
||||
|
||||
### Dedicated Shardok Instances Per Game
|
||||
|
||||
For competitive/tournament play, could spin up dedicated instances:
|
||||
- Guaranteed resources, no contention
|
||||
- Isolated failures
|
||||
- Higher cost
|
||||
|
||||
**Current decision**: Shared instance is fine. Revisit if multiplayer competitive mode launches.
|
||||
Reference in New Issue
Block a user