admin/eagle0
1
0
Fork 0
mirror of https://github.com/nolen777/eagle0.git synced 2026-07-28 21:35:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Replace Hetzner plan doc with latency hiding strategies (#5053)

Browse Source 
The Hetzner on-demand compute implementation is complete:
- ARM64 builds working in CI
- Shardok running on Hetzner CAX41 in Helsinki
- Production Eagle connected via TLS + token auth
- IPv6/NAT64 networking configured

Delete the completed planning doc and add a new doc covering
future latency optimization strategies:
1. Client-side animation masking (recommended first step)
2. Split Shardok architecture (future if needed)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
admin
2026-01-04 13:37:53 -08:00
committed by GitHub
co-authored by Claude Opus 4.5
parent 811de67c40
commit 66efefc8d6
2 changed files with 216 additions and 769 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/SHARDOK_LATENCY_HIDING.md
+216
View File
@@ -0,0 +1,216 @@
# Shardok Latency Hiding Strategies
## Problem Statement
With Shardok running on Hetzner (Helsinki) and Eagle on DigitalOcean (US), the round-trip latency for human commands is ~200-400ms:
```
Human posts command:
Unity → Eagle (DO) → Shardok (Hetzner) → Eagle (DO) → Unity
~10ms ~100ms ~100ms ~10ms
Total: ~220ms round-trip
```
This latency is acceptable for AI turns (users watch animations anyway), but creates noticeable lag when humans post commands.
---
## Strategy 1: Client-Side Animation Masking
### Concept
Start animations immediately when the user clicks, before server confirmation arrives. The animation duration masks the network latency.
### Implementation by Command Type
**Movement Commands** (deterministic):
- Client knows the destination hex and movement path
- Start movement animation immediately on click
- Server confirms the move (should always match)
- If server rejects (invalid state), snap unit back to origin
**Attack Commands** (RNG-dependent):
- Show attack animation immediately (unit swings sword, fires arrow)
- Wait for server to return dice roll result
- Show damage numbers / hit effects when server responds
- Animation typically takes 300-500ms, masking most of the latency
**End Turn**:
- Latency not noticeable (user expects transition delay)
### Unity Implementation Sketch
```csharp
// In CommandHandler.cs
public void OnCommandSelected(Command command) {
// Start animation immediately
if (command.Type == CommandType.Move) {
unitController.StartMoveAnimation(command.TargetHex);
} else if (command.Type == CommandType.Attack) {
unitController.StartAttackAnimation(command.TargetUnit);
}
// Send to server in parallel
connection.SendCommand(command, (response) => {
if (response.Success) {
// Animation continues, apply result
ApplyCommandResult(response);
} else {
// Rollback animation
unitController.CancelAnimation();
ShowError(response.ErrorMessage);
}
});
}
```
### Pros
- Simple implementation
- No server-side changes
- Works with existing architecture
### Cons
- Doesn't eliminate latency for attacks with RNG (must wait for dice roll)
- Rollback needed if server rejects command (rare but possible)
### Estimated Improvement
- Movement: ~200ms latency hidden (feels instant)
- Attacks: ~100-200ms hidden by animation, ~100ms visible wait for dice result
---
## Strategy 2: Split Shardok Architecture
### Concept
Run two Shardok instances:
- **Shardok-Primary (DigitalOcean)**: Handles command processing, source of truth
- **Shardok-AI (Hetzner)**: AI computation only
Human commands go to the nearby Primary for low latency. AI computation uses the powerful Hetzner instance.
### Architecture
```
Human commands (low latency ~20ms)
Unity ←→ Eagle ←→ Shardok-Primary (DigitalOcean)
↓ state sync (when AI turn starts)
Shardok-AI (Hetzner)
↑ AI command response
```
### How It Works
**Human Turn:**
1. Human posts command → Eagle → Shardok-Primary (DO)
2. Primary processes command immediately (~10ms local)
3. Primary streams result to client via Eagle (~10ms)
4. **Total latency: ~20ms** (vs ~220ms current)
**AI Turn:**
1. When AI's turn starts, Primary sends game state snapshot to Hetzner
2. Shardok-AI computes best command using full CPU power
3. Shardok-AI returns command index to Primary
4. Primary executes command locally and streams to client
5. Repeat until AI turn ends
### Protocol Changes
```protobuf
// New service for AI-only computation
service ShardokAIService {
// Send game state, receive AI's chosen command
rpc GetAICommand(AICommandRequest) returns (AICommandResponse);
}
message AICommandRequest {
bytes game_state = 1; // Serialized game state
int32 player_id = 2; // Which AI player
repeated bytes available_commands = 3; // Available command descriptors
}
message AICommandResponse {
int32 command_index = 1; // Index into available_commands
int32 search_depth = 2; // For debugging
double best_score = 3; // For debugging
}
```
### Shardok-Primary Requirements
Shardok-Primary on DigitalOcean needs to:
- Process all commands (human and AI)
- Maintain authoritative game state
- Serialize/deserialize game state for AI requests
- Run on minimal CPU (command processing is fast)
This is essentially the current Shardok, but without running the AI search.
### Shardok-AI Requirements
Shardok-AI on Hetzner needs to:
- Receive game state snapshots
- Run AI evaluation (IterativeDeepeningAI or MCTS)
- Return best command index
- No persistent state (stateless worker)
### AI Turn Latency
Each AI command has ~200ms network latency. This is acceptable because:
1. User is watching animations anyway
2. Natural pacing lets user observe AI decisions
3. AI computation is fast on Hetzner's 16 cores
For a typical AI turn with 5 commands: 5 × 200ms = 1 second network overhead, plus AI thinking time. With animations, this feels natural.
### Implementation Phases
**Phase 1: Add Shardok-Primary (minimal)**
- Deploy existing Shardok container to DigitalOcean
- Configure Eagle to use local Shardok for all commands
- Human latency immediately improves
**Phase 2: Add AI offload**
- Implement `ShardokAIService` RPC
- Shardok-Primary calls Hetzner for AI commands
- Shardok-AI processes requests statelessly
**Phase 3: Optimize**
- Batch multiple AI actions if possible
- Pre-warm Shardok-AI connection
- Add fallback if Hetzner unavailable
### Pros
- Human command latency drops from ~220ms to ~20ms
- AI still gets Hetzner's CPU power
- Clear separation of concerns
- Shardok-Primary can fall back to local AI if Hetzner unavailable
### Cons
- Two Shardok instances to maintain
- State serialization overhead for AI requests
- Each AI action has network round-trip (acceptable with animations)
---
## Comparison
| Approach | Human Latency | AI Throughput | Complexity | Changes Required |
|----------|---------------|---------------|------------|------------------|
| Current | ~220ms | High | - | - |
| Animation masking | ~220ms (perceived ~50ms) | High | Low | Unity only |
| Split architecture | ~20ms | High | Medium | New RPC, two deployments |
---
## Recommendation
**Phase 1 (now)**: Implement animation masking in Unity client
- Quick win, no server changes
- Improves perceived latency significantly for movement
- Attacks still show dice animation while waiting
**Phase 2 (future)**: Split architecture if animation masking insufficient
- Only needed if users complain about attack latency
- More complex but provides true low latency
- Natural evolution of current architecture
docs/SHARDOK_ON_DEMAND_COMPUTE.md
-769
View File
@@ -1,769 +0,0 @@
# Shardok On-Demand Compute Infrastructure Analysis
## Overview
This document analyzes options for running Shardok (the tactical combat AI server) on powerful on-demand compute, separate from the low-power droplet that hosts Eagle.
**Current State**: Shardok runs on the same DigitalOcean droplet as Eagle, limiting AI performance.
**Goal**: Spin up powerful compute only when battles occur, keeping costs low while improving AI quality.
---
## Cloud Provider Pricing Comparison
### High-Performance CPU Options (8-16 cores)
| Provider | Instance Type | vCPUs | RAM | Hourly Cost | Monthly Est.* |
|----------|--------------|-------|-----|-------------|---------------|
| **Hetzner** | CCX23 | 8 | 32GB | €0.078 (~$0.085) | ~$6-8 |
| **Hetzner** | CCX33 | 16 | 64GB | €0.155 (~$0.17) | ~$12-16 |
| **Vultr** | High Frequency 8-core | 8 | 32GB | $0.238 | ~$20-25 |
| **Vultr** | High Frequency 16-core | 16 | 64GB | $0.476 | ~$40-50 |
| **DigitalOcean** | Premium CPU 8-core | 8 | 32GB | $0.250 | ~$22-28 |
| **DigitalOcean** | Premium CPU 16-core | 16 | 64GB | $0.500 | ~$44-55 |
*Monthly estimates assume ~4 hours/week of battle usage (~17 hours/month)
### Key Finding: Hetzner is 3-5x Cheaper
Hetzner's dedicated vCPU instances offer the best price-performance ratio by a significant margin. For equivalent 16-core compute:
- Hetzner: ~$0.17/hr
- DigitalOcean: ~$0.50/hr (2.9x more expensive)
- Vultr: ~$0.48/hr (2.8x more expensive)
---
## Instance Startup Time Analysis
### Current Startup Times by Provider
| Provider | Cold Start | Warm Image | Container Registry |
|----------|-----------|------------|-------------------|
| Hetzner | 30-60s | 20-40s | Native support |
| Vultr | 30-45s | 20-30s | Native support |
| DigitalOcean | 30-60s | 25-45s | Already in use |
### 10-Second Max Wait Time Constraint
A 10-second startup constraint eliminates traditional VM spin-up as an option. Alternatives:
#### Option 1: Pre-warmed Instance Pool (Recommended)
- Keep one small instance running 24/7 as a "hot standby"
- Cost: ~$5-10/month for a minimal Hetzner instance
- Startup time: **< 1 second** (already running)
- Upgrade on demand when battle starts (resize takes 2-3 min, but battle can start immediately on small instance)
#### Option 2: Predictive Spin-up
- Start spinning up when user enters battle setup
- Battle setup typically takes 30-60 seconds (map selection, army placement)
- Could achieve "instant" availability if spin-up completes during setup
- Risk: User might cancel, wasting spin-up cost (minimal at ~$0.01-0.02)
#### Option 3: Container-Based Quick Start
- Use pre-pulled Docker images on a warm container host
- Providers like Fly.io or Railway can start containers in 1-5 seconds
- Trade-off: Container platforms typically cost more than raw VMs
#### Option 4: Serverless/Function Compute
- Not viable for Shardok due to:
- Long-running connections (battles last minutes)
- High memory requirements
- Stateful game state
### Recommendation for 10s Constraint
**Hybrid approach**: Keep a minimal "always-on" instance ($5-10/month) that can handle battles at reduced AI quality. When a battle is anticipated (entering battle setup), start spinning up a powerful instance. If the powerful instance is ready when the battle starts, use it; otherwise, fall back to the always-on instance.
---
## ARM Options Outside AWS
### Available ARM Compute
| Provider | ARM Offering | vCPUs | RAM | Hourly Cost | Notes |
|----------|-------------|-------|-----|-------------|-------|
| **Hetzner** | CAX21 | 4 | 8GB | €0.009 | Ampere Altra |
| **Hetzner** | CAX31 | 8 | 16GB | €0.018 | Ampere Altra |
| **Hetzner** | CAX41 | 16 | 32GB | €0.035 | Ampere Altra |
| **Oracle Cloud** | A1 Flex | 4 | 24GB | Free tier | Always Free eligible |
| **Oracle Cloud** | A1 Flex | 24 | 144GB | ~$0.10 | Ampere Altra |
| **Scaleway** | AMP2-C8 | 8 | 16GB | €0.104 | Ampere Altra |
### Key Finding: Hetzner ARM is Extremely Cheap
Hetzner's ARM instances are 4-5x cheaper than their x86 equivalents:
- CAX41 (16 ARM cores): €0.035/hr (~$0.04)
- CCX33 (16 x86 cores): €0.155/hr (~$0.17)
### ARM Considerations for Shardok
**Pros:**
- Significant cost savings (4-5x)
- Good single-threaded performance on Ampere Altra
- Native ARM builds already supported (macOS development)
**Cons:**
- Requires ARM64 Linux build (currently only building x86_64 Linux)
- Some performance tuning may be needed
- Less mature ecosystem than x86
**Build Effort**: Medium. Bazel supports cross-compilation, and the codebase is portable C++. Main work is adding ARM64 Linux to the build matrix.
---
## Hetzner Geographic Presence
### Current Locations
Hetzner operates datacenters in:
- **Germany**: Nuremberg (NBG1), Falkenstein (FSN1)
- **Finland**: Helsinki (HEL1)
- **USA**: Ashburn, Virginia (ASH) - **opened 2023**
- **Singapore**: (coming 2024/2025)
### Latency Implications
For users primarily in North America:
- **Hetzner US (Ashburn)**: 20-40ms to East Coast, 60-80ms to West Coast
- **Hetzner EU**: 80-120ms from US East Coast, 140-180ms from US West Coast
Since Shardok handles AI computation (not real-time player input), latency is less critical. The Eagle-Shardok communication adds some overhead, but AI "thinking time" dominates.
### Recommendation
Use **Hetzner US (Ashburn)** for North American users. The US datacenter has the same pricing as EU datacenters and provides better latency.
---
## Implementation Options Summary
### Option A: Simple On-Demand (30-60s startup)
- Spin up Hetzner CCX33 when battle starts
- Cost: ~$0.17/hr × 4 hrs/week = **~$3/month**
- Latency: 30-60s wait at battle start
- Complexity: Low
### Option B: Predictive Spin-up (0-30s startup)
- Start spinning up during battle setup phase
- Cost: Same as Option A, plus occasional wasted spin-ups (~$1/month)
- Latency: 0-30s depending on setup duration
- Complexity: Medium
### Option C: Hybrid Always-On + On-Demand (< 1s startup)
- Keep minimal instance always running ($5-10/month)
- Spin up powerful instance during battle setup
- Use powerful if ready, fall back to minimal
- Cost: **~$8-15/month**
- Latency: < 1s (always-on fallback)
- Complexity: Medium-High
### Option D: ARM Compute (Cheapest)
- Build ARM64 Linux target
- Use Hetzner CAX41 (16 ARM cores) at €0.035/hr
- Cost: **~$0.60/month** for compute (+ always-on if needed)
- Latency: Same as x86 options
- Complexity: Medium (build system changes)
---
## Recommendation
**Phase 1 (Quick Win)**: Implement Option B (Predictive Spin-up) with Hetzner CCX33 in Ashburn.
- Minimal code changes (add spin-up trigger in Eagle when battle setup begins)
- Good user experience (usually ready by battle start)
- Low cost (~$3-5/month)
**Phase 2 (Future)**: Add ARM64 Linux build and switch to ARM instances.
- 4-5x cost reduction
- Same or better performance
- Requires build system work
**Phase 3 (If Needed)**: Add always-on fallback for guaranteed instant start.
- Only if Phase 1 startup times prove problematic
- Adds ~$5-10/month fixed cost
---
# Implementation Plan
## Chosen Architecture
- **Provider**: Hetzner Cloud
- **Instance**: CAX41 (16 ARM cores, 32GB RAM)
- **Location**: Ashburn, Virginia (ash)
- **Cost**: €0.035/hr (~$0.04/hr)
- **Strategy**: On-demand with predictive spin-up and idle keep-alive
---
## System Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ DigitalOcean │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Eagle Server │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ ShardokManager │ │ ActivityTracker │ │ │
│ │ │ - spin up/down │ │ - player active │ │ │
│ │ │ - health check │ │ - trigger start │ │ │
│ │ └────────┬────────┘ └────────┬────────┘ │ │
│ │ │ │ │ │
│ │ └────────┬───────────┘ │ │
│ │ ▼ │ │
│ │ ┌────────────────┐ │ │
│ │ │ ShardokClient │◄─── TLS + Token Auth │ │
│ │ └────────┬───────┘ │ │
│ └────────────────────┼─────────────────────────────────────────┘ │
└───────────────────────┼──────────────────────────────────────────────┘
│ gRPC over TLS (Let's Encrypt)
▼
┌───────────────────────────────────────────────────────────────────────┐
│ Hetzner Cloud (Ashburn) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ CAX41 (On-Demand) │ │
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
│ │ │ Shardok Server │ │ │
│ │ │ - 16 ARM cores for AI computation │ │ │
│ │ │ - TLS cert from Let's Encrypt (auto-renewed) │ │ │
│ │ │ - Validates auth token on each request │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────┘
```
---
## Instance Lifecycle Management
### State Machine
```
┌──────────────┐
│ STOPPED │◄────────────────────┐
└──────┬───────┘ │
│ human takes turn │ idle timeout
▼ │ (10 minutes)
┌──────────────┐ │
│ STARTING │ │
└──────┬───────┘ │
│ health check passes │
▼ │
┌──────────────┐ │
┌─────────►│ READY │─────────────────────┤
│ └──────┬───────┘ │
│ │ battle starts │
│ ▼ │
│ ┌──────────────┐ │
│ │ IN_USE │ │
│ └──────┬───────┘ │
│ │ battle ends │
└─────────────────┘
```
Note: The IDLE state was removed - we now simply track time since last human turn and shut down after 10 minutes of inactivity.
### Lifecycle Rules
1. **Spin-up Trigger**: Human player takes a turn
- Any command submitted by a human player to Eagle
- Simple and reliable - no prediction needed
2. **Keep-Alive**: Reset timer on each human turn
- Covers thinking time between turns
- Natural pauses (snacks, calls) under 10 min don't cause shutdown
3. **Shutdown**: 10 minutes since last human turn
- Hetzner API: `DELETE /servers/{id}` to stop billing
- Short timeout keeps costs low during idle periods
4. **Health Checks**: Every 10 seconds while STARTING
- gRPC connectivity check to Shardok
- Mark READY only when Shardok responds successfully
---
## Spin-up Strategy
### Simple Approach: Human Turn-Based
Spin up Shardok whenever a human player takes a turn. Shut down after 10 minutes of inactivity.
This is simpler than trying to predict battles. Battles happen regularly during active play, so keeping Shardok running while humans are playing ensures it's ready when needed.
**Spin-up trigger:**
- Any human player takes a turn (submits commands to Eagle)
**Shutdown trigger:**
- No human player has taken a turn in the last 10 minutes
**Why this works:**
- Battles are common during active play sessions
- 10-minute idle timeout is long enough to cover thinking time between turns
- Cost is so low (~$0.04/hr) that running during entire play sessions is fine
- No complex prediction logic needed
### Cost Impact
| Scenario | Hours/Week | Cost/Month |
|----------|------------|------------|
| Human turn-based (10 min idle) | ~15-20 hrs | ~$2.50-3.50 |
| Always-on 24/7 | 168 hrs | ~$29 |
Human turn-based is simple and cheap. The 10-minute timeout catches natural breaks (getting a snack, answering a call) while shutting down during longer idle periods.
### Implementation in Eagle
```scala
class ShardokInstanceManager {
private var instanceState: InstanceState = STOPPED
private var lastHumanTurnTime: Instant = Instant.MIN
private val idleTimeoutMinutes = 10
// Called when any human player submits a turn
def onHumanTurn(): Unit = {
lastHumanTurnTime = Instant.now()
if (instanceState == STOPPED) {
startInstance()
}
}
// Run every minute
def periodicShutdownCheck(): Unit = {
val idleTime = Duration.between(lastHumanTurnTime, Instant.now())
if (instanceState != STOPPED && idleTime > Duration.ofMinutes(idleTimeoutMinutes)) {
stopInstance()
}
}
}
```
### Integration Point
Wire `onHumanTurn()` into Eagle's command processing. The cleanest place is where Eagle receives player commands from the Unity client - likely in `EagleServiceImpl` when handling the gRPC streaming commands.
---
## Security: Eagle ↔ Shardok Authentication
Since Eagle (DigitalOcean) and Shardok (Hetzner) are on different cloud providers, they cannot share a private network. We need secure authentication over the public internet.
### Approach: TLS + Token Authentication
This is the same pattern used by Stripe, GitHub, and most cloud APIs.
**Layer 1: TLS Encryption (Let's Encrypt)**
- Server certificate from Let's Encrypt, auto-renewed via certbot
- All gRPC traffic encrypted with TLS 1.3
- Prevents eavesdropping and MITM attacks
**Layer 2: Token Authentication**
- 256-bit random shared secret in gRPC metadata
- Only requests with valid token are processed
- Easy to rotate without any certificate changes
### How Token Auth Works
**Setup (once):**
```bash
# Generate a 256-bit random token
openssl rand -hex 32
# Output: a3f8b2c9d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1
# Store in GitHub Secrets as SHARDOK_AUTH_TOKEN
# Eagle reads from environment, Shardok receives via cloud-init
```
**Every Request:**
```
Eagle Shardok
────── ───────
gRPC call with metadata: → Validate token:
"authorization: Bearer a3f8b2c9..." if token != expected:
return UNAUTHENTICATED
else:
process request
```
### Server Certificate (Let's Encrypt)
Shardok needs a domain name for Let's Encrypt. Options:
1. **Dynamic DNS**: Assign `shardok.prod.eagle0.net` to instance IP on spin-up
2. **Elastic IP**: Reserve a Hetzner floating IP, always points to active instance
Cloud-init runs certbot on first boot:
```yaml
runcmd:
# Install certbot
- apt-get install -y certbot
# Get certificate (DNS must already point to this IP)
- certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net
# Certificate auto-renews via systemd timer (certbot installs this)
# Certs at: /etc/letsencrypt/live/shardok.prod.eagle0.net/
```
### Implementation
**Shardok gRPC Server (C++):**
```cpp
// Load Let's Encrypt certificates
std::string cert = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem");
std::string key = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem");
grpc::SslServerCredentialsOptions ssl_opts;
ssl_opts.pem_key_cert_pairs.push_back({key, cert});
auto creds = grpc::SslServerCredentials(ssl_opts);
builder.AddListeningPort("0.0.0.0:50051", creds);
// Token validation - check on every RPC
Status PostCommand(ServerContext* context, const Request* req, Response* resp) {
auto auth = context->client_metadata().find("authorization");
std::string expected = "Bearer " + auth_token_;
if (auth == context->client_metadata().end() ||
auth->second != expected) {
return Status(grpc::UNAUTHENTICATED, "Invalid token");
}
// ... handle request
}
```
**Eagle gRPC Client (Scala):**
```scala
// Standard TLS (trusts Let's Encrypt via system CA store)
val channel = NettyChannelBuilder
.forAddress("shardok.prod.eagle0.net", 50051)
.useTransportSecurity() // TLS with system trust store
.intercept(new ClientInterceptor {
override def interceptCall[Req, Resp](
method: MethodDescriptor[Req, Resp],
callOptions: CallOptions,
next: Channel): ClientCall[Req, Resp] = {
new ForwardingClientCall.SimpleForwardingClientCall(
next.newCall(method, callOptions)) {
override def start(listener: Listener[Resp], headers: Metadata): Unit = {
headers.put(
Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER),
s"Bearer $authToken"
)
super.start(listener, headers)
}
}
}
})
.build()
```
### Token Rotation
When rotating tokens (e.g., if compromised or as routine hygiene):
1. Generate new token
2. Update Shardok to accept **both** old and new tokens temporarily
3. Update Eagle to use new token
4. Remove old token from Shardok
No downtime, no certificate regeneration.
### Firewall Rules
```bash
# Hetzner firewall via cloud-init
ufw default deny incoming
ufw allow 80/tcp # Let's Encrypt HTTP-01 challenge (certbot)
ufw allow 50051/tcp # gRPC (protected by TLS + token)
ufw allow from <admin-ip>/32 to any port 22 # SSH for debugging
ufw enable
```
### Security Summary
| Threat | Mitigation |
|--------|------------|
| Eavesdropping | TLS 1.3 encryption |
| MITM attack | Let's Encrypt certificate validates server identity |
| Unauthorized access | 256-bit token required on every request |
| Token theft | TLS prevents sniffing; rotate if compromised |
This is the industry-standard approach for API security.
---
## Hetzner API Integration
### API Authentication
```bash
# Store in secrets manager
HETZNER_API_TOKEN=<token from Hetzner Cloud Console>
```
### Key API Operations
**Create Instance:**
```bash
curl -X POST "https://api.hetzner.cloud/v1/servers" \
-H "Authorization: Bearer $HETZNER_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "shardok-prod",
"server_type": "cax41",
"location": "ash",
"image": "docker-ce",
"ssh_keys": ["eagle-deploy"],
"user_data": "<cloud-init script>",
"labels": {"service": "shardok", "env": "prod"}
}'
```
**Check Status:**
```bash
curl "https://api.hetzner.cloud/v1/servers/{id}" \
-H "Authorization: Bearer $HETZNER_API_TOKEN"
# Response includes: status (running/starting/off), public IP
```
**Delete Instance (Stop Billing):**
```bash
curl -X DELETE "https://api.hetzner.cloud/v1/servers/{id}" \
-H "Authorization: Bearer $HETZNER_API_TOKEN"
```
### Cloud-Init Script
```yaml
#cloud-config
package_update: true
packages:
- docker.io
- certbot
write_files:
- path: /etc/shardok/auth_token
permissions: '0600'
content: |
${AUTH_TOKEN}
runcmd:
# Get Let's Encrypt certificate (DNS must point to this IP first)
- certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net
# Pull and run Shardok container
- echo ${DO_REGISTRY_TOKEN} | docker login registry.digitalocean.com -u ${DO_REGISTRY_TOKEN} --password-stdin
- docker pull registry.digitalocean.com/eagle0/shardok-server:arm64-latest
- docker run -d --name shardok \
-p 50051:50051 \
-v /etc/letsencrypt:/etc/letsencrypt:ro \
-v /etc/shardok:/etc/shardok:ro \
registry.digitalocean.com/eagle0/shardok-server:arm64-latest \
--cert=/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem \
--key=/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem \
--token-file=/etc/shardok/auth_token
```
---
## Container Registry
Using DigitalOcean Container Registry for all images (consistent with Eagle and other services).
Both architectures use the same repository with different tag prefixes:
- ARM64 image: `registry.digitalocean.com/eagle0/shardok-server:arm64-latest`
- x86 image: `registry.digitalocean.com/eagle0/shardok-server:latest`
CI automatically builds and pushes ARM64 images when C++ code changes.
---
## Fallback Strategy
### If Shardok Instance Not Ready
When a battle starts but the Hetzner instance isn't ready:
1. **Option A: Wait with Progress Indicator**
- Show "Preparing battle arena..." with spinner
- Most users won't notice 20-30s wait if UI is engaging
- Simplest implementation
2. **Option B: Local Fallback** (Current behavior)
- Run Shardok on Eagle's droplet temporarily
- Lower AI quality but instant start
- Migrate battle to Hetzner when ready (complex)
3. **Option C: Reduced AI Settings**
- Start battle immediately with very short time budgets
- Improve AI quality when Hetzner ready
- Seamless to user but requires AI hot-swap
**Recommendation**: Start with Option A. If user feedback indicates wait time is problematic, implement Option B.
---
## Monitoring & Observability
### Metrics to Track
| Metric | Purpose |
|--------|---------|
| `shardok_instance_state` | Current lifecycle state |
| `shardok_startup_duration_seconds` | Time from spin-up to ready |
| `shardok_battles_per_instance` | Efficiency of keep-alive |
| `shardok_wasted_spinups` | Spin-ups without battles |
| `shardok_hourly_cost` | Running cost tracking |
### Alerting
- Alert if startup takes > 90 seconds
- Alert if instance stuck in STARTING for > 3 minutes
- Alert if Hetzner API errors persist
---
## Cost Estimate (Activity-Based, ARM)
With activity-based spin-up, Shardok runs during active play sessions:
| Play Pattern | Hours/Week | Compute/Month |
|--------------|------------|---------------|
| Casual (few sessions) | ~10 hrs | **~$1.75** |
| Regular (daily play) | ~20 hrs | **~$3.50** |
| Heavy (multiple daily) | ~40 hrs | **~$7.00** |
Plus fixed costs:
- Container registry: $0 (GitHub Container Registry)
- Secrets management: $0 (GitHub Secrets)
**Total estimated cost: $2-7/month** for typical usage.
Compare to current DigitalOcean droplet upgrade (~$20-40/month for equivalent CPU power).
---
## Implementation Phases
### Phase 1: Basic On-Demand (1-2 weeks)
1. ✅ Add ARM64 Linux to CI build matrix (PR #4990, merged 2026-01-02)
- Added LLVM toolchain for ARM64 cross-compilation
- Created ARM64 sysroot build workflow
- Added Ubuntu 24.04 ARM64 base image for containers
2. ✅ Push ARM64 container to DigitalOcean Container Registry (PR #4990)
- Created shardok_arm64_build.yml workflow
- Pushes to registry.digitalocean.com/eagle0/shardok-server:arm64-latest
3. ✅ Implement Hetzner API client in Eagle (Scala) (PR #4996, merged 2026-01-02)
- HetznerApiClient with create/delete/list/power operations
- Async HTTP via OkHttp with Future-based API
4. ✅ Add ShardokInstanceManager with spin-up/shutdown logic (PR #4998, merged 2026-01-02)
- Activity-based spin-up when players connect
- Idle timeout shutdown (default 60 minutes)
- State machine: Stopped → Starting → Ready → InUse → Stopping
5. ✅ Add TLS + token authentication to Shardok server (PR #5001, merged 2026-01-02)
- C++ server: TLS via Let's Encrypt, Bearer token validation
- Scala client: TLS and auth token support in ServerSetupHelpers
6. 🔄 Test end-to-end with manual triggers ← **CURRENT**
- ✅ Hetzner server created (CAX41 ARM64 in Helsinki)
- ✅ Floating IPv6 configured and persisted via netplan
- ✅ DNS configured (shardok.prod.eagle0.net → floating IP)
- ✅ Let's Encrypt certificate obtained
- ✅ NAT64 configured for IPv6-only server to reach IPv4 registries
- ✅ Fixed runfiles error (missing SHARDOK_RESOURCES_PATH env var in docker run)
- 🔄 Re-test container startup on Hetzner
### Known Issues
**IPv6-Only Servers Need NAT64**
Hetzner ARM instances are IPv6-only by default. DigitalOcean Container Registry's blob storage is IPv4-only. Solution: configure NAT64 DNS:
```bash
mkdir -p /etc/systemd/resolved.conf.d
cat > /etc/systemd/resolved.conf.d/nat64.conf << 'EOF'
[Resolve]
DNS=2a00:1098:2b::1 2a00:1098:2c::1
EOF
systemctl restart systemd-resolved
```
**Floating IP Requires Manual Config**
Hetzner floating IPs must be added to netplan:
```yaml
# In /etc/netplan/50-cloud-init.yaml, add to addresses list:
addresses:
- "2a01:4f9:xxxx:xxxx::1/64" # Server's own IP
- "2a01:4f9:yyyy:yyyy::1/64" # Floating IP
```
Then run `netplan apply`.
### Phase 2: Wire into Eagle
1. Call `onHumanTurn()` when human players submit commands
2. Wire ShardokInstanceManager into Eagle's startup
3. Add environment variables for Hetzner API token, floating IP
4. Add "Preparing battle..." UI state for when Shardok is starting
### Phase 3: Monitoring & Polish
1. Add metrics and dashboards
2. Implement alerting
3. Cost tracking
4. Documentation and runbooks
---
## Design Decisions
### Certificate Management
**Fully automated via Let's Encrypt + certbot.**
- Certbot runs on instance startup to obtain certificate
- Systemd timer auto-renews every 60 days (certs valid 90 days)
- No manual intervention required
- Token rotation is manual but simple (generate new token, update both sides)
### Concurrent Battles
**Decision**: Single shared Shardok instance handles all concurrent battles.
Shardok already supports multiple simultaneous games. With 16 cores and activity-based spin-up, one instance is sufficient for the foreseeable future.
---
## Future Enhancements
These are not needed now but documented for future consideration:
### Multi-Region Deployment
If player base expands globally:
- Deploy Shardok instances in multiple Hetzner regions (Ashburn, Helsinki, Singapore)
- Route players to nearest region based on their Eagle connection
- Each region independent (no cross-region state sharing needed)
**Trigger**: Consistent complaints about battle latency from non-US players.
### Dynamic Instance Sizing
Could adjust instance size based on battle complexity:
- Small battles (< 10 units): CAX21 (4 cores)
- Medium battles: CAX31 (8 cores)
- Large battles: CAX41 (16 cores)
**Current decision**: Start with CAX41 for all battles. The cost difference is minimal (~$0.02/hr between sizes) and simplicity is valuable.
### Dedicated Shardok Instances Per Game
For competitive/tournament play, could spin up dedicated instances:
- Guaranteed resources, no contention
- Isolated failures
- Higher cost
**Current decision**: Shared instance is fine. Revisit if multiplayer competitive mode launches.