From 66efefc8d66c1104e26489053a2b9b512dabbe2c Mon Sep 17 00:00:00 2001 From: Dan Crosby Date: Sun, 4 Jan 2026 13:37:53 -0800 Subject: [PATCH] Replace Hetzner plan doc with latency hiding strategies (#5053) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Hetzner on-demand compute implementation is complete: - ARM64 builds working in CI - Shardok running on Hetzner CAX41 in Helsinki - Production Eagle connected via TLS + token auth - IPv6/NAT64 networking configured Delete the completed planning doc and add a new doc covering future latency optimization strategies: 1. Client-side animation masking (recommended first step) 2. Split Shardok architecture (future if needed) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 --- docs/SHARDOK_LATENCY_HIDING.md | 216 +++++++++ docs/SHARDOK_ON_DEMAND_COMPUTE.md | 769 ------------------------------ 2 files changed, 216 insertions(+), 769 deletions(-) create mode 100644 docs/SHARDOK_LATENCY_HIDING.md delete mode 100644 docs/SHARDOK_ON_DEMAND_COMPUTE.md diff --git a/docs/SHARDOK_LATENCY_HIDING.md b/docs/SHARDOK_LATENCY_HIDING.md new file mode 100644 index 0000000000..c2b986fab1 --- /dev/null +++ b/docs/SHARDOK_LATENCY_HIDING.md @@ -0,0 +1,216 @@ +# Shardok Latency Hiding Strategies + +## Problem Statement + +With Shardok running on Hetzner (Helsinki) and Eagle on DigitalOcean (US), the round-trip latency for human commands is ~200-400ms: + +``` +Human posts command: +Unity → Eagle (DO) → Shardok (Hetzner) → Eagle (DO) → Unity + ~10ms ~100ms ~100ms ~10ms + Total: ~220ms round-trip +``` + +This latency is acceptable for AI turns (users watch animations anyway), but creates noticeable lag when humans post commands. + +--- + +## Strategy 1: Client-Side Animation Masking + +### Concept + +Start animations immediately when the user clicks, before server confirmation arrives. The animation duration masks the network latency. + +### Implementation by Command Type + +**Movement Commands** (deterministic): +- Client knows the destination hex and movement path +- Start movement animation immediately on click +- Server confirms the move (should always match) +- If server rejects (invalid state), snap unit back to origin + +**Attack Commands** (RNG-dependent): +- Show attack animation immediately (unit swings sword, fires arrow) +- Wait for server to return dice roll result +- Show damage numbers / hit effects when server responds +- Animation typically takes 300-500ms, masking most of the latency + +**End Turn**: +- Latency not noticeable (user expects transition delay) + +### Unity Implementation Sketch + +```csharp +// In CommandHandler.cs +public void OnCommandSelected(Command command) { + // Start animation immediately + if (command.Type == CommandType.Move) { + unitController.StartMoveAnimation(command.TargetHex); + } else if (command.Type == CommandType.Attack) { + unitController.StartAttackAnimation(command.TargetUnit); + } + + // Send to server in parallel + connection.SendCommand(command, (response) => { + if (response.Success) { + // Animation continues, apply result + ApplyCommandResult(response); + } else { + // Rollback animation + unitController.CancelAnimation(); + ShowError(response.ErrorMessage); + } + }); +} +``` + +### Pros +- Simple implementation +- No server-side changes +- Works with existing architecture + +### Cons +- Doesn't eliminate latency for attacks with RNG (must wait for dice roll) +- Rollback needed if server rejects command (rare but possible) + +### Estimated Improvement +- Movement: ~200ms latency hidden (feels instant) +- Attacks: ~100-200ms hidden by animation, ~100ms visible wait for dice result + +--- + +## Strategy 2: Split Shardok Architecture + +### Concept + +Run two Shardok instances: +- **Shardok-Primary (DigitalOcean)**: Handles command processing, source of truth +- **Shardok-AI (Hetzner)**: AI computation only + +Human commands go to the nearby Primary for low latency. AI computation uses the powerful Hetzner instance. + +### Architecture + +``` + Human commands (low latency ~20ms) +Unity ←→ Eagle ←→ Shardok-Primary (DigitalOcean) + ↓ state sync (when AI turn starts) + Shardok-AI (Hetzner) + ↑ AI command response +``` + +### How It Works + +**Human Turn:** +1. Human posts command → Eagle → Shardok-Primary (DO) +2. Primary processes command immediately (~10ms local) +3. Primary streams result to client via Eagle (~10ms) +4. **Total latency: ~20ms** (vs ~220ms current) + +**AI Turn:** +1. When AI's turn starts, Primary sends game state snapshot to Hetzner +2. Shardok-AI computes best command using full CPU power +3. Shardok-AI returns command index to Primary +4. Primary executes command locally and streams to client +5. Repeat until AI turn ends + +### Protocol Changes + +```protobuf +// New service for AI-only computation +service ShardokAIService { + // Send game state, receive AI's chosen command + rpc GetAICommand(AICommandRequest) returns (AICommandResponse); +} + +message AICommandRequest { + bytes game_state = 1; // Serialized game state + int32 player_id = 2; // Which AI player + repeated bytes available_commands = 3; // Available command descriptors +} + +message AICommandResponse { + int32 command_index = 1; // Index into available_commands + int32 search_depth = 2; // For debugging + double best_score = 3; // For debugging +} +``` + +### Shardok-Primary Requirements + +Shardok-Primary on DigitalOcean needs to: +- Process all commands (human and AI) +- Maintain authoritative game state +- Serialize/deserialize game state for AI requests +- Run on minimal CPU (command processing is fast) + +This is essentially the current Shardok, but without running the AI search. + +### Shardok-AI Requirements + +Shardok-AI on Hetzner needs to: +- Receive game state snapshots +- Run AI evaluation (IterativeDeepeningAI or MCTS) +- Return best command index +- No persistent state (stateless worker) + +### AI Turn Latency + +Each AI command has ~200ms network latency. This is acceptable because: +1. User is watching animations anyway +2. Natural pacing lets user observe AI decisions +3. AI computation is fast on Hetzner's 16 cores + +For a typical AI turn with 5 commands: 5 × 200ms = 1 second network overhead, plus AI thinking time. With animations, this feels natural. + +### Implementation Phases + +**Phase 1: Add Shardok-Primary (minimal)** +- Deploy existing Shardok container to DigitalOcean +- Configure Eagle to use local Shardok for all commands +- Human latency immediately improves + +**Phase 2: Add AI offload** +- Implement `ShardokAIService` RPC +- Shardok-Primary calls Hetzner for AI commands +- Shardok-AI processes requests statelessly + +**Phase 3: Optimize** +- Batch multiple AI actions if possible +- Pre-warm Shardok-AI connection +- Add fallback if Hetzner unavailable + +### Pros +- Human command latency drops from ~220ms to ~20ms +- AI still gets Hetzner's CPU power +- Clear separation of concerns +- Shardok-Primary can fall back to local AI if Hetzner unavailable + +### Cons +- Two Shardok instances to maintain +- State serialization overhead for AI requests +- Each AI action has network round-trip (acceptable with animations) + +--- + +## Comparison + +| Approach | Human Latency | AI Throughput | Complexity | Changes Required | +|----------|---------------|---------------|------------|------------------| +| Current | ~220ms | High | - | - | +| Animation masking | ~220ms (perceived ~50ms) | High | Low | Unity only | +| Split architecture | ~20ms | High | Medium | New RPC, two deployments | + +--- + +## Recommendation + +**Phase 1 (now)**: Implement animation masking in Unity client +- Quick win, no server changes +- Improves perceived latency significantly for movement +- Attacks still show dice animation while waiting + +**Phase 2 (future)**: Split architecture if animation masking insufficient +- Only needed if users complain about attack latency +- More complex but provides true low latency +- Natural evolution of current architecture diff --git a/docs/SHARDOK_ON_DEMAND_COMPUTE.md b/docs/SHARDOK_ON_DEMAND_COMPUTE.md deleted file mode 100644 index cb91a26fab..0000000000 --- a/docs/SHARDOK_ON_DEMAND_COMPUTE.md +++ /dev/null @@ -1,769 +0,0 @@ -# Shardok On-Demand Compute Infrastructure Analysis - -## Overview - -This document analyzes options for running Shardok (the tactical combat AI server) on powerful on-demand compute, separate from the low-power droplet that hosts Eagle. - -**Current State**: Shardok runs on the same DigitalOcean droplet as Eagle, limiting AI performance. - -**Goal**: Spin up powerful compute only when battles occur, keeping costs low while improving AI quality. - ---- - -## Cloud Provider Pricing Comparison - -### High-Performance CPU Options (8-16 cores) - -| Provider | Instance Type | vCPUs | RAM | Hourly Cost | Monthly Est.* | -|----------|--------------|-------|-----|-------------|---------------| -| **Hetzner** | CCX23 | 8 | 32GB | €0.078 (~$0.085) | ~$6-8 | -| **Hetzner** | CCX33 | 16 | 64GB | €0.155 (~$0.17) | ~$12-16 | -| **Vultr** | High Frequency 8-core | 8 | 32GB | $0.238 | ~$20-25 | -| **Vultr** | High Frequency 16-core | 16 | 64GB | $0.476 | ~$40-50 | -| **DigitalOcean** | Premium CPU 8-core | 8 | 32GB | $0.250 | ~$22-28 | -| **DigitalOcean** | Premium CPU 16-core | 16 | 64GB | $0.500 | ~$44-55 | - -*Monthly estimates assume ~4 hours/week of battle usage (~17 hours/month) - -### Key Finding: Hetzner is 3-5x Cheaper - -Hetzner's dedicated vCPU instances offer the best price-performance ratio by a significant margin. For equivalent 16-core compute: -- Hetzner: ~$0.17/hr -- DigitalOcean: ~$0.50/hr (2.9x more expensive) -- Vultr: ~$0.48/hr (2.8x more expensive) - ---- - -## Instance Startup Time Analysis - -### Current Startup Times by Provider - -| Provider | Cold Start | Warm Image | Container Registry | -|----------|-----------|------------|-------------------| -| Hetzner | 30-60s | 20-40s | Native support | -| Vultr | 30-45s | 20-30s | Native support | -| DigitalOcean | 30-60s | 25-45s | Already in use | - -### 10-Second Max Wait Time Constraint - -A 10-second startup constraint eliminates traditional VM spin-up as an option. Alternatives: - -#### Option 1: Pre-warmed Instance Pool (Recommended) -- Keep one small instance running 24/7 as a "hot standby" -- Cost: ~$5-10/month for a minimal Hetzner instance -- Startup time: **< 1 second** (already running) -- Upgrade on demand when battle starts (resize takes 2-3 min, but battle can start immediately on small instance) - -#### Option 2: Predictive Spin-up -- Start spinning up when user enters battle setup -- Battle setup typically takes 30-60 seconds (map selection, army placement) -- Could achieve "instant" availability if spin-up completes during setup -- Risk: User might cancel, wasting spin-up cost (minimal at ~$0.01-0.02) - -#### Option 3: Container-Based Quick Start -- Use pre-pulled Docker images on a warm container host -- Providers like Fly.io or Railway can start containers in 1-5 seconds -- Trade-off: Container platforms typically cost more than raw VMs - -#### Option 4: Serverless/Function Compute -- Not viable for Shardok due to: - - Long-running connections (battles last minutes) - - High memory requirements - - Stateful game state - -### Recommendation for 10s Constraint - -**Hybrid approach**: Keep a minimal "always-on" instance ($5-10/month) that can handle battles at reduced AI quality. When a battle is anticipated (entering battle setup), start spinning up a powerful instance. If the powerful instance is ready when the battle starts, use it; otherwise, fall back to the always-on instance. - ---- - -## ARM Options Outside AWS - -### Available ARM Compute - -| Provider | ARM Offering | vCPUs | RAM | Hourly Cost | Notes | -|----------|-------------|-------|-----|-------------|-------| -| **Hetzner** | CAX21 | 4 | 8GB | €0.009 | Ampere Altra | -| **Hetzner** | CAX31 | 8 | 16GB | €0.018 | Ampere Altra | -| **Hetzner** | CAX41 | 16 | 32GB | €0.035 | Ampere Altra | -| **Oracle Cloud** | A1 Flex | 4 | 24GB | Free tier | Always Free eligible | -| **Oracle Cloud** | A1 Flex | 24 | 144GB | ~$0.10 | Ampere Altra | -| **Scaleway** | AMP2-C8 | 8 | 16GB | €0.104 | Ampere Altra | - -### Key Finding: Hetzner ARM is Extremely Cheap - -Hetzner's ARM instances are 4-5x cheaper than their x86 equivalents: -- CAX41 (16 ARM cores): €0.035/hr (~$0.04) -- CCX33 (16 x86 cores): €0.155/hr (~$0.17) - -### ARM Considerations for Shardok - -**Pros:** -- Significant cost savings (4-5x) -- Good single-threaded performance on Ampere Altra -- Native ARM builds already supported (macOS development) - -**Cons:** -- Requires ARM64 Linux build (currently only building x86_64 Linux) -- Some performance tuning may be needed -- Less mature ecosystem than x86 - -**Build Effort**: Medium. Bazel supports cross-compilation, and the codebase is portable C++. Main work is adding ARM64 Linux to the build matrix. - ---- - -## Hetzner Geographic Presence - -### Current Locations - -Hetzner operates datacenters in: -- **Germany**: Nuremberg (NBG1), Falkenstein (FSN1) -- **Finland**: Helsinki (HEL1) -- **USA**: Ashburn, Virginia (ASH) - **opened 2023** -- **Singapore**: (coming 2024/2025) - -### Latency Implications - -For users primarily in North America: -- **Hetzner US (Ashburn)**: 20-40ms to East Coast, 60-80ms to West Coast -- **Hetzner EU**: 80-120ms from US East Coast, 140-180ms from US West Coast - -Since Shardok handles AI computation (not real-time player input), latency is less critical. The Eagle-Shardok communication adds some overhead, but AI "thinking time" dominates. - -### Recommendation - -Use **Hetzner US (Ashburn)** for North American users. The US datacenter has the same pricing as EU datacenters and provides better latency. - ---- - -## Implementation Options Summary - -### Option A: Simple On-Demand (30-60s startup) -- Spin up Hetzner CCX33 when battle starts -- Cost: ~$0.17/hr × 4 hrs/week = **~$3/month** -- Latency: 30-60s wait at battle start -- Complexity: Low - -### Option B: Predictive Spin-up (0-30s startup) -- Start spinning up during battle setup phase -- Cost: Same as Option A, plus occasional wasted spin-ups (~$1/month) -- Latency: 0-30s depending on setup duration -- Complexity: Medium - -### Option C: Hybrid Always-On + On-Demand (< 1s startup) -- Keep minimal instance always running ($5-10/month) -- Spin up powerful instance during battle setup -- Use powerful if ready, fall back to minimal -- Cost: **~$8-15/month** -- Latency: < 1s (always-on fallback) -- Complexity: Medium-High - -### Option D: ARM Compute (Cheapest) -- Build ARM64 Linux target -- Use Hetzner CAX41 (16 ARM cores) at €0.035/hr -- Cost: **~$0.60/month** for compute (+ always-on if needed) -- Latency: Same as x86 options -- Complexity: Medium (build system changes) - ---- - -## Recommendation - -**Phase 1 (Quick Win)**: Implement Option B (Predictive Spin-up) with Hetzner CCX33 in Ashburn. -- Minimal code changes (add spin-up trigger in Eagle when battle setup begins) -- Good user experience (usually ready by battle start) -- Low cost (~$3-5/month) - -**Phase 2 (Future)**: Add ARM64 Linux build and switch to ARM instances. -- 4-5x cost reduction -- Same or better performance -- Requires build system work - -**Phase 3 (If Needed)**: Add always-on fallback for guaranteed instant start. -- Only if Phase 1 startup times prove problematic -- Adds ~$5-10/month fixed cost - ---- - -# Implementation Plan - -## Chosen Architecture - -- **Provider**: Hetzner Cloud -- **Instance**: CAX41 (16 ARM cores, 32GB RAM) -- **Location**: Ashburn, Virginia (ash) -- **Cost**: €0.035/hr (~$0.04/hr) -- **Strategy**: On-demand with predictive spin-up and idle keep-alive - ---- - -## System Architecture - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ DigitalOcean │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ Eagle Server │ │ -│ │ ┌─────────────────┐ ┌─────────────────┐ │ │ -│ │ │ ShardokManager │ │ ActivityTracker │ │ │ -│ │ │ - spin up/down │ │ - player active │ │ │ -│ │ │ - health check │ │ - trigger start │ │ │ -│ │ └────────┬────────┘ └────────┬────────┘ │ │ -│ │ │ │ │ │ -│ │ └────────┬───────────┘ │ │ -│ │ ▼ │ │ -│ │ ┌────────────────┐ │ │ -│ │ │ ShardokClient │◄─── TLS + Token Auth │ │ -│ │ └────────┬───────┘ │ │ -│ └────────────────────┼─────────────────────────────────────────┘ │ -└───────────────────────┼──────────────────────────────────────────────┘ - │ gRPC over TLS (Let's Encrypt) - ▼ -┌───────────────────────────────────────────────────────────────────────┐ -│ Hetzner Cloud (Ashburn) │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ CAX41 (On-Demand) │ │ -│ │ ┌─────────────────────────────────────────────────────────┐ │ │ -│ │ │ Shardok Server │ │ │ -│ │ │ - 16 ARM cores for AI computation │ │ │ -│ │ │ - TLS cert from Let's Encrypt (auto-renewed) │ │ │ -│ │ │ - Validates auth token on each request │ │ │ -│ │ └─────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└───────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## Instance Lifecycle Management - -### State Machine - -``` - ┌──────────────┐ - │ STOPPED │◄────────────────────┐ - └──────┬───────┘ │ - │ human takes turn │ idle timeout - ▼ │ (10 minutes) - ┌──────────────┐ │ - │ STARTING │ │ - └──────┬───────┘ │ - │ health check passes │ - ▼ │ - ┌──────────────┐ │ - ┌─────────►│ READY │─────────────────────┤ - │ └──────┬───────┘ │ - │ │ battle starts │ - │ ▼ │ - │ ┌──────────────┐ │ - │ │ IN_USE │ │ - │ └──────┬───────┘ │ - │ │ battle ends │ - └─────────────────┘ -``` - -Note: The IDLE state was removed - we now simply track time since last human turn and shut down after 10 minutes of inactivity. - -### Lifecycle Rules - -1. **Spin-up Trigger**: Human player takes a turn - - Any command submitted by a human player to Eagle - - Simple and reliable - no prediction needed - -2. **Keep-Alive**: Reset timer on each human turn - - Covers thinking time between turns - - Natural pauses (snacks, calls) under 10 min don't cause shutdown - -3. **Shutdown**: 10 minutes since last human turn - - Hetzner API: `DELETE /servers/{id}` to stop billing - - Short timeout keeps costs low during idle periods - -4. **Health Checks**: Every 10 seconds while STARTING - - gRPC connectivity check to Shardok - - Mark READY only when Shardok responds successfully - ---- - -## Spin-up Strategy - -### Simple Approach: Human Turn-Based - -Spin up Shardok whenever a human player takes a turn. Shut down after 10 minutes of inactivity. - -This is simpler than trying to predict battles. Battles happen regularly during active play, so keeping Shardok running while humans are playing ensures it's ready when needed. - -**Spin-up trigger:** -- Any human player takes a turn (submits commands to Eagle) - -**Shutdown trigger:** -- No human player has taken a turn in the last 10 minutes - -**Why this works:** -- Battles are common during active play sessions -- 10-minute idle timeout is long enough to cover thinking time between turns -- Cost is so low (~$0.04/hr) that running during entire play sessions is fine -- No complex prediction logic needed - -### Cost Impact - -| Scenario | Hours/Week | Cost/Month | -|----------|------------|------------| -| Human turn-based (10 min idle) | ~15-20 hrs | ~$2.50-3.50 | -| Always-on 24/7 | 168 hrs | ~$29 | - -Human turn-based is simple and cheap. The 10-minute timeout catches natural breaks (getting a snack, answering a call) while shutting down during longer idle periods. - -### Implementation in Eagle - -```scala -class ShardokInstanceManager { - private var instanceState: InstanceState = STOPPED - private var lastHumanTurnTime: Instant = Instant.MIN - private val idleTimeoutMinutes = 10 - - // Called when any human player submits a turn - def onHumanTurn(): Unit = { - lastHumanTurnTime = Instant.now() - if (instanceState == STOPPED) { - startInstance() - } - } - - // Run every minute - def periodicShutdownCheck(): Unit = { - val idleTime = Duration.between(lastHumanTurnTime, Instant.now()) - if (instanceState != STOPPED && idleTime > Duration.ofMinutes(idleTimeoutMinutes)) { - stopInstance() - } - } -} -``` - -### Integration Point - -Wire `onHumanTurn()` into Eagle's command processing. The cleanest place is where Eagle receives player commands from the Unity client - likely in `EagleServiceImpl` when handling the gRPC streaming commands. - ---- - -## Security: Eagle ↔ Shardok Authentication - -Since Eagle (DigitalOcean) and Shardok (Hetzner) are on different cloud providers, they cannot share a private network. We need secure authentication over the public internet. - -### Approach: TLS + Token Authentication - -This is the same pattern used by Stripe, GitHub, and most cloud APIs. - -**Layer 1: TLS Encryption (Let's Encrypt)** -- Server certificate from Let's Encrypt, auto-renewed via certbot -- All gRPC traffic encrypted with TLS 1.3 -- Prevents eavesdropping and MITM attacks - -**Layer 2: Token Authentication** -- 256-bit random shared secret in gRPC metadata -- Only requests with valid token are processed -- Easy to rotate without any certificate changes - -### How Token Auth Works - -**Setup (once):** -```bash -# Generate a 256-bit random token -openssl rand -hex 32 -# Output: a3f8b2c9d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1 - -# Store in GitHub Secrets as SHARDOK_AUTH_TOKEN -# Eagle reads from environment, Shardok receives via cloud-init -``` - -**Every Request:** -``` -Eagle Shardok -────── ─────── -gRPC call with metadata: → Validate token: - "authorization: Bearer a3f8b2c9..." if token != expected: - return UNAUTHENTICATED - else: - process request -``` - -### Server Certificate (Let's Encrypt) - -Shardok needs a domain name for Let's Encrypt. Options: - -1. **Dynamic DNS**: Assign `shardok.prod.eagle0.net` to instance IP on spin-up -2. **Elastic IP**: Reserve a Hetzner floating IP, always points to active instance - -Cloud-init runs certbot on first boot: - -```yaml -runcmd: - # Install certbot - - apt-get install -y certbot - - # Get certificate (DNS must already point to this IP) - - certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net - - # Certificate auto-renews via systemd timer (certbot installs this) - # Certs at: /etc/letsencrypt/live/shardok.prod.eagle0.net/ -``` - -### Implementation - -**Shardok gRPC Server (C++):** -```cpp -// Load Let's Encrypt certificates -std::string cert = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem"); -std::string key = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem"); - -grpc::SslServerCredentialsOptions ssl_opts; -ssl_opts.pem_key_cert_pairs.push_back({key, cert}); -auto creds = grpc::SslServerCredentials(ssl_opts); -builder.AddListeningPort("0.0.0.0:50051", creds); - -// Token validation - check on every RPC -Status PostCommand(ServerContext* context, const Request* req, Response* resp) { - auto auth = context->client_metadata().find("authorization"); - std::string expected = "Bearer " + auth_token_; - if (auth == context->client_metadata().end() || - auth->second != expected) { - return Status(grpc::UNAUTHENTICATED, "Invalid token"); - } - // ... handle request -} -``` - -**Eagle gRPC Client (Scala):** -```scala -// Standard TLS (trusts Let's Encrypt via system CA store) -val channel = NettyChannelBuilder - .forAddress("shardok.prod.eagle0.net", 50051) - .useTransportSecurity() // TLS with system trust store - .intercept(new ClientInterceptor { - override def interceptCall[Req, Resp]( - method: MethodDescriptor[Req, Resp], - callOptions: CallOptions, - next: Channel): ClientCall[Req, Resp] = { - new ForwardingClientCall.SimpleForwardingClientCall( - next.newCall(method, callOptions)) { - override def start(listener: Listener[Resp], headers: Metadata): Unit = { - headers.put( - Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER), - s"Bearer $authToken" - ) - super.start(listener, headers) - } - } - } - }) - .build() -``` - -### Token Rotation - -When rotating tokens (e.g., if compromised or as routine hygiene): - -1. Generate new token -2. Update Shardok to accept **both** old and new tokens temporarily -3. Update Eagle to use new token -4. Remove old token from Shardok - -No downtime, no certificate regeneration. - -### Firewall Rules - -```bash -# Hetzner firewall via cloud-init -ufw default deny incoming -ufw allow 80/tcp # Let's Encrypt HTTP-01 challenge (certbot) -ufw allow 50051/tcp # gRPC (protected by TLS + token) -ufw allow from /32 to any port 22 # SSH for debugging -ufw enable -``` - -### Security Summary - -| Threat | Mitigation | -|--------|------------| -| Eavesdropping | TLS 1.3 encryption | -| MITM attack | Let's Encrypt certificate validates server identity | -| Unauthorized access | 256-bit token required on every request | -| Token theft | TLS prevents sniffing; rotate if compromised | - -This is the industry-standard approach for API security. - ---- - -## Hetzner API Integration - -### API Authentication - -```bash -# Store in secrets manager -HETZNER_API_TOKEN= -``` - -### Key API Operations - -**Create Instance:** -```bash -curl -X POST "https://api.hetzner.cloud/v1/servers" \ - -H "Authorization: Bearer $HETZNER_API_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "shardok-prod", - "server_type": "cax41", - "location": "ash", - "image": "docker-ce", - "ssh_keys": ["eagle-deploy"], - "user_data": "", - "labels": {"service": "shardok", "env": "prod"} - }' -``` - -**Check Status:** -```bash -curl "https://api.hetzner.cloud/v1/servers/{id}" \ - -H "Authorization: Bearer $HETZNER_API_TOKEN" -# Response includes: status (running/starting/off), public IP -``` - -**Delete Instance (Stop Billing):** -```bash -curl -X DELETE "https://api.hetzner.cloud/v1/servers/{id}" \ - -H "Authorization: Bearer $HETZNER_API_TOKEN" -``` - -### Cloud-Init Script - -```yaml -#cloud-config -package_update: true -packages: - - docker.io - - certbot - -write_files: - - path: /etc/shardok/auth_token - permissions: '0600' - content: | - ${AUTH_TOKEN} - -runcmd: - # Get Let's Encrypt certificate (DNS must point to this IP first) - - certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net - - # Pull and run Shardok container - - echo ${DO_REGISTRY_TOKEN} | docker login registry.digitalocean.com -u ${DO_REGISTRY_TOKEN} --password-stdin - - docker pull registry.digitalocean.com/eagle0/shardok-server:arm64-latest - - docker run -d --name shardok \ - -p 50051:50051 \ - -v /etc/letsencrypt:/etc/letsencrypt:ro \ - -v /etc/shardok:/etc/shardok:ro \ - registry.digitalocean.com/eagle0/shardok-server:arm64-latest \ - --cert=/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem \ - --key=/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem \ - --token-file=/etc/shardok/auth_token -``` - ---- - -## Container Registry - -Using DigitalOcean Container Registry for all images (consistent with Eagle and other services). - -Both architectures use the same repository with different tag prefixes: -- ARM64 image: `registry.digitalocean.com/eagle0/shardok-server:arm64-latest` -- x86 image: `registry.digitalocean.com/eagle0/shardok-server:latest` - -CI automatically builds and pushes ARM64 images when C++ code changes. - ---- - -## Fallback Strategy - -### If Shardok Instance Not Ready - -When a battle starts but the Hetzner instance isn't ready: - -1. **Option A: Wait with Progress Indicator** - - Show "Preparing battle arena..." with spinner - - Most users won't notice 20-30s wait if UI is engaging - - Simplest implementation - -2. **Option B: Local Fallback** (Current behavior) - - Run Shardok on Eagle's droplet temporarily - - Lower AI quality but instant start - - Migrate battle to Hetzner when ready (complex) - -3. **Option C: Reduced AI Settings** - - Start battle immediately with very short time budgets - - Improve AI quality when Hetzner ready - - Seamless to user but requires AI hot-swap - -**Recommendation**: Start with Option A. If user feedback indicates wait time is problematic, implement Option B. - ---- - -## Monitoring & Observability - -### Metrics to Track - -| Metric | Purpose | -|--------|---------| -| `shardok_instance_state` | Current lifecycle state | -| `shardok_startup_duration_seconds` | Time from spin-up to ready | -| `shardok_battles_per_instance` | Efficiency of keep-alive | -| `shardok_wasted_spinups` | Spin-ups without battles | -| `shardok_hourly_cost` | Running cost tracking | - -### Alerting - -- Alert if startup takes > 90 seconds -- Alert if instance stuck in STARTING for > 3 minutes -- Alert if Hetzner API errors persist - ---- - -## Cost Estimate (Activity-Based, ARM) - -With activity-based spin-up, Shardok runs during active play sessions: - -| Play Pattern | Hours/Week | Compute/Month | -|--------------|------------|---------------| -| Casual (few sessions) | ~10 hrs | **~$1.75** | -| Regular (daily play) | ~20 hrs | **~$3.50** | -| Heavy (multiple daily) | ~40 hrs | **~$7.00** | - -Plus fixed costs: -- Container registry: $0 (GitHub Container Registry) -- Secrets management: $0 (GitHub Secrets) - -**Total estimated cost: $2-7/month** for typical usage. - -Compare to current DigitalOcean droplet upgrade (~$20-40/month for equivalent CPU power). - ---- - -## Implementation Phases - -### Phase 1: Basic On-Demand (1-2 weeks) - -1. ✅ Add ARM64 Linux to CI build matrix (PR #4990, merged 2026-01-02) - - Added LLVM toolchain for ARM64 cross-compilation - - Created ARM64 sysroot build workflow - - Added Ubuntu 24.04 ARM64 base image for containers -2. ✅ Push ARM64 container to DigitalOcean Container Registry (PR #4990) - - Created shardok_arm64_build.yml workflow - - Pushes to registry.digitalocean.com/eagle0/shardok-server:arm64-latest -3. ✅ Implement Hetzner API client in Eagle (Scala) (PR #4996, merged 2026-01-02) - - HetznerApiClient with create/delete/list/power operations - - Async HTTP via OkHttp with Future-based API -4. ✅ Add ShardokInstanceManager with spin-up/shutdown logic (PR #4998, merged 2026-01-02) - - Activity-based spin-up when players connect - - Idle timeout shutdown (default 60 minutes) - - State machine: Stopped → Starting → Ready → InUse → Stopping -5. ✅ Add TLS + token authentication to Shardok server (PR #5001, merged 2026-01-02) - - C++ server: TLS via Let's Encrypt, Bearer token validation - - Scala client: TLS and auth token support in ServerSetupHelpers -6. 🔄 Test end-to-end with manual triggers ← **CURRENT** - - ✅ Hetzner server created (CAX41 ARM64 in Helsinki) - - ✅ Floating IPv6 configured and persisted via netplan - - ✅ DNS configured (shardok.prod.eagle0.net → floating IP) - - ✅ Let's Encrypt certificate obtained - - ✅ NAT64 configured for IPv6-only server to reach IPv4 registries - - ✅ Fixed runfiles error (missing SHARDOK_RESOURCES_PATH env var in docker run) - - 🔄 Re-test container startup on Hetzner - -### Known Issues - -**IPv6-Only Servers Need NAT64** - -Hetzner ARM instances are IPv6-only by default. DigitalOcean Container Registry's blob storage is IPv4-only. Solution: configure NAT64 DNS: - -```bash -mkdir -p /etc/systemd/resolved.conf.d -cat > /etc/systemd/resolved.conf.d/nat64.conf << 'EOF' -[Resolve] -DNS=2a00:1098:2b::1 2a00:1098:2c::1 -EOF -systemctl restart systemd-resolved -``` - -**Floating IP Requires Manual Config** - -Hetzner floating IPs must be added to netplan: - -```yaml -# In /etc/netplan/50-cloud-init.yaml, add to addresses list: -addresses: -- "2a01:4f9:xxxx:xxxx::1/64" # Server's own IP -- "2a01:4f9:yyyy:yyyy::1/64" # Floating IP -``` - -Then run `netplan apply`. - -### Phase 2: Wire into Eagle - -1. Call `onHumanTurn()` when human players submit commands -2. Wire ShardokInstanceManager into Eagle's startup -3. Add environment variables for Hetzner API token, floating IP -4. Add "Preparing battle..." UI state for when Shardok is starting - -### Phase 3: Monitoring & Polish - -1. Add metrics and dashboards -2. Implement alerting -3. Cost tracking -4. Documentation and runbooks - ---- - -## Design Decisions - -### Certificate Management - -**Fully automated via Let's Encrypt + certbot.** - -- Certbot runs on instance startup to obtain certificate -- Systemd timer auto-renews every 60 days (certs valid 90 days) -- No manual intervention required -- Token rotation is manual but simple (generate new token, update both sides) - -### Concurrent Battles - -**Decision**: Single shared Shardok instance handles all concurrent battles. - -Shardok already supports multiple simultaneous games. With 16 cores and activity-based spin-up, one instance is sufficient for the foreseeable future. - ---- - -## Future Enhancements - -These are not needed now but documented for future consideration: - -### Multi-Region Deployment - -If player base expands globally: -- Deploy Shardok instances in multiple Hetzner regions (Ashburn, Helsinki, Singapore) -- Route players to nearest region based on their Eagle connection -- Each region independent (no cross-region state sharing needed) - -**Trigger**: Consistent complaints about battle latency from non-US players. - -### Dynamic Instance Sizing - -Could adjust instance size based on battle complexity: -- Small battles (< 10 units): CAX21 (4 cores) -- Medium battles: CAX31 (8 cores) -- Large battles: CAX41 (16 cores) - -**Current decision**: Start with CAX41 for all battles. The cost difference is minimal (~$0.02/hr between sizes) and simplicity is valuable. - -### Dedicated Shardok Instances Per Game - -For competitive/tournament play, could spin up dedicated instances: -- Guaranteed resources, no contention -- Isolated failures -- Higher cost - -**Current decision**: Shared instance is fine. Revisit if multiplayer competitive mode launches.