eagle0/docs/SHARDOK_ON_DEMAND_COMPUTE.md

Shardok On-Demand Compute Infrastructure Analysis

Overview

This document analyzes options for running Shardok (the tactical combat AI server) on powerful on-demand compute, separate from the low-power droplet that hosts Eagle.

Current State: Shardok runs on the same DigitalOcean droplet as Eagle, limiting AI performance.

Goal: Spin up powerful compute only when battles occur, keeping costs low while improving AI quality.

---

Cloud Provider Pricing Comparison

High-Performance CPU Options (8-16 cores)

Provider	Instance Type	vCPUs	RAM	Hourly Cost	Monthly Est.*
Hetzner	CCX23	8	32GB	€0.078 (~$0.085)	~$6-8
Hetzner	CCX33	16	64GB	€0.155 (~$0.17)	~$12-16
Vultr	High Frequency 8-core	8	32GB	$0.238	~$20-25
Vultr	High Frequency 16-core	16	64GB	$0.476	~$40-50
DigitalOcean	Premium CPU 8-core	8	32GB	$0.250	~$22-28
DigitalOcean	Premium CPU 16-core	16	64GB	$0.500	~$44-55

*Monthly estimates assume ~4 hours/week of battle usage (~17 hours/month)

Key Finding: Hetzner is 3-5x Cheaper

Hetzner's dedicated vCPU instances offer the best price-performance ratio by a significant margin. For equivalent 16-core compute:

• Hetzner: ~$0.17/hr
• DigitalOcean: ~$0.50/hr (2.9x more expensive)
• Vultr: ~$0.48/hr (2.8x more expensive)

---

Instance Startup Time Analysis

Current Startup Times by Provider

Provider	Cold Start	Warm Image	Container Registry
Hetzner	30-60s	20-40s	Native support
Vultr	30-45s	20-30s	Native support
DigitalOcean	30-60s	25-45s	Already in use

10-Second Max Wait Time Constraint

A 10-second startup constraint eliminates traditional VM spin-up as an option. Alternatives:

Option 1: Pre-warmed Instance Pool (Recommended)

• Keep one small instance running 24/7 as a "hot standby"
• Cost: ~$5-10/month for a minimal Hetzner instance
• Startup time: < 1 second (already running)
• Upgrade on demand when battle starts (resize takes 2-3 min, but battle can start immediately on small instance)

Option 2: Predictive Spin-up

• Start spinning up when user enters battle setup
• Battle setup typically takes 30-60 seconds (map selection, army placement)
• Could achieve "instant" availability if spin-up completes during setup
• Risk: User might cancel, wasting spin-up cost (minimal at ~$0.01-0.02)

Option 3: Container-Based Quick Start

• Use pre-pulled Docker images on a warm container host
• Providers like Fly.io or Railway can start containers in 1-5 seconds
• Trade-off: Container platforms typically cost more than raw VMs

Option 4: Serverless/Function Compute

• Not viable for Shardok due to:
  • Long-running connections (battles last minutes)
  • High memory requirements
  • Stateful game state

Recommendation for 10s Constraint

Hybrid approach: Keep a minimal "always-on" instance ($5-10/month) that can handle battles at reduced AI quality. When a battle is anticipated (entering battle setup), start spinning up a powerful instance. If the powerful instance is ready when the battle starts, use it; otherwise, fall back to the always-on instance.

---

ARM Options Outside AWS

Available ARM Compute

Provider	ARM Offering	vCPUs	RAM	Hourly Cost	Notes
Hetzner	CAX21	4	8GB	€0.009	Ampere Altra
Hetzner	CAX31	8	16GB	€0.018	Ampere Altra
Hetzner	CAX41	16	32GB	€0.035	Ampere Altra
Oracle Cloud	A1 Flex	4	24GB	Free tier	Always Free eligible
Oracle Cloud	A1 Flex	24	144GB	~$0.10	Ampere Altra
Scaleway	AMP2-C8	8	16GB	€0.104	Ampere Altra

Key Finding: Hetzner ARM is Extremely Cheap

Hetzner's ARM instances are 4-5x cheaper than their x86 equivalents:

• CAX41 (16 ARM cores): €0.035/hr (~$0.04)
• CCX33 (16 x86 cores): €0.155/hr (~$0.17)

ARM Considerations for Shardok

Pros:

• Significant cost savings (4-5x)
• Good single-threaded performance on Ampere Altra
• Native ARM builds already supported (macOS development)

Cons:

• Requires ARM64 Linux build (currently only building x86_64 Linux)
• Some performance tuning may be needed
• Less mature ecosystem than x86

Build Effort: Medium. Bazel supports cross-compilation, and the codebase is portable C++. Main work is adding ARM64 Linux to the build matrix.

---

Hetzner Geographic Presence

Current Locations

Hetzner operates datacenters in:

• Germany: Nuremberg (NBG1), Falkenstein (FSN1)
• Finland: Helsinki (HEL1)
• USA: Ashburn, Virginia (ASH) - opened 2023
• Singapore: (coming 2024/2025)

Latency Implications

For users primarily in North America:

• Hetzner US (Ashburn): 20-40ms to East Coast, 60-80ms to West Coast
• Hetzner EU: 80-120ms from US East Coast, 140-180ms from US West Coast

Since Shardok handles AI computation (not real-time player input), latency is less critical. The Eagle-Shardok communication adds some overhead, but AI "thinking time" dominates.

Recommendation

Use Hetzner US (Ashburn) for North American users. The US datacenter has the same pricing as EU datacenters and provides better latency.

---

Implementation Options Summary

Option A: Simple On-Demand (30-60s startup)

• Spin up Hetzner CCX33 when battle starts
• Cost: $0.17/hr × 4 hrs/week = **$3/month**
• Latency: 30-60s wait at battle start
• Complexity: Low

Option B: Predictive Spin-up (0-30s startup)

• Start spinning up during battle setup phase
• Cost: Same as Option A, plus occasional wasted spin-ups (~$1/month)
• Latency: 0-30s depending on setup duration
• Complexity: Medium

Option C: Hybrid Always-On + On-Demand (< 1s startup)

• Keep minimal instance always running ($5-10/month)
• Spin up powerful instance during battle setup
• Use powerful if ready, fall back to minimal
• Cost: ~$8-15/month
• Latency: < 1s (always-on fallback)
• Complexity: Medium-High

Option D: ARM Compute (Cheapest)

• Build ARM64 Linux target
• Use Hetzner CAX41 (16 ARM cores) at €0.035/hr
• Cost: ~$0.60/month for compute (+ always-on if needed)
• Latency: Same as x86 options
• Complexity: Medium (build system changes)

---

Recommendation

Phase 1 (Quick Win): Implement Option B (Predictive Spin-up) with Hetzner CCX33 in Ashburn.

• Minimal code changes (add spin-up trigger in Eagle when battle setup begins)
• Good user experience (usually ready by battle start)
• Low cost (~$3-5/month)

Phase 2 (Future): Add ARM64 Linux build and switch to ARM instances.

• 4-5x cost reduction
• Same or better performance
• Requires build system work

Phase 3 (If Needed): Add always-on fallback for guaranteed instant start.

• Only if Phase 1 startup times prove problematic
• Adds ~$5-10/month fixed cost

---

Implementation Plan

Chosen Architecture

• Provider: Hetzner Cloud
• Instance: CAX41 (16 ARM cores, 32GB RAM)
• Location: Ashburn, Virginia (ash)
• Cost: €0.035/hr (~$0.04/hr)
• Strategy: On-demand with predictive spin-up and idle keep-alive

---

System Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        DigitalOcean                                  │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │                     Eagle Server                             │    │
│  │  ┌─────────────────┐  ┌─────────────────┐                   │    │
│  │  │ ShardokManager  │  │ ActivityTracker │                   │    │
│  │  │ - spin up/down  │  │ - player active │                   │    │
│  │  │ - health check  │  │ - trigger start │                   │    │
│  │  └────────┬────────┘  └────────┬────────┘                   │    │
│  │           │                    │                             │    │
│  │           └────────┬───────────┘                             │    │
│  │                    ▼                                         │    │
│  │           ┌────────────────┐                                 │    │
│  │           │ ShardokClient  │◄─── TLS + Token Auth            │    │
│  │           └────────┬───────┘                                 │    │
│  └────────────────────┼─────────────────────────────────────────┘    │
└───────────────────────┼──────────────────────────────────────────────┘
                        │ gRPC over TLS (Let's Encrypt)
                        ▼
┌───────────────────────────────────────────────────────────────────────┐
│                         Hetzner Cloud (Ashburn)                       │
│  ┌─────────────────────────────────────────────────────────────────┐ │
│  │                    CAX41 (On-Demand)                            │ │
│  │  ┌─────────────────────────────────────────────────────────┐   │ │
│  │  │                  Shardok Server                          │   │ │
│  │  │  - 16 ARM cores for AI computation                       │   │ │
│  │  │  - TLS cert from Let's Encrypt (auto-renewed)            │   │ │
│  │  │  - Validates auth token on each request                  │   │ │
│  │  └─────────────────────────────────────────────────────────┘   │ │
│  └─────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────┘

---

Instance Lifecycle Management

State Machine

                    ┌──────────────┐
                    │   STOPPED    │◄────────────────────┐
                    └──────┬───────┘                     │
                           │ human takes turn            │ idle timeout
                           ▼                             │ (10 minutes)
                    ┌──────────────┐                     │
                    │  STARTING    │                     │
                    └──────┬───────┘                     │
                           │ health check passes         │
                           ▼                             │
                    ┌──────────────┐                     │
         ┌─────────►│    READY     │─────────────────────┤
         │          └──────┬───────┘                     │
         │                 │ battle starts               │
         │                 ▼                             │
         │          ┌──────────────┐                     │
         │          │   IN_USE     │                     │
         │          └──────┬───────┘                     │
         │                 │ battle ends                 │
         └─────────────────┘

Note: The IDLE state was removed - we now simply track time since last human turn and shut down after 10 minutes of inactivity.

Lifecycle Rules

1. Spin-up Trigger: Human player takes a turn

   • Any command submitted by a human player to Eagle
   • Simple and reliable - no prediction needed

2. Keep-Alive: Reset timer on each human turn

   • Covers thinking time between turns
   • Natural pauses (snacks, calls) under 10 min don't cause shutdown

3. Shutdown: 10 minutes since last human turn

   • Hetzner API: DELETE /servers/{id} to stop billing
   • Short timeout keeps costs low during idle periods

4. Health Checks: Every 10 seconds while STARTING

   • gRPC connectivity check to Shardok
   • Mark READY only when Shardok responds successfully

---

Spin-up Strategy

Simple Approach: Human Turn-Based

Spin up Shardok whenever a human player takes a turn. Shut down after 10 minutes of inactivity.

This is simpler than trying to predict battles. Battles happen regularly during active play, so keeping Shardok running while humans are playing ensures it's ready when needed.

Spin-up trigger:

• Any human player takes a turn (submits commands to Eagle)

Shutdown trigger:

• No human player has taken a turn in the last 10 minutes

Why this works:

• Battles are common during active play sessions
• 10-minute idle timeout is long enough to cover thinking time between turns
• Cost is so low (~$0.04/hr) that running during entire play sessions is fine
• No complex prediction logic needed

Cost Impact

Scenario	Hours/Week	Cost/Month
Human turn-based (10 min idle)	~15-20 hrs	~$2.50-3.50
Always-on 24/7	168 hrs	~$29

Human turn-based is simple and cheap. The 10-minute timeout catches natural breaks (getting a snack, answering a call) while shutting down during longer idle periods.

Implementation in Eagle

class ShardokInstanceManager {
  private var instanceState: InstanceState = STOPPED
  private var lastHumanTurnTime: Instant = Instant.MIN
  private val idleTimeoutMinutes = 10

  // Called when any human player submits a turn
  def onHumanTurn(): Unit = {
    lastHumanTurnTime = Instant.now()
    if (instanceState == STOPPED) {
      startInstance()
    }
  }

  // Run every minute
  def periodicShutdownCheck(): Unit = {
    val idleTime = Duration.between(lastHumanTurnTime, Instant.now())
    if (instanceState != STOPPED && idleTime > Duration.ofMinutes(idleTimeoutMinutes)) {
      stopInstance()
    }
  }
}

Integration Point

Wire onHumanTurn() into Eagle's command processing. The cleanest place is where Eagle receives player commands from the Unity client - likely in EagleServiceImpl when handling the gRPC streaming commands.

---

Security: Eagle ↔ Shardok Authentication

Since Eagle (DigitalOcean) and Shardok (Hetzner) are on different cloud providers, they cannot share a private network. We need secure authentication over the public internet.

Approach: TLS + Token Authentication

This is the same pattern used by Stripe, GitHub, and most cloud APIs.

Layer 1: TLS Encryption (Let's Encrypt)

• Server certificate from Let's Encrypt, auto-renewed via certbot
• All gRPC traffic encrypted with TLS 1.3
• Prevents eavesdropping and MITM attacks

Layer 2: Token Authentication

• 256-bit random shared secret in gRPC metadata
• Only requests with valid token are processed
• Easy to rotate without any certificate changes

How Token Auth Works

Setup (once):

# Generate a 256-bit random token
openssl rand -hex 32
# Output: a3f8b2c9d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1

# Store in GitHub Secrets as SHARDOK_AUTH_TOKEN
# Eagle reads from environment, Shardok receives via cloud-init

Every Request:

Eagle                                              Shardok
──────                                             ───────
gRPC call with metadata:                      →    Validate token:
  "authorization: Bearer a3f8b2c9..."              if token != expected:
                                                     return UNAUTHENTICATED
                                                   else:
                                                     process request

Server Certificate (Let's Encrypt)

Shardok needs a domain name for Let's Encrypt. Options:

1. Dynamic DNS: Assign shardok.prod.eagle0.net to instance IP on spin-up
2. Elastic IP: Reserve a Hetzner floating IP, always points to active instance

Cloud-init runs certbot on first boot:

runcmd:
  # Install certbot
  - apt-get install -y certbot

  # Get certificate (DNS must already point to this IP)
  - certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net

  # Certificate auto-renews via systemd timer (certbot installs this)
  # Certs at: /etc/letsencrypt/live/shardok.prod.eagle0.net/

Implementation

Shardok gRPC Server (C++):

// Load Let's Encrypt certificates
std::string cert = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem");
std::string key = ReadFile("/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem");

grpc::SslServerCredentialsOptions ssl_opts;
ssl_opts.pem_key_cert_pairs.push_back({key, cert});
auto creds = grpc::SslServerCredentials(ssl_opts);
builder.AddListeningPort("0.0.0.0:50051", creds);

// Token validation - check on every RPC
Status PostCommand(ServerContext* context, const Request* req, Response* resp) {
    auto auth = context->client_metadata().find("authorization");
    std::string expected = "Bearer " + auth_token_;
    if (auth == context->client_metadata().end() ||
        auth->second != expected) {
        return Status(grpc::UNAUTHENTICATED, "Invalid token");
    }
    // ... handle request
}

Eagle gRPC Client (Scala):

// Standard TLS (trusts Let's Encrypt via system CA store)
val channel = NettyChannelBuilder
  .forAddress("shardok.prod.eagle0.net", 50051)
  .useTransportSecurity()  // TLS with system trust store
  .intercept(new ClientInterceptor {
    override def interceptCall[Req, Resp](
        method: MethodDescriptor[Req, Resp],
        callOptions: CallOptions,
        next: Channel): ClientCall[Req, Resp] = {
      new ForwardingClientCall.SimpleForwardingClientCall(
          next.newCall(method, callOptions)) {
        override def start(listener: Listener[Resp], headers: Metadata): Unit = {
          headers.put(
            Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER),
            s"Bearer $authToken"
          )
          super.start(listener, headers)
        }
      }
    }
  })
  .build()

Token Rotation

When rotating tokens (e.g., if compromised or as routine hygiene):

1. Generate new token
2. Update Shardok to accept both old and new tokens temporarily
3. Update Eagle to use new token
4. Remove old token from Shardok

No downtime, no certificate regeneration.

Firewall Rules

# Hetzner firewall via cloud-init
ufw default deny incoming
ufw allow 80/tcp    # Let's Encrypt HTTP-01 challenge (certbot)
ufw allow 50051/tcp # gRPC (protected by TLS + token)
ufw allow from <admin-ip>/32 to any port 22  # SSH for debugging
ufw enable

Security Summary

Threat	Mitigation
Eavesdropping	TLS 1.3 encryption
MITM attack	Let's Encrypt certificate validates server identity
Unauthorized access	256-bit token required on every request
Token theft	TLS prevents sniffing; rotate if compromised

This is the industry-standard approach for API security.

---

Hetzner API Integration

API Authentication

# Store in secrets manager
HETZNER_API_TOKEN=<token from Hetzner Cloud Console>

Key API Operations

Create Instance:

curl -X POST "https://api.hetzner.cloud/v1/servers" \
  -H "Authorization: Bearer $HETZNER_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "shardok-prod",
    "server_type": "cax41",
    "location": "ash",
    "image": "docker-ce",
    "ssh_keys": ["eagle-deploy"],
    "user_data": "<cloud-init script>",
    "labels": {"service": "shardok", "env": "prod"}
  }'

Check Status:

curl "https://api.hetzner.cloud/v1/servers/{id}" \
  -H "Authorization: Bearer $HETZNER_API_TOKEN"
# Response includes: status (running/starting/off), public IP

Delete Instance (Stop Billing):

curl -X DELETE "https://api.hetzner.cloud/v1/servers/{id}" \
  -H "Authorization: Bearer $HETZNER_API_TOKEN"

Cloud-Init Script

#cloud-config
package_update: true
packages:
  - docker.io
  - certbot

write_files:
  - path: /etc/shardok/auth_token
    permissions: '0600'
    content: |
      ${AUTH_TOKEN}

runcmd:
  # Get Let's Encrypt certificate (DNS must point to this IP first)
  - certbot certonly --standalone -d shardok.prod.eagle0.net --non-interactive --agree-tos -m admin@eagle0.net

  # Pull and run Shardok container
  - echo ${DO_REGISTRY_TOKEN} | docker login registry.digitalocean.com -u ${DO_REGISTRY_TOKEN} --password-stdin
  - docker pull registry.digitalocean.com/eagle0/shardok-server:arm64-latest
  - docker run -d --name shardok \
      -p 50051:50051 \
      -v /etc/letsencrypt:/etc/letsencrypt:ro \
      -v /etc/shardok:/etc/shardok:ro \
      registry.digitalocean.com/eagle0/shardok-server:arm64-latest \
      --cert=/etc/letsencrypt/live/shardok.prod.eagle0.net/fullchain.pem \
      --key=/etc/letsencrypt/live/shardok.prod.eagle0.net/privkey.pem \
      --token-file=/etc/shardok/auth_token

---

Container Registry

Using DigitalOcean Container Registry for all images (consistent with Eagle and other services).

Both architectures use the same repository with different tag prefixes:

• ARM64 image: registry.digitalocean.com/eagle0/shardok-server:arm64-latest
• x86 image: registry.digitalocean.com/eagle0/shardok-server:latest

CI automatically builds and pushes ARM64 images when C++ code changes.

---

Fallback Strategy

If Shardok Instance Not Ready

When a battle starts but the Hetzner instance isn't ready:

1. Option A: Wait with Progress Indicator

   • Show "Preparing battle arena..." with spinner
   • Most users won't notice 20-30s wait if UI is engaging
   • Simplest implementation

2. Option B: Local Fallback (Current behavior)

   • Run Shardok on Eagle's droplet temporarily
   • Lower AI quality but instant start
   • Migrate battle to Hetzner when ready (complex)

3. Option C: Reduced AI Settings

   • Start battle immediately with very short time budgets
   • Improve AI quality when Hetzner ready
   • Seamless to user but requires AI hot-swap

Recommendation: Start with Option A. If user feedback indicates wait time is problematic, implement Option B.

---

Monitoring & Observability

Metrics to Track

Metric	Purpose
shardok_instance_state	Current lifecycle state
shardok_startup_duration_seconds	Time from spin-up to ready
shardok_battles_per_instance	Efficiency of keep-alive
shardok_wasted_spinups	Spin-ups without battles
shardok_hourly_cost	Running cost tracking

Alerting

• Alert if startup takes > 90 seconds
• Alert if instance stuck in STARTING for > 3 minutes
• Alert if Hetzner API errors persist

---

Cost Estimate (Activity-Based, ARM)

With activity-based spin-up, Shardok runs during active play sessions:

Play Pattern	Hours/Week	Compute/Month
Casual (few sessions)	~10 hrs	~$1.75
Regular (daily play)	~20 hrs	~$3.50
Heavy (multiple daily)	~40 hrs	~$7.00

Plus fixed costs:

• Container registry: $0 (GitHub Container Registry)
• Secrets management: $0 (GitHub Secrets)

Total estimated cost: $2-7/month for typical usage.

Compare to current DigitalOcean droplet upgrade (~$20-40/month for equivalent CPU power).

---

Implementation Phases

Phase 1: Basic On-Demand (1-2 weeks)

1. ✅ Add ARM64 Linux to CI build matrix (PR #4990, merged 2026-01-02)
   • Added LLVM toolchain for ARM64 cross-compilation
   • Created ARM64 sysroot build workflow
   • Added Ubuntu 24.04 ARM64 base image for containers
2. ✅ Push ARM64 container to DigitalOcean Container Registry (PR #4990)
   • Created shardok_arm64_build.yml workflow
   • Pushes to registry.digitalocean.com/eagle0/shardok-server:arm64-latest
3. ✅ Implement Hetzner API client in Eagle (Scala) (PR #4996, merged 2026-01-02)
   • HetznerApiClient with create/delete/list/power operations
   • Async HTTP via OkHttp with Future-based API
4. ✅ Add ShardokInstanceManager with spin-up/shutdown logic (PR #4998, merged 2026-01-02)
   • Activity-based spin-up when players connect
   • Idle timeout shutdown (default 60 minutes)
   • State machine: Stopped → Starting → Ready → InUse → Stopping
5. ✅ Add TLS + token authentication to Shardok server (PR #5001, merged 2026-01-02)
   • C++ server: TLS via Let's Encrypt, Bearer token validation
   • Scala client: TLS and auth token support in ServerSetupHelpers
6. 🔄 Test end-to-end with manual triggers ← CURRENT
   • ✅ Hetzner server created (CAX41 ARM64 in Helsinki)
   • ✅ Floating IPv6 configured and persisted via netplan
   • ✅ DNS configured (shardok.prod.eagle0.net → floating IP)
   • ✅ Let's Encrypt certificate obtained
   • ✅ NAT64 configured for IPv6-only server to reach IPv4 registries
   • ✅ Fixed runfiles error (missing SHARDOK_RESOURCES_PATH env var in docker run)
   • 🔄 Re-test container startup on Hetzner

Known Issues

IPv6-Only Servers Need NAT64

Hetzner ARM instances are IPv6-only by default. DigitalOcean Container Registry's blob storage is IPv4-only. Solution: configure NAT64 DNS:

mkdir -p /etc/systemd/resolved.conf.d
cat > /etc/systemd/resolved.conf.d/nat64.conf << 'EOF'
[Resolve]
DNS=2a00:1098:2b::1 2a00:1098:2c::1
EOF
systemctl restart systemd-resolved

Floating IP Requires Manual Config

Hetzner floating IPs must be added to netplan:

# In /etc/netplan/50-cloud-init.yaml, add to addresses list:
addresses:
- "2a01:4f9:xxxx:xxxx::1/64"  # Server's own IP
- "2a01:4f9:yyyy:yyyy::1/64"  # Floating IP

Then run netplan apply.

Phase 2: Wire into Eagle

1. Call onHumanTurn() when human players submit commands
2. Wire ShardokInstanceManager into Eagle's startup
3. Add environment variables for Hetzner API token, floating IP
4. Add "Preparing battle..." UI state for when Shardok is starting

Phase 3: Monitoring & Polish

1. Add metrics and dashboards
2. Implement alerting
3. Cost tracking
4. Documentation and runbooks

---

Design Decisions

Certificate Management

Fully automated via Let's Encrypt + certbot.

• Certbot runs on instance startup to obtain certificate
• Systemd timer auto-renews every 60 days (certs valid 90 days)
• No manual intervention required
• Token rotation is manual but simple (generate new token, update both sides)

Concurrent Battles

Decision: Single shared Shardok instance handles all concurrent battles.

Shardok already supports multiple simultaneous games. With 16 cores and activity-based spin-up, one instance is sufficient for the foreseeable future.

---

Future Enhancements

These are not needed now but documented for future consideration:

Multi-Region Deployment

If player base expands globally:

• Deploy Shardok instances in multiple Hetzner regions (Ashburn, Helsinki, Singapore)
• Route players to nearest region based on their Eagle connection
• Each region independent (no cross-region state sharing needed)

Trigger: Consistent complaints about battle latency from non-US players.

Dynamic Instance Sizing

Could adjust instance size based on battle complexity:

• Small battles (< 10 units): CAX21 (4 cores)
• Medium battles: CAX31 (8 cores)
• Large battles: CAX41 (16 cores)

Current decision: Start with CAX41 for all battles. The cost difference is minimal (~$0.02/hr between sizes) and simplicity is valuable.

Dedicated Shardok Instances Per Game

For competitive/tournament play, could spin up dedicated instances:

• Guaranteed resources, no contention
• Isolated failures
• Higher cost

Current decision: Shared instance is fine. Revisit if multiplayer competitive mode launches.