admin/eagle0
1
0
Fork 0
mirror of https://github.com/nolen777/eagle0.git synced 2026-07-28 21:55:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
c0920026776e5b269ae2fa360396ee32538b9041
eagle0/docs/PRODUCTIONIZATION.md
T

28 KiB
Raw Blame History

Eagle0 Productionization Plan

Executive Summary

This document outlines a plan to move Eagle0's Eagle and Shardok servers from a home Mac to cloud infrastructure while maintaining a QA environment on the Mac. The architecture uses DigitalOcean (already integrated via Spaces) with containerized deployments and GitHub Actions CI/CD.

Current Architecture

Unity Client
     │
     │ gRPC/TLS (eagle0.net:443)
     ▼
nginx (home Mac, via router port forward)
     │
     ├─► Eagle Server (Scala/JVM, port 40032)
     │        │
     │        │ internal gRPC (port 40042)
     │        ▼
     └─► Shardok Server (C++, port 40042/40052)

Storage: DigitalOcean Spaces (sfo3.digitaloceanspaces.com)
DNS: eagle0.net → home IP

Key observations:

  • Already using DigitalOcean Spaces for S3-compatible storage
  • Eagle server is a deployable JAR (eagle_server_deploy.jar)
  • Shardok server is a native C++ binary
  • nginx handles TLS termination and gRPC routing
  • Self-hosted GitHub Actions runner on Mac

Target Architecture

Production Environment (DigitalOcean)

Unity Client
     │
     ├─► eagle0.net (Production)
     │        │
     │        ▼
     │   DigitalOcean Load Balancer (TLS termination)
     │        │
     │        ▼
     │   ┌─────────────────────────────────────┐
     │   │     DigitalOcean Droplet(s)         │
     │   │  ┌─────────────┬─────────────────┐  │
     │   │  │   Eagle     │    Shardok      │  │
     │   │  │   (Docker)  │    (Docker)     │  │
     │   │  │   :40032    │    :40042       │  │
     │   │  └─────────────┴─────────────────┘  │
     │   └─────────────────────────────────────┘
     │
     └─► qa.eagle0.net (QA - home Mac, unchanged)
              │
              ▼
         nginx → Eagle/Shardok (current setup)

Environment Switching

The Unity client already supports configurable server URLs via the connection screen:

  • Production: eagle0.net (default)
  • QA: qa.eagle0.net

No client code changes needed - users can simply type the desired URL.

Cloud Provider: DigitalOcean

Why DigitalOcean

  1. Already integrated - S3 Spaces storage at sfo3.digitaloceanspaces.com with credentials configured
  2. Simple pricing - Predictable monthly costs, no surprise bills
  3. Good performance - SFO3 datacenter is geographically close
  4. Managed services - Load balancers, managed databases if needed later
  5. Not AWS/GCP - Per your requirements

Alternative Considered: Hetzner

  • Cheaper for compute ($3.29/mo for 2 vCPU/4GB vs DO's $24/mo)
  • Good EU presence but less US coverage
  • No managed load balancer (need to run HAProxy/nginx yourself)
  • Recommendation: Start with DigitalOcean for simplicity; migrate to Hetzner later if cost becomes a concern

Infrastructure Components

1. Compute: DigitalOcean Droplets

Resource Characteristics

  • Eagle server: CPU-light, possibly RAM-heavy (JVM). Should be always available.
  • Shardok server: Very CPU-heavy (AI algorithms). Only needed during tactical battles. Startup time <1 second.

Recommended: On-Demand Shardok (Same Droplet)

For current low player count, optimize for cost while maintaining availability:

Component Config Monthly Cost
Droplet s-2vcpu-4gb $24/mo
Eagle Always running -
Shardok Started on-demand by Eagle, stopped after idle -

How it works:

  1. Eagle server runs 24/7 - game is always available
  2. When battle starts, Eagle launches Shardok container/process (<1s startup)
  3. After battle ends, idle timer starts (e.g., 5 minutes)
  4. On timeout, Eagle stops Shardok to free CPU
  5. If new battle starts during idle period, Shardok is already warm

Shardok lifecycle management (implement in Eagle):

// Pseudocode for Eagle's Shardok management
object ShardokManager {
  private var process: Option[Process] = None
  private var idleTimer: Option[Timer] = None

  def ensureRunning(): Unit = {
    cancelIdleTimer()
    if (process.isEmpty) {
      process = Some(startShardokProcess())
      waitForHealthCheck()
    }
  }

  def onBattleEnd(): Unit = {
    idleTimer = Some(scheduleShutdown(5.minutes))
  }

  private def shutdown(): Unit = {
    process.foreach(_.destroy())
    process = None
  }
}

Future Scaling Options

As player count grows and Shardok needs more CPU:

Option A: Upgrade Droplet (Simplest)

Droplet vCPU RAM Cost Use Case
s-2vcpu-4gb 2 shared 4 GB $24/mo Current (few players)
s-4vcpu-8gb 4 shared 8 GB $48/mo Moderate usage
c-4 4 dedicated 8 GB $84/mo CPU-intensive battles
c-8 8 dedicated 16 GB $168/mo Multiple concurrent battles

Option B: Separate Shardok Droplet (API-driven)

For heavy Shardok usage with cost optimization:

  • Eagle droplet: s-1vcpu-2gb ($12/mo) - always on
  • Shardok droplet: Created on-demand via DigitalOcean API
    • c-4 CPU-optimized: $0.125/hour
    • c-8 CPU-optimized: $0.25/hour
    • Created when battle starts, destroyed after idle
    • 30-60s droplet creation time (acceptable if battles are requested in advance)

Option C: Fly.io for Shardok (Scale-to-Zero)

For true pay-per-use with fast cold starts:

  • Eagle: DigitalOcean or Fly.io (~$10/mo)
  • Shardok: Fly.io with auto-scaling
    • Performance VMs: ~$0.0000022/second when running
    • Cold start: 2-5 seconds
    • Scales to zero when idle

Requires learning Fly.io but offers best cost efficiency for sporadic usage.

Option D: Multiple Shardok Instances (High Scale)

For many concurrent battles:

  • Eagle: Dedicated droplet with more RAM
  • Shardok pool: Multiple Shardok containers/droplets
  • Eagle routes battles to available Shardok instances
  • Could use Kubernetes or Docker Swarm for orchestration

This is overkill for now but documented for future reference.

2. Load Balancer

DigitalOcean Load Balancer: $12/mo

  • TLS termination with Let's Encrypt
  • Health checks
  • Sticky sessions (if needed)
  • Can add more droplets later for HA

Alternative: Run nginx on the droplet for $0 extra, but lose automatic failover.

3. Storage

Already configured: DigitalOcean Spaces

  • Bucket: eagle0
  • Region: sfo3
  • Used for game saves and assets
  • Cost: $5/mo base + $0.02/GB storage + $0.01/GB transfer

4. DNS

Option A: DigitalOcean DNS (Recommended)

  • Free with droplets
  • Easy integration
  • API for automated updates

Option B: Keep current DNS provider

  • Update A records manually or via script

DNS Records:

eagle0.net          A     <DO Load Balancer IP>
qa.eagle0.net       A     <Home IP> (unchanged)
*.eagle0.net        A     <DO Load Balancer IP> (wildcard for future)

5. Firewall

DigitalOcean Cloud Firewall: Free

Inbound Rules:
- TCP 443 (HTTPS/gRPC) from anywhere → Load Balancer
- TCP 22 (SSH) from your IP only → Droplets
- TCP 40032 (Eagle) from Load Balancer only
- TCP 40042 (Shardok) from Load Balancer only

Outbound Rules:
- All traffic allowed (for external APIs, Spaces, etc.)

Container Strategy

Dockerfiles

Eagle Server (ci/eagle_run.Dockerfile - already exists, needs enhancement):

FROM eclipse-temurin:17-jre-alpine

# Add non-root user
RUN addgroup -S eagle && adduser -S eagle -G eagle

WORKDIR /app

# Copy the deploy JAR
COPY --chown=eagle:eagle deploy/eagle_server_deploy.jar ./

# Copy game resources
COPY --chown=eagle:eagle src/main/resources/net/eagle0/eagle/ ./resources/

USER eagle

# Health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD wget -q --spider http://localhost:40032/health || exit 1

EXPOSE 40032

ENTRYPOINT ["java", "-Xmx4g", "-jar", "eagle_server_deploy.jar"]
CMD ["--eagle-grpc-port", "40032", "--shardok-interface-remote-address", "shardok:40042", "--gpt-model-name", "gpt-4"]

Shardok Server (new: ci/shardok_run.Dockerfile):

FROM ubuntu:22.04

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    libstdc++6 \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Add non-root user
RUN useradd -r -s /bin/false shardok

WORKDIR /app

# Copy the Shardok binary and dependencies
COPY --chown=shardok:shardok bazel-bin/src/main/cpp/net/eagle0/shardok/shardok-server ./
COPY --chown=shardok:shardok src/main/resources/net/eagle0/shardok/maps/ ./maps/

USER shardok

# Health check (need to implement gRPC health endpoint)
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD ./shardok-server --health-check || exit 1

EXPOSE 40042 40052

ENTRYPOINT ["./shardok-server"]

Docker Compose (new: docker-compose.prod.yml):

version: '3.8'

services:
  eagle:
    build:
      context: .
      dockerfile: ci/eagle_run.Dockerfile
    image: eagle0/eagle-server:${VERSION:-latest}
    ports:
      - "40032:40032"
    environment:
      - SHARDOK_ADDRESS=shardok:40042
      - GPT_MODEL_NAME=${GPT_MODEL_NAME:-gpt-4}
      - JAVA_OPTS=-Xmx4g -XX:+UseG1GC
    depends_on:
      - shardok
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

  shardok:
    build:
      context: .
      dockerfile: ci/shardok_run.Dockerfile
    image: eagle0/shardok-server:${VERSION:-latest}
    ports:
      - "40042:40042"
      - "40052:40052"
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

Docker Networking & Resource Configuration

Container Network Binding

In Docker, each container has its own network namespace. When a server binds to localhost (127.0.0.1), it only accepts connections from within the same container. Other containers on the Docker bridge network cannot connect, even if they can resolve the hostname.

Shardok must bind to 0.0.0.0:40042 to accept connections from Eagle running in a separate container. This is configured in ServerConfiguration.cpp:

// Default to 0.0.0.0 for container networking (accepts connections from any interface)
{ServerConfiguration::kEagleInterfaceGrpcAddress, "0.0.0.0:40042"}

This still works on the local Mac because 0.0.0.0 means "all interfaces", which includes the loopback interface that localhost uses.

gRPC Plaintext for Internal Communication

By default, gRPC's ManagedChannelBuilder attempts TLS connections. Since Shardok runs in plaintext mode, Eagle must use .usePlaintext() for internal container-to-container communication:

// ServerSetupHelpers.scala
val channelBuilder = ManagedChannelBuilder
  .forAddress(shardokInterfaceAddress, shardokInterfacePort)
  .usePlaintext()  // Required for internal Docker communication

TLS termination happens at nginx for external clients. Internal traffic between Eagle and Shardok stays within the Docker network and doesn't need encryption.

Future consideration: If Shardok moves to a separate droplet, traffic would traverse the network. Options:

  1. Enable TLS between Eagle and Shardok (configure Shardok with certificates)
  2. Use DigitalOcean VPC (private network, still unencrypted but isolated)
  3. Use WireGuard/VPN tunnel between hosts

Bazel Runfiles in Docker

Bazel's runfiles system locates resources relative to the executable using a manifest or directory structure that Bazel sets up at runtime. This doesn't exist in Docker containers.

Solution: Environment variable fallbacks in FilesystemUtils.cpp:

// Check for Docker deployment env vars first
const char* resourcesPath = getenv("SHARDOK_RESOURCES_PATH");
if (resourcesPath != nullptr) {
    return string(resourcesPath) + "/";
}
// Fall back to Bazel runfiles for local development
return rLocation + "/src/main/resources/net/eagle0/shardok/";

The Docker Compose configuration sets these:

environment:
  SHARDOK_RESOURCES_PATH: "/app/resources"
  SHARDOK_MAPS_PATH: "/app/resources/maps"

This allows the same binary to work in both environments:

  • Local dev: No env var → uses Bazel runfiles
  • Docker: Env var set → uses /app/resources/ directly

Build Pipeline

Build Artifacts

The build process produces:

  1. Eagle JAR: bazel-bin/src/main/scala/net/eagle0/eagle/eagle_server_deploy.jar
  2. Shardok binary: bazel-bin/src/main/cpp/net/eagle0/shardok/shardok-server

Cross-Compilation for Linux

Current builds target macOS (self-hosted runner). For production Linux deployment:

Option A: Build in Docker (Recommended)

# Build Eagle (JVM - platform independent)
bazel build //src/main/scala/net/eagle0/eagle:eagle_server_deploy.jar

# Build Shardok in Linux container
docker run --rm -v $(pwd):/workspace -w /workspace \
  ubuntu:22.04 \
  bash -c "apt-get update && apt-get install -y build-essential && bazel build -c opt //src/main/cpp/net/eagle0/shardok:shardok-server"

Option B: Use GitHub-hosted Linux runner

  • Add runs-on: ubuntu-latest workflow
  • Builds directly on Linux
  • May need to cache Bazel to avoid long build times

Option C: Cross-compile on Mac

  • Configure Bazel for Linux cross-compilation
  • More complex setup but faster iteration

Recommendation: Option A (Docker build) for Shardok, since Eagle's JAR is platform-independent.

CI/CD Pipeline

GitHub Actions Workflows

New workflow: .github/workflows/deploy_production.yml

name: Deploy to Production

on:
  push:
    branches: [main]
    paths:
      - 'src/main/scala/**'
      - 'src/main/cpp/**'
      - 'src/main/protobuf/**'
      - 'ci/*.Dockerfile'
      - 'docker-compose.prod.yml'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deployment environment'
        required: true
        default: 'production'
        type: choice
        options:
          - production
          - staging

env:
  REGISTRY: registry.digitalocean.com
  EAGLE_IMAGE: eagle0/eagle-server
  SHARDOK_IMAGE: eagle0/shardok-server

jobs:
  build-eagle:
    runs-on: self-hosted
    outputs:
      version: ${{ steps.version.outputs.version }}
    steps:
      - uses: actions/checkout@v6
        with:
          lfs: false

      - name: Set version
        id: version
        run: echo "version=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Build Eagle server JAR
        run: bazel build //src/main/scala/net/eagle0/eagle:eagle_server_deploy.jar

      - name: Copy artifacts
        run: |
          mkdir -p deploy
          cp bazel-bin/src/main/scala/net/eagle0/eagle/eagle_server_deploy.jar deploy/

      - name: Build Docker image
        run: |
          docker build -f ci/eagle_run.Dockerfile -t ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:${{ steps.version.outputs.version }} .
          docker tag ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:${{ steps.version.outputs.version }} ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:latest

      - name: Push to registry
        run: |
          echo "${{ secrets.DO_REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.DO_REGISTRY_TOKEN }} --password-stdin
          docker push ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:${{ steps.version.outputs.version }}
          docker push ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:latest

  build-shardok:
    runs-on: ubuntu-latest
    needs: []
    steps:
      - uses: actions/checkout@v6
        with:
          lfs: false

      - name: Set version
        id: version
        run: echo "version=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Set up Bazel
        uses: bazelbuild/setup-bazelisk@v2

      - name: Cache Bazel
        uses: actions/cache@v3
        with:
          path: ~/.cache/bazel
          key: bazel-linux-${{ hashFiles('MODULE.bazel', 'MODULE.bazel.lock', 'WORKSPACE') }}

      - name: Build Shardok server
        run: bazel build -c opt //src/main/cpp/net/eagle0/shardok:shardok-server

      - name: Build Docker image
        run: |
          mkdir -p bazel-bin/src/main/cpp/net/eagle0/shardok/
          cp bazel-bin/src/main/cpp/net/eagle0/shardok/shardok-server bazel-bin/src/main/cpp/net/eagle0/shardok/
          docker build -f ci/shardok_run.Dockerfile -t ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:${{ steps.version.outputs.version }} .
          docker tag ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:${{ steps.version.outputs.version }} ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:latest

      - name: Push to registry
        run: |
          echo "${{ secrets.DO_REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.DO_REGISTRY_TOKEN }} --password-stdin
          docker push ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:${{ steps.version.outputs.version }}
          docker push ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:latest

  deploy:
    runs-on: ubuntu-latest
    needs: [build-eagle, build-shardok]
    environment: production
    steps:
      - uses: actions/checkout@v6

      - name: Deploy to DigitalOcean
        uses: appleboy/ssh-action@v1.0.0
        with:
          host: ${{ secrets.DO_DROPLET_IP }}
          username: deploy
          key: ${{ secrets.DO_SSH_KEY }}
          script: |
            cd /opt/eagle0

            # Pull latest images
            docker-compose -f docker-compose.prod.yml pull

            # Rolling restart (zero-downtime if load balancer configured)
            docker-compose -f docker-compose.prod.yml up -d --remove-orphans

            # Wait for health checks
            sleep 30
            docker-compose -f docker-compose.prod.yml ps

            # Cleanup old images
            docker image prune -f

      - name: Verify deployment
        run: |
          # Health check endpoint
          curl -f https://eagle0.net/health || exit 1

      - name: Notify on failure
        if: failure()
        run: |
          # Add Slack/Discord notification here
          echo "Deployment failed!"

Deployment Steps

  1. On push to main:

    • Build Eagle JAR (self-hosted Mac runner - for consistency)
    • Build Shardok binary (GitHub-hosted Ubuntu runner)
    • Build Docker images
    • Push to DigitalOcean Container Registry

  2. Deploy to droplet:

    • SSH to production server
    • Pull new images
    • docker-compose up -d (rolling update)
    • Verify health checks

  3. Rollback:

    # On server
docker-compose -f docker-compose.prod.yml down
docker-compose -f docker-compose.prod.yml pull eagle0/eagle-server:<previous-version>
docker-compose -f docker-compose.prod.yml up -d

Server Setup

Initial Droplet Setup

#!/bin/bash
# Run on fresh DigitalOcean droplet (Ubuntu 22.04)

# Update system
apt-get update && apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

# Install Docker Compose
apt-get install -y docker-compose-plugin

# Create deploy user
useradd -m -s /bin/bash -G docker deploy
mkdir -p /home/deploy/.ssh
# Add your SSH public key to /home/deploy/.ssh/authorized_keys

# Create app directory
mkdir -p /opt/eagle0
chown deploy:deploy /opt/eagle0

# Configure Docker to use DigitalOcean Container Registry
docker login registry.digitalocean.com

# Create systemd service for auto-start
cat > /etc/systemd/system/eagle0.service << 'EOF'
[Unit]
Description=Eagle0 Game Servers
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/eagle0
ExecStart=/usr/bin/docker compose -f docker-compose.prod.yml up -d
ExecStop=/usr/bin/docker compose -f docker-compose.prod.yml down
User=deploy
Group=deploy

[Install]
WantedBy=multi-user.target
EOF

systemctl enable eagle0

Load Balancer Configuration

DigitalOcean Load Balancer settings:

  • Forwarding Rules:

    • HTTPS 443 → HTTP 40032 (Eagle)
    • gRPC is HTTP/2, handled automatically

  • Health Checks:

    • Protocol: HTTP
    • Port: 40032
    • Path: /health (need to implement)

  • SSL:

    • Let's Encrypt certificate for eagle0.net

  • Settings:

    • Sticky sessions: Disabled (gRPC streams handle this)
    • Proxy protocol: Disabled

nginx on Droplet (Alternative to LB)

If using nginx instead of managed LB:

# /etc/nginx/sites-available/eagle0
upstream eagle_backend {
    server 127.0.0.1:40032;
    keepalive 100;
}

server {
    listen 443 ssl http2;
    server_name eagle0.net;

    ssl_certificate /etc/letsencrypt/live/eagle0.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/eagle0.net/privkey.pem;

    # gRPC settings
    location /net.eagle0.eagle.api.Eagle {
        grpc_pass grpc://eagle_backend;
        grpc_read_timeout 1200s;
        grpc_send_timeout 1200s;
        grpc_socket_keepalive on;

        # Rate limiting
        limit_req zone=eagle burst=50 nodelay;
    }

    # Health check endpoint
    location /health {
        proxy_pass http://127.0.0.1:40032/health;
    }
}

# Rate limit zone
limit_req_zone $binary_remote_addr zone=eagle:10m rate=100r/s;

Environment Configuration

Secrets Management

GitHub Secrets (for CI/CD):

  • DO_REGISTRY_TOKEN - DigitalOcean Container Registry token
  • DO_DROPLET_IP - Production server IP
  • DO_SSH_KEY - SSH private key for deployment
  • OPENAI_API_KEY - For LLM integration (if used in production)
  • DO_SPACES_KEY - Already exists for S3

On Server (environment variables):

# /opt/eagle0/.env
GPT_MODEL_NAME=gpt-4
OPENAI_API_KEY=sk-...
DO_SPACES_KEY=...
DO_SPACES_SECRET=...
JAVA_OPTS=-Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=200

Configuration Files

Production config (/opt/eagle0/config/eagle0.conf):

monteCarloIterations = 150000
monteCarloThreads = 4
grpcAddress = 0.0.0.0:40042

Monitoring & Observability

Logging

Docker logging driver: json-file with rotation

  • Logs stored in /var/lib/docker/containers/<id>/
  • Max 5 files of 100MB each

Log aggregation options:

  1. DigitalOcean Logs - $0 for basic, integrates with Droplets
  2. Papertrail - $7/mo for 1GB, good search
  3. Self-hosted Loki - Free, more complex

Metrics

Prometheus + Grafana (optional, for advanced monitoring):

# Add to docker-compose.prod.yml
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}

Key metrics to track:

  • Connection count
  • Request latency (p50, p95, p99)
  • Error rate
  • CPU/memory usage
  • Shardok AI search depth/time

Alerting

DigitalOcean Monitoring Alerts:

  • CPU > 80% for 5 minutes
  • Memory > 90%
  • Disk > 85%
  • Droplet unreachable

Uptime monitoring:

  • Use UptimeRobot (free tier) or Better Uptime
  • Check https://eagle0.net/health every minute

Cost Estimate

Recommended Starting Configuration

Component Monthly Cost Notes
Droplet (s-2vcpu-4gb) $24 Eagle always-on, Shardok on-demand
Spaces ~$5 Already paying
Container Registry $5 For Docker images
DNS $0 Included
Bandwidth ~$0 1TB free, then $0.01/GB
Total ~$34/mo

Optional Add-ons

Component Monthly Cost Notes
Load Balancer +$12 Only if need HA/failover
Monitoring (Papertrail) +$7 Better log search

Scaling Costs

Scenario Droplet Monthly Cost
Current (few players) s-2vcpu-4gb $24
Growing usage s-4vcpu-8gb $48
CPU-intensive battles c-4 (dedicated) $84
High concurrency c-8 (dedicated) $168
Separate Shardok (on-demand) s-1vcpu-2gb + c-4 hourly $12 + usage

Migration Plan

Phase 1: Infrastructure Setup (Day 1)

  1. Create DigitalOcean resources:

    • Droplet in SFO3 region
    • Container Registry
    • (Optional) Load Balancer

  2. Configure DNS:

    • Point eagle0.net to new infrastructure
    • Keep qa.eagle0.net pointing to home IP

  3. Set up server:

    • Run initial setup script
    • Configure Docker and docker-compose
    • Test SSH access

Phase 2: Build Pipeline (Day 2)

  1. Create Dockerfiles:

    • Enhance ci/eagle_run.Dockerfile
    • Create ci/shardok_run.Dockerfile

  2. Create docker-compose.prod.yml

  3. Set up GitHub Actions:

    • Add deployment workflow
    • Configure secrets
    • Test build pipeline

Phase 3: Deployment (Day 3)

  1. Deploy to production:

    • Push first images
    • Start containers
    • Verify health checks

  2. Configure TLS:

    • Set up Let's Encrypt
    • Update nginx/LB configuration

  3. Update DNS:

    • Switch eagle0.net to production
    • Verify client can connect

Phase 4: Validation (Day 4)

  1. Test gameplay:

    • Connect from Unity client
    • Play through Eagle gameplay
    • Test Shardok combat

  2. Monitor:

    • Check logs for errors
    • Verify resource usage
    • Test reconnection behavior

  3. Document:

    • Update runbooks
    • Document rollback procedures

Phase 5: QA Environment (Day 5)

  1. Configure qa.eagle0.net:

    • Keep pointing to home Mac
    • Ensure nginx routes correctly

  2. Test environment switching:

    • Connect to production
    • Switch to QA
    • Verify different game states

Rollback Procedure

Quick Rollback (< 5 minutes)

# SSH to production server
ssh deploy@<droplet-ip>

# Roll back to previous version
cd /opt/eagle0
docker-compose -f docker-compose.prod.yml down
docker pull registry.digitalocean.com/eagle0/eagle-server:<previous-tag>
docker pull registry.digitalocean.com/eagle0/shardok-server:<previous-tag>
VERSION=<previous-tag> docker-compose -f docker-compose.prod.yml up -d

Full Rollback to Home Mac

  1. Update DNS: Point eagle0.net back to home IP
  2. Ensure home Mac servers are running
  3. Wait for DNS propagation (5-30 minutes)

Security Checklist

  • SSH key authentication only (disable password)
  • Firewall configured (only 443, 22 from trusted IPs)
  • TLS 1.3 enforced
  • Secrets in environment variables, not in code
  • Container runs as non-root user
  • Rate limiting on gRPC endpoints
  • Regular security updates (unattended-upgrades)
  • Remove Shardok internal interface from public nginx (per CONNECTION_ARCHITECTURE.md)

Resolved Questions

  1. Game state migration: No migration needed. Saves are currently local only. The codebase has a Persister pattern (with existing AWS support) that could be adapted to save to DO Spaces in the future.

  2. LLM API keys: Yes, OpenAI API keys are needed on the Eagle server. Add OPENAI_API_KEY to the .env file on the production droplet.

  3. Multiple regions: Not needed for now. Single SFO3 region is sufficient.

Remaining Open Questions

  1. Backup strategy: Should we add automated backups for local persistence, or migrate to DO Spaces persistence first?

  2. Autoscaling: Is traffic predictable enough to use fixed instance, or need autoscaling later?

Next Steps

Phase 1: Infrastructure

  1. Approve this plan - Review and discuss any changes
  2. Create DigitalOcean resources - Droplet (s-2vcpu-4gb), Container Registry
  3. Set up droplet - Docker, deploy user, firewall, nginx with TLS

Phase 2: Containerization

  1. Implement Dockerfiles - Eagle and Shardok containers
  2. Implement Shardok lifecycle management - Eagle starts/stops Shardok on-demand
    • Add ShardokProcessManager to Eagle server
    • Start Shardok when battle requested
    • Stop after idle timeout (5 min)
    • Health check before routing traffic

Phase 3: CI/CD

  1. Set up GitHub Actions - Build and push Docker images
  2. Add deployment workflow - SSH deploy to droplet

Phase 4: Migration

  1. Deploy to production - Push images, start services
  2. Update DNS - Point eagle0.net to droplet
  3. Validate - Test gameplay, monitor logs
  4. Configure QA - Ensure qa.eagle0.net still works (home Mac)
Reference in New Issue View Git Blame Copy Permalink