# Eagle0 Productionization Plan

## Executive Summary

This document outlines a plan to move Eagle0's Eagle and Shardok servers from a home Mac to cloud infrastructure while maintaining a QA environment on the Mac. The architecture uses DigitalOcean (already integrated via Spaces) with containerized deployments and GitHub Actions CI/CD.

## Current Architecture

```
Unity Client
     │
     │ gRPC/TLS (eagle0.net:443)
     ▼
nginx (home Mac, via router port forward)
     │
     ├─► Eagle Server (Scala/JVM, port 40032)
     │        │
     │        │ internal gRPC (port 40042)
     │        ▼
     └─► Shardok Server (C++, port 40042/40052)

Storage: DigitalOcean Spaces (sfo3.digitaloceanspaces.com)
DNS: eagle0.net → home IP
```

**Key observations:**
- Already using DigitalOcean Spaces for S3-compatible storage
- Eagle server is a deployable JAR (`eagle_server_deploy.jar`)
- Shardok server is a native C++ binary
- nginx handles TLS termination and gRPC routing
- Self-hosted GitHub Actions runner on Mac

---

## Target Architecture

### Production Environment (DigitalOcean)

```
Unity Client
     │
     ├─► eagle0.net (Production)
     │        │
     │        ▼
     │   DigitalOcean Load Balancer (TLS termination)
     │        │
     │        ▼
     │   ┌─────────────────────────────────────┐
     │   │     DigitalOcean Droplet(s)         │
     │   │  ┌─────────────┬─────────────────┐  │
     │   │  │   Eagle     │    Shardok      │  │
     │   │  │   (Docker)  │    (Docker)     │  │
     │   │  │   :40032    │    :40042       │  │
     │   │  └─────────────┴─────────────────┘  │
     │   └─────────────────────────────────────┘
     │
     └─► qa.eagle0.net (QA - home Mac, unchanged)
              │
              ▼
         nginx → Eagle/Shardok (current setup)
```

### Environment Switching

The Unity client already supports configurable server URLs via the connection screen:
- **Production:** `eagle0.net` (default)
- **QA:** `qa.eagle0.net`

No client code changes needed - users can simply type the desired URL.

---

## Cloud Provider: DigitalOcean

### Why DigitalOcean

1. **Already integrated** - S3 Spaces storage at `sfo3.digitaloceanspaces.com` with credentials configured
2. **Simple pricing** - Predictable monthly costs, no surprise bills
3. **Good performance** - SFO3 datacenter is geographically close
4. **Managed services** - Load balancers, managed databases if needed later
5. **Not AWS/GCP** - Per your requirements

### Alternative Considered: Hetzner

- Cheaper for compute ($3.29/mo for 2 vCPU/4GB vs DO's $24/mo)
- Good EU presence but less US coverage
- No managed load balancer (need to run HAProxy/nginx yourself)
- **Recommendation:** Start with DigitalOcean for simplicity; migrate to Hetzner later if cost becomes a concern

---

## Infrastructure Components

### 1. Compute: DigitalOcean Droplets

#### Resource Characteristics

- **Eagle server:** CPU-light, possibly RAM-heavy (JVM). Should be always available.
- **Shardok server:** Very CPU-heavy (AI algorithms). Only needed during tactical battles. Startup time <1 second.

#### Recommended: On-Demand Shardok (Same Droplet)

For current low player count, optimize for cost while maintaining availability:

| Component | Config | Monthly Cost |
|-----------|--------|--------------|
| Droplet | s-2vcpu-4gb | $24/mo |
| Eagle | Always running | - |
| Shardok | Started on-demand by Eagle, stopped after idle | - |

**How it works:**
1. Eagle server runs 24/7 - game is always available
2. When battle starts, Eagle launches Shardok container/process (<1s startup)
3. After battle ends, idle timer starts (e.g., 5 minutes)
4. On timeout, Eagle stops Shardok to free CPU
5. If new battle starts during idle period, Shardok is already warm

**Shardok lifecycle management** (implement in Eagle):
```scala
// Pseudocode for Eagle's Shardok management
object ShardokManager {
  private var process: Option[Process] = None
  private var idleTimer: Option[Timer] = None

  def ensureRunning(): Unit = {
    cancelIdleTimer()
    if (process.isEmpty) {
      process = Some(startShardokProcess())
      waitForHealthCheck()
    }
  }

  def onBattleEnd(): Unit = {
    idleTimer = Some(scheduleShutdown(5.minutes))
  }

  private def shutdown(): Unit = {
    process.foreach(_.destroy())
    process = None
  }
}
```

#### Future Scaling Options

As player count grows and Shardok needs more CPU:

**Option A: Upgrade Droplet (Simplest)**

| Droplet | vCPU | RAM | Cost | Use Case |
|---------|------|-----|------|----------|
| s-2vcpu-4gb | 2 shared | 4 GB | $24/mo | Current (few players) |
| s-4vcpu-8gb | 4 shared | 8 GB | $48/mo | Moderate usage |
| c-4 | 4 dedicated | 8 GB | $84/mo | CPU-intensive battles |
| c-8 | 8 dedicated | 16 GB | $168/mo | Multiple concurrent battles |

**Option B: Separate Shardok Droplet (API-driven)**

For heavy Shardok usage with cost optimization:

- **Eagle droplet:** s-1vcpu-2gb ($12/mo) - always on
- **Shardok droplet:** Created on-demand via DigitalOcean API
  - c-4 CPU-optimized: $0.125/hour
  - c-8 CPU-optimized: $0.25/hour
  - Created when battle starts, destroyed after idle
  - 30-60s droplet creation time (acceptable if battles are requested in advance)

**Option C: Fly.io for Shardok (Scale-to-Zero)**

For true pay-per-use with fast cold starts:

- **Eagle:** DigitalOcean or Fly.io (~$10/mo)
- **Shardok:** Fly.io with auto-scaling
  - Performance VMs: ~$0.0000022/second when running
  - Cold start: 2-5 seconds
  - Scales to zero when idle

Requires learning Fly.io but offers best cost efficiency for sporadic usage.

**Option D: Multiple Shardok Instances (High Scale)**

For many concurrent battles:

- **Eagle:** Dedicated droplet with more RAM
- **Shardok pool:** Multiple Shardok containers/droplets
- Eagle routes battles to available Shardok instances
- Could use Kubernetes or Docker Swarm for orchestration

This is overkill for now but documented for future reference.

### 2. Load Balancer

**DigitalOcean Load Balancer:** $12/mo

- TLS termination with Let's Encrypt
- Health checks
- Sticky sessions (if needed)
- Can add more droplets later for HA

**Alternative:** Run nginx on the droplet for $0 extra, but lose automatic failover.

### 3. Storage

**Already configured:** DigitalOcean Spaces
- Bucket: `eagle0`
- Region: `sfo3`
- Used for game saves and assets
- Cost: $5/mo base + $0.02/GB storage + $0.01/GB transfer

### 4. DNS

**Option A: DigitalOcean DNS (Recommended)**
- Free with droplets
- Easy integration
- API for automated updates

**Option B: Keep current DNS provider**
- Update A records manually or via script

**DNS Records:**
```
eagle0.net          A     <DO Load Balancer IP>
qa.eagle0.net       A     <Home IP> (unchanged)
*.eagle0.net        A     <DO Load Balancer IP> (wildcard for future)
```

### 5. Firewall

**DigitalOcean Cloud Firewall:** Free

```
Inbound Rules:
- TCP 443 (HTTPS/gRPC) from anywhere → Load Balancer
- TCP 22 (SSH) from your IP only → Droplets
- TCP 40032 (Eagle) from Load Balancer only
- TCP 40042 (Shardok) from Load Balancer only

Outbound Rules:
- All traffic allowed (for external APIs, Spaces, etc.)
```

---

## Container Strategy

### Dockerfiles

**Eagle Server** (`ci/eagle_run.Dockerfile` - already exists, needs enhancement):

```dockerfile
FROM eclipse-temurin:17-jre-alpine

# Add non-root user
RUN addgroup -S eagle && adduser -S eagle -G eagle

WORKDIR /app

# Copy the deploy JAR
COPY --chown=eagle:eagle deploy/eagle_server_deploy.jar ./

# Copy game resources
COPY --chown=eagle:eagle src/main/resources/net/eagle0/eagle/ ./resources/

USER eagle

# Health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD wget -q --spider http://localhost:40032/health || exit 1

EXPOSE 40032

ENTRYPOINT ["java", "-Xmx4g", "-jar", "eagle_server_deploy.jar"]
CMD ["--eagle-grpc-port", "40032", "--shardok-interface-remote-address", "shardok:40042", "--gpt-model-name", "gpt-4"]
```

**Shardok Server** (new: `ci/shardok_run.Dockerfile`):

```dockerfile
FROM ubuntu:22.04

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    libstdc++6 \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Add non-root user
RUN useradd -r -s /bin/false shardok

WORKDIR /app

# Copy the Shardok binary and dependencies
COPY --chown=shardok:shardok bazel-bin/src/main/cpp/net/eagle0/shardok/shardok-server ./
COPY --chown=shardok:shardok src/main/resources/net/eagle0/shardok/maps/ ./maps/

USER shardok

# Health check (need to implement gRPC health endpoint)
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD ./shardok-server --health-check || exit 1

EXPOSE 40042 40052

ENTRYPOINT ["./shardok-server"]
```

**Docker Compose** (new: `docker-compose.prod.yml`):

```yaml
version: '3.8'

services:
  eagle:
    build:
      context: .
      dockerfile: ci/eagle_run.Dockerfile
    image: eagle0/eagle-server:${VERSION:-latest}
    ports:
      - "40032:40032"
    environment:
      - SHARDOK_ADDRESS=shardok:40042
      - GPT_MODEL_NAME=${GPT_MODEL_NAME:-gpt-4}
      - JAVA_OPTS=-Xmx4g -XX:+UseG1GC
    depends_on:
      - shardok
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

  shardok:
    build:
      context: .
      dockerfile: ci/shardok_run.Dockerfile
    image: eagle0/shardok-server:${VERSION:-latest}
    ports:
      - "40042:40042"
      - "40052:40052"
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"
```

---

## Docker Networking & Resource Configuration

### Container Network Binding

In Docker, each container has its own network namespace. When a server binds to `localhost` (127.0.0.1), it only accepts connections from within the same container. Other containers on the Docker bridge network cannot connect, even if they can resolve the hostname.

**Shardok must bind to `0.0.0.0:40042`** to accept connections from Eagle running in a separate container. This is configured in `ServerConfiguration.cpp`:

```cpp
// Default to 0.0.0.0 for container networking (accepts connections from any interface)
{ServerConfiguration::kEagleInterfaceGrpcAddress, "0.0.0.0:40042"}
```

This still works on the local Mac because `0.0.0.0` means "all interfaces", which includes the loopback interface that localhost uses.

### gRPC Plaintext for Internal Communication

By default, gRPC's `ManagedChannelBuilder` attempts TLS connections. Since Shardok runs in plaintext mode, Eagle must use `.usePlaintext()` for internal container-to-container communication:

```scala
// ServerSetupHelpers.scala
val channelBuilder = ManagedChannelBuilder
  .forAddress(shardokInterfaceAddress, shardokInterfacePort)
  .usePlaintext()  // Required for internal Docker communication
```

TLS termination happens at nginx for external clients. Internal traffic between Eagle and Shardok stays within the Docker network and doesn't need encryption.

**Future consideration:** If Shardok moves to a separate droplet, traffic would traverse the network. Options:
1. Enable TLS between Eagle and Shardok (configure Shardok with certificates)
2. Use DigitalOcean VPC (private network, still unencrypted but isolated)
3. Use WireGuard/VPN tunnel between hosts

### Bazel Runfiles in Docker

Bazel's runfiles system locates resources relative to the executable using a manifest or directory structure that Bazel sets up at runtime. This doesn't exist in Docker containers.

**Solution:** Environment variable fallbacks in `FilesystemUtils.cpp`:

```cpp
// Check for Docker deployment env vars first
const char* resourcesPath = getenv("SHARDOK_RESOURCES_PATH");
if (resourcesPath != nullptr) {
    return string(resourcesPath) + "/";
}
// Fall back to Bazel runfiles for local development
return rLocation + "/src/main/resources/net/eagle0/shardok/";
```

The Docker Compose configuration sets these:
```yaml
environment:
  SHARDOK_RESOURCES_PATH: "/app/resources"
  SHARDOK_MAPS_PATH: "/app/resources/maps"
```

This allows the same binary to work in both environments:
- **Local dev:** No env var → uses Bazel runfiles
- **Docker:** Env var set → uses `/app/resources/` directly

---

## Build Pipeline

### Build Artifacts

The build process produces:
1. **Eagle JAR:** `bazel-bin/src/main/scala/net/eagle0/eagle/eagle_server_deploy.jar`
2. **Shardok binary:** `bazel-bin/src/main/cpp/net/eagle0/shardok/shardok-server`

### Cross-Compilation for Linux

Current builds target macOS (self-hosted runner). For production Linux deployment:

**Option A: Build in Docker (Recommended)**
```bash
# Build Eagle (JVM - platform independent)
bazel build //src/main/scala/net/eagle0/eagle:eagle_server_deploy.jar

# Build Shardok in Linux container
docker run --rm -v $(pwd):/workspace -w /workspace \
  ubuntu:22.04 \
  bash -c "apt-get update && apt-get install -y build-essential && bazel build -c opt //src/main/cpp/net/eagle0/shardok:shardok-server"
```

**Option B: Use GitHub-hosted Linux runner**
- Add `runs-on: ubuntu-latest` workflow
- Builds directly on Linux
- May need to cache Bazel to avoid long build times

**Option C: Cross-compile on Mac**
- Configure Bazel for Linux cross-compilation
- More complex setup but faster iteration

**Recommendation:** Option A (Docker build) for Shardok, since Eagle's JAR is platform-independent.

---

## CI/CD Pipeline

### GitHub Actions Workflows

**New workflow:** `.github/workflows/deploy_production.yml`

```yaml
name: Deploy to Production

on:
  push:
    branches: [main]
    paths:
      - 'src/main/scala/**'
      - 'src/main/cpp/**'
      - 'src/main/protobuf/**'
      - 'ci/*.Dockerfile'
      - 'docker-compose.prod.yml'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deployment environment'
        required: true
        default: 'production'
        type: choice
        options:
          - production
          - staging

env:
  REGISTRY: registry.digitalocean.com
  EAGLE_IMAGE: eagle0/eagle-server
  SHARDOK_IMAGE: eagle0/shardok-server

jobs:
  build-eagle:
    runs-on: self-hosted
    outputs:
      version: ${{ steps.version.outputs.version }}
    steps:
      - uses: actions/checkout@v6
        with:
          lfs: false

      - name: Set version
        id: version
        run: echo "version=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Build Eagle server JAR
        run: bazel build //src/main/scala/net/eagle0/eagle:eagle_server_deploy.jar

      - name: Copy artifacts
        run: |
          mkdir -p deploy
          cp bazel-bin/src/main/scala/net/eagle0/eagle/eagle_server_deploy.jar deploy/

      - name: Build Docker image
        run: |
          docker build -f ci/eagle_run.Dockerfile -t ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:${{ steps.version.outputs.version }} .
          docker tag ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:${{ steps.version.outputs.version }} ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:latest

      - name: Push to registry
        run: |
          echo "${{ secrets.DO_REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.DO_REGISTRY_TOKEN }} --password-stdin
          docker push ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:${{ steps.version.outputs.version }}
          docker push ${{ env.REGISTRY }}/${{ env.EAGLE_IMAGE }}:latest

  build-shardok:
    runs-on: ubuntu-latest
    needs: []
    steps:
      - uses: actions/checkout@v6
        with:
          lfs: false

      - name: Set version
        id: version
        run: echo "version=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Set up Bazel
        uses: bazelbuild/setup-bazelisk@v2

      - name: Cache Bazel
        uses: actions/cache@v3
        with:
          path: ~/.cache/bazel
          key: bazel-linux-${{ hashFiles('MODULE.bazel', 'MODULE.bazel.lock', 'WORKSPACE') }}

      - name: Build Shardok server
        run: bazel build -c opt //src/main/cpp/net/eagle0/shardok:shardok-server

      - name: Build Docker image
        run: |
          mkdir -p bazel-bin/src/main/cpp/net/eagle0/shardok/
          cp bazel-bin/src/main/cpp/net/eagle0/shardok/shardok-server bazel-bin/src/main/cpp/net/eagle0/shardok/
          docker build -f ci/shardok_run.Dockerfile -t ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:${{ steps.version.outputs.version }} .
          docker tag ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:${{ steps.version.outputs.version }} ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:latest

      - name: Push to registry
        run: |
          echo "${{ secrets.DO_REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.DO_REGISTRY_TOKEN }} --password-stdin
          docker push ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:${{ steps.version.outputs.version }}
          docker push ${{ env.REGISTRY }}/${{ env.SHARDOK_IMAGE }}:latest

  deploy:
    runs-on: ubuntu-latest
    needs: [build-eagle, build-shardok]
    environment: production
    steps:
      - uses: actions/checkout@v6

      - name: Deploy to DigitalOcean
        uses: appleboy/ssh-action@v1.0.0
        with:
          host: ${{ secrets.DO_DROPLET_IP }}
          username: deploy
          key: ${{ secrets.DO_SSH_KEY }}
          script: |
            cd /opt/eagle0

            # Pull latest images
            docker-compose -f docker-compose.prod.yml pull

            # Rolling restart (zero-downtime if load balancer configured)
            docker-compose -f docker-compose.prod.yml up -d --remove-orphans

            # Wait for health checks
            sleep 30
            docker-compose -f docker-compose.prod.yml ps

            # Cleanup old images
            docker image prune -f

      - name: Verify deployment
        run: |
          # Health check endpoint
          curl -f https://eagle0.net/health || exit 1

      - name: Notify on failure
        if: failure()
        run: |
          # Add Slack/Discord notification here
          echo "Deployment failed!"
```

### Deployment Steps

1. **On push to main:**
   - Build Eagle JAR (self-hosted Mac runner - for consistency)
   - Build Shardok binary (GitHub-hosted Ubuntu runner)
   - Build Docker images
   - Push to DigitalOcean Container Registry

2. **Deploy to droplet:**
   - SSH to production server
   - Pull new images
   - `docker-compose up -d` (rolling update)
   - Verify health checks

3. **Rollback:**
   ```bash
   # On server
   docker-compose -f docker-compose.prod.yml down
   docker-compose -f docker-compose.prod.yml pull eagle0/eagle-server:<previous-version>
   docker-compose -f docker-compose.prod.yml up -d
   ```

---

## Server Setup

### Initial Droplet Setup

```bash
#!/bin/bash
# Run on fresh DigitalOcean droplet (Ubuntu 22.04)

# Update system
apt-get update && apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

# Install Docker Compose
apt-get install -y docker-compose-plugin

# Create deploy user
useradd -m -s /bin/bash -G docker deploy
mkdir -p /home/deploy/.ssh
# Add your SSH public key to /home/deploy/.ssh/authorized_keys

# Create app directory
mkdir -p /opt/eagle0
chown deploy:deploy /opt/eagle0

# Configure Docker to use DigitalOcean Container Registry
docker login registry.digitalocean.com

# Create systemd service for auto-start
cat > /etc/systemd/system/eagle0.service << 'EOF'
[Unit]
Description=Eagle0 Game Servers
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/eagle0
ExecStart=/usr/bin/docker compose -f docker-compose.prod.yml up -d
ExecStop=/usr/bin/docker compose -f docker-compose.prod.yml down
User=deploy
Group=deploy

[Install]
WantedBy=multi-user.target
EOF

systemctl enable eagle0
```

### Load Balancer Configuration

**DigitalOcean Load Balancer settings:**

- **Forwarding Rules:**
  - HTTPS 443 → HTTP 40032 (Eagle)
  - gRPC is HTTP/2, handled automatically

- **Health Checks:**
  - Protocol: HTTP
  - Port: 40032
  - Path: `/health` (need to implement)

- **SSL:**
  - Let's Encrypt certificate for `eagle0.net`

- **Settings:**
  - Sticky sessions: Disabled (gRPC streams handle this)
  - Proxy protocol: Disabled

### nginx on Droplet (Alternative to LB)

If using nginx instead of managed LB:

```nginx
# /etc/nginx/sites-available/eagle0
upstream eagle_backend {
    server 127.0.0.1:40032;
    keepalive 100;
}

server {
    listen 443 ssl http2;
    server_name eagle0.net;

    ssl_certificate /etc/letsencrypt/live/eagle0.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/eagle0.net/privkey.pem;

    # gRPC settings
    location /net.eagle0.eagle.api.Eagle {
        grpc_pass grpc://eagle_backend;
        grpc_read_timeout 1200s;
        grpc_send_timeout 1200s;
        grpc_socket_keepalive on;

        # Rate limiting
        limit_req zone=eagle burst=50 nodelay;
    }

    # Health check endpoint
    location /health {
        proxy_pass http://127.0.0.1:40032/health;
    }
}

# Rate limit zone
limit_req_zone $binary_remote_addr zone=eagle:10m rate=100r/s;
```

---

## Environment Configuration

### Secrets Management

**GitHub Secrets (for CI/CD):**
- `DO_REGISTRY_TOKEN` - DigitalOcean Container Registry token
- `DO_DROPLET_IP` - Production server IP
- `DO_SSH_KEY` - SSH private key for deployment
- `OPENAI_API_KEY` - For LLM integration (if used in production)
- `DO_SPACES_KEY` - Already exists for S3

**On Server (environment variables):**
```bash
# /opt/eagle0/.env
GPT_MODEL_NAME=gpt-4
OPENAI_API_KEY=sk-...
DO_SPACES_KEY=...
DO_SPACES_SECRET=...
JAVA_OPTS=-Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=200
```

### Configuration Files

**Production config** (`/opt/eagle0/config/eagle0.conf`):
```
monteCarloIterations = 150000
monteCarloThreads = 4
grpcAddress = 0.0.0.0:40042
```

---

## Monitoring & Observability

### Logging

**Docker logging driver:** json-file with rotation
- Logs stored in `/var/lib/docker/containers/<id>/`
- Max 5 files of 100MB each

**Log aggregation options:**
1. **DigitalOcean Logs** - $0 for basic, integrates with Droplets
2. **Papertrail** - $7/mo for 1GB, good search
3. **Self-hosted Loki** - Free, more complex

### Metrics

**Prometheus + Grafana** (optional, for advanced monitoring):
```yaml
# Add to docker-compose.prod.yml
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
```

**Key metrics to track:**
- Connection count
- Request latency (p50, p95, p99)
- Error rate
- CPU/memory usage
- Shardok AI search depth/time

### Alerting

**DigitalOcean Monitoring Alerts:**
- CPU > 80% for 5 minutes
- Memory > 90%
- Disk > 85%
- Droplet unreachable

**Uptime monitoring:**
- Use UptimeRobot (free tier) or Better Uptime
- Check `https://eagle0.net/health` every minute

---

## Cost Estimate

### Recommended Starting Configuration

| Component | Monthly Cost | Notes |
|-----------|-------------|-------|
| Droplet (s-2vcpu-4gb) | $24 | Eagle always-on, Shardok on-demand |
| Spaces | ~$5 | Already paying |
| Container Registry | $5 | For Docker images |
| DNS | $0 | Included |
| Bandwidth | ~$0 | 1TB free, then $0.01/GB |
| **Total** | **~$34/mo** | |

### Optional Add-ons

| Component | Monthly Cost | Notes |
|-----------|-------------|-------|
| Load Balancer | +$12 | Only if need HA/failover |
| Monitoring (Papertrail) | +$7 | Better log search |

### Scaling Costs

| Scenario | Droplet | Monthly Cost |
|----------|---------|--------------|
| Current (few players) | s-2vcpu-4gb | $24 |
| Growing usage | s-4vcpu-8gb | $48 |
| CPU-intensive battles | c-4 (dedicated) | $84 |
| High concurrency | c-8 (dedicated) | $168 |
| Separate Shardok (on-demand) | s-1vcpu-2gb + c-4 hourly | $12 + usage |

---

## Migration Plan

### Phase 1: Infrastructure Setup (Day 1)

1. Create DigitalOcean resources:
   - Droplet in SFO3 region
   - Container Registry
   - (Optional) Load Balancer

2. Configure DNS:
   - Point `eagle0.net` to new infrastructure
   - Keep `qa.eagle0.net` pointing to home IP

3. Set up server:
   - Run initial setup script
   - Configure Docker and docker-compose
   - Test SSH access

### Phase 2: Build Pipeline (Day 2)

1. Create Dockerfiles:
   - Enhance `ci/eagle_run.Dockerfile`
   - Create `ci/shardok_run.Dockerfile`

2. Create `docker-compose.prod.yml`

3. Set up GitHub Actions:
   - Add deployment workflow
   - Configure secrets
   - Test build pipeline

### Phase 3: Deployment (Day 3)

1. Deploy to production:
   - Push first images
   - Start containers
   - Verify health checks

2. Configure TLS:
   - Set up Let's Encrypt
   - Update nginx/LB configuration

3. Update DNS:
   - Switch `eagle0.net` to production
   - Verify client can connect

### Phase 4: Validation (Day 4)

1. Test gameplay:
   - Connect from Unity client
   - Play through Eagle gameplay
   - Test Shardok combat

2. Monitor:
   - Check logs for errors
   - Verify resource usage
   - Test reconnection behavior

3. Document:
   - Update runbooks
   - Document rollback procedures

### Phase 5: QA Environment (Day 5)

1. Configure `qa.eagle0.net`:
   - Keep pointing to home Mac
   - Ensure nginx routes correctly

2. Test environment switching:
   - Connect to production
   - Switch to QA
   - Verify different game states

---

## Rollback Procedure

### Quick Rollback (< 5 minutes)

```bash
# SSH to production server
ssh deploy@<droplet-ip>

# Roll back to previous version
cd /opt/eagle0
docker-compose -f docker-compose.prod.yml down
docker pull registry.digitalocean.com/eagle0/eagle-server:<previous-tag>
docker pull registry.digitalocean.com/eagle0/shardok-server:<previous-tag>
VERSION=<previous-tag> docker-compose -f docker-compose.prod.yml up -d
```

### Full Rollback to Home Mac

1. Update DNS: Point `eagle0.net` back to home IP
2. Ensure home Mac servers are running
3. Wait for DNS propagation (5-30 minutes)

---

## Security Checklist

- [ ] SSH key authentication only (disable password)
- [ ] Firewall configured (only 443, 22 from trusted IPs)
- [ ] TLS 1.3 enforced
- [ ] Secrets in environment variables, not in code
- [ ] Container runs as non-root user
- [ ] Rate limiting on gRPC endpoints
- [ ] Regular security updates (`unattended-upgrades`)
- [ ] Remove Shardok internal interface from public nginx (per CONNECTION_ARCHITECTURE.md)

---

## Resolved Questions

1. **Game state migration:** No migration needed. Saves are currently local only. The codebase has a `Persister` pattern (with existing AWS support) that could be adapted to save to DO Spaces in the future.

2. **LLM API keys:** Yes, OpenAI API keys are needed on the Eagle server. Add `OPENAI_API_KEY` to the `.env` file on the production droplet.

3. **Multiple regions:** Not needed for now. Single SFO3 region is sufficient.

## Remaining Open Questions

1. **Backup strategy:** Should we add automated backups for local persistence, or migrate to DO Spaces persistence first?

2. **Autoscaling:** Is traffic predictable enough to use fixed instance, or need autoscaling later?

---

## Next Steps

### Phase 1: Infrastructure
1. **Approve this plan** - Review and discuss any changes
2. **Create DigitalOcean resources** - Droplet (s-2vcpu-4gb), Container Registry
3. **Set up droplet** - Docker, deploy user, firewall, nginx with TLS

### Phase 2: Containerization
4. **Implement Dockerfiles** - Eagle and Shardok containers
5. **Implement Shardok lifecycle management** - Eagle starts/stops Shardok on-demand
   - Add `ShardokProcessManager` to Eagle server
   - Start Shardok when battle requested
   - Stop after idle timeout (5 min)
   - Health check before routing traffic

### Phase 3: CI/CD
6. **Set up GitHub Actions** - Build and push Docker images
7. **Add deployment workflow** - SSH deploy to droplet

### Phase 4: Migration
8. **Deploy to production** - Push images, start services
9. **Update DNS** - Point eagle0.net to droplet
10. **Validate** - Test gameplay, monitor logs
11. **Configure QA** - Ensure qa.eagle0.net still works (home Mac)
