# Proposal: Server-Based AI Effectiveness Testing

## Executive Summary

Replace proxy-based performance metrics (search depth, nodes visited) with **actual effectiveness testing** (win rates, battle outcomes) by running AI battles through the production Shardok server. This measures what matters: whether AI improvements make the AI smarter, not just faster.

## Problem Statement

### Current State: Measuring the Wrong Things

The existing `ai_performance_runner` measures proxy metrics:
- Search depth achieved
- Number of commands evaluated
- Time spent searching

**Problem**: These metrics don't tell us if the AI is making good decisions.

**Example failure mode**:
- AI searches to depth 4 (looks impressive!)
- But uses terrible heuristics (all decisions are bad)
- Result: Loses every battle despite "good" metrics

### What We Actually Care About

- **Does the AI win?** (win rate)
- **By what margin?** (survivors, rounds taken)
- **Is it tactically sound?** (decision quality in specific scenarios)
- **Does it handle edge cases?** (terrain, heroes, special abilities)

## Proposed Solution

Build a **server-based AI effectiveness testing framework** that:
1. Runs battles through the production Shardok server (real code paths)
2. Measures actual effectiveness (win rates, outcomes)
3. Supports repeatable test scenarios
4. Enables comparison between AI algorithms (MCTS vs Iterative Deepening)
5. Eventually allows Unity clients to watch battles

## Architecture

### Component Overview

```
┌─────────────────────┐
│  Test Scenarios     │
│  (JSON configs)     │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│  Go Test Client     │
│  - Loads scenarios  │
│  - Sends gRPC reqs  │
│  - Collects results │
└──────────┬──────────┘
           │ gRPC
           ▼
┌─────────────────────┐
│  Shardok Server     │
│  - Creates games    │
│  - Runs AI vs AI    │
│  - Returns outcomes │
└─────────────────────┘
```

### Why Go for the Client?

- Native gRPC support with `protoc-gen-go-grpc`
- Easy JSON config parsing
- Good for CLI tools
- Fast compile times for iteration
- Excellent concurrency for running parallel test scenarios

## Implementation Plan

### Phase 1: Core Framework (1 week)

**Goal**: Basic server-based effectiveness testing

#### 1.1 Protocol Extensions

Add to `src/main/protobuf/net/eagle0/common/shardok_internal_interface.proto`:

```protobuf
message TestBattleRequest {
    string game_id = 1;
    string map_name = 2;

    message PlayerSetup {
        bool is_defender = 1;
        AIAlgorithmType ai_algorithm = 2;  // MCTS or ITERATIVE_DEEPENING
        ScoringCalculatorType scoring_type = 3;  // STANDARD, MCTS_OPTIMIZED
        repeated UnitPlacement units = 4;
    }

    repeated PlayerSetup players = 3;
    int32 max_rounds = 4;
}

message UnitPlacement {
    string battalion_type = 1;  // "HEAVY_INFANTRY", "ARCHERS", etc.
    int32 row = 2;
    int32 column = 3;
    int32 strength = 4;
    bool has_hero = 5;
    string hero_profession = 6;  // "WARRIOR", "RANGER", "MAGE"
    int32 hero_level = 7;
}

message TestBattleResponse {
    string game_id = 1;
    int32 winner_player_id = 2;  // -1 for draw
    int32 rounds_taken = 3;
    string end_reason = 4;  // "VICTORY", "DRAW", "MAX_ROUNDS"
    repeated FinalUnitStatus final_units = 5;
}

message FinalUnitStatus {
    int32 player_id = 1;
    string battalion_type = 2;
    int32 strength_remaining = 3;
    int32 row = 4;
    int32 column = 5;
}

// Add to ShardokInternalInterface service:
rpc StartTestBattle(TestBattleRequest) returns (TestBattleResponse) {}
```

**Estimated effort**: 1 day

#### 1.2 Server Implementation

Add handler to `src/main/cpp/net/eagle0/shardok/server/EagleInterfaceGrpcServer.cpp`:

```cpp
Status EagleInterfaceImpl::StartTestBattle(
        ServerContext* context,
        const TestBattleRequest* request,
        TestBattleResponse* response) {
    // 1. Create game with specified setup
    // 2. Mark all players as is_ai=true
    // 3. Wait for game to complete (AI thread handles it)
    // 4. Collect final state and return results
}
```

**Key insight**: The infrastructure already exists! `ShardokGameController` already:
- Creates AI clients automatically for `is_ai=true` players
- Runs AI decisions in background thread
- Tracks game state and completion

**Estimated effort**: 2 days

#### 1.3 Go Test Client

Create `src/main/go/net/eagle0/shardok/ai_effectiveness_runner/`:

```go
package main

import (
    "context"
    "encoding/json"
    "flag"
    "fmt"
    "log"

    pb "eagle0/protobuf/net/eagle0/common"
    "google.golang.org/grpc"
)

type ScenarioConfig struct {
    Name      string        `json:"name"`
    MapName   string        `json:"map_name"`
    MaxRounds int32         `json:"max_rounds"`
    Attacker  PlayerConfig  `json:"attacker"`
    Defender  PlayerConfig  `json:"defender"`
}

type PlayerConfig struct {
    AIAlgorithm  string           `json:"ai_algorithm"`
    ScoringType  string           `json:"scoring_type"`
    Units        []UnitPlacement  `json:"units"`
}

func runTestBattle(client pb.ShardokInternalInterfaceClient, scenario ScenarioConfig) (*pb.TestBattleResponse, error) {
    ctx := context.Background()

    req := &pb.TestBattleRequest{
        GameId:    fmt.Sprintf("test_%s_%d", scenario.Name, time.Now().Unix()),
        MapName:   scenario.MapName,
        MaxRounds: scenario.MaxRounds,
        Players: []*pb.TestBattleRequest_PlayerSetup{
            {
                IsDefender:   false,
                AiAlgorithm:  parseAIAlgorithm(scenario.Attacker.AIAlgorithm),
                ScoringType:  parseScoringType(scenario.Attacker.ScoringType),
                Units:        convertUnits(scenario.Attacker.Units),
            },
            {
                IsDefender:   true,
                AiAlgorithm:  parseAIAlgorithm(scenario.Defender.AIAlgorithm),
                ScoringType:  parseScoringType(scenario.Defender.ScoringType),
                Units:        convertUnits(scenario.Defender.Units),
            },
        },
    }

    return client.StartTestBattle(ctx, req)
}

func main() {
    serverAddr := flag.String("server", "localhost:50051", "Shardok server address")
    scenariosFile := flag.String("scenarios", "test_scenarios.json", "Test scenarios JSON file")
    iterations := flag.Int("iterations", 20, "Number of iterations per scenario")

    flag.Parse()

    // Load scenarios
    scenarios := loadScenarios(*scenariosFile)

    // Connect to server
    conn, err := grpc.Dial(*serverAddr, grpc.WithInsecure())
    if err != nil {
        log.Fatalf("Failed to connect: %v", err)
    }
    defer conn.Close()

    client := pb.NewShardokInternalInterfaceClient(conn)

    // Run effectiveness tests
    results := make(map[string]*ScenarioResults)

    for _, scenario := range scenarios {
        fmt.Printf("\n=== Testing Scenario: %s ===\n", scenario.Name)

        scenarioResults := &ScenarioResults{
            ScenarioName: scenario.Name,
        }

        for i := 0; i < *iterations; i++ {
            resp, err := runTestBattle(client, scenario)
            if err != nil {
                log.Printf("Battle %d failed: %v", i+1, err)
                continue
            }

            scenarioResults.TotalBattles++
            if resp.WinnerPlayerId == 0 {  // Attacker wins
                scenarioResults.AttackerWins++
            } else if resp.WinnerPlayerId == 1 {  // Defender wins
                scenarioResults.DefenderWins++
            } else {
                scenarioResults.Draws++
            }

            scenarioResults.TotalRounds += int(resp.RoundsTaken)
            scenarioResults.TotalSurvivors += len(resp.FinalUnits)

            fmt.Printf("  Battle %d/%d: Winner=%d, Rounds=%d, Survivors=%d\n",
                i+1, *iterations, resp.WinnerPlayerId, resp.RoundsTaken, len(resp.FinalUnits))
        }

        results[scenario.Name] = scenarioResults
    }

    // Print summary
    printSummary(results)
}
```

**Estimated effort**: 2 days

#### 1.4 Test Scenario Definitions

Create `test_scenarios.json`:

```json
{
  "scenarios": [
    {
      "name": "mcts_vs_id_balanced",
      "map_name": "FourWay",
      "max_rounds": 100,
      "attacker": {
        "ai_algorithm": "MCTS",
        "scoring_type": "MCTS_OPTIMIZED",
        "units": [
          {
            "battalion_type": "HEAVY_INFANTRY",
            "row": 2,
            "column": 3,
            "strength": 1000,
            "has_hero": true,
            "hero_profession": "WARRIOR",
            "hero_level": 5
          },
          {
            "battalion_type": "ARCHERS",
            "row": 2,
            "column": 4,
            "strength": 800,
            "has_hero": false
          }
        ]
      },
      "defender": {
        "ai_algorithm": "ITERATIVE_DEEPENING",
        "scoring_type": "STANDARD",
        "units": [
          {
            "battalion_type": "HEAVY_INFANTRY",
            "row": 8,
            "column": 3,
            "strength": 1000,
            "has_hero": true,
            "hero_profession": "WARRIOR",
            "hero_level": 5
          },
          {
            "battalion_type": "ARCHERS",
            "row": 8,
            "column": 4,
            "strength": 800,
            "has_hero": false
          }
        ]
      }
    },
    {
      "name": "terrain_advantage_test",
      "map_name": "Bridge",
      "max_rounds": 100,
      "attacker": {
        "ai_algorithm": "MCTS",
        "scoring_type": "MCTS_OPTIMIZED",
        "units": [
          {
            "battalion_type": "LIGHT_INFANTRY",
            "row": 2,
            "column": 5,
            "strength": 1000,
            "has_hero": false
          }
        ]
      },
      "defender": {
        "ai_algorithm": "MCTS",
        "scoring_type": "MCTS_OPTIMIZED",
        "units": [
          {
            "battalion_type": "HEAVY_INFANTRY",
            "row": 10,
            "column": 5,
            "strength": 800,
            "has_hero": false
          }
        ]
      }
    }
  ]
}
```

**Estimated effort**: 1 day (create comprehensive test suite)

### Phase 2: Enhanced Observability (1 week)

**Goal**: Stream battle updates for real-time monitoring

#### 2.1 Streaming Protocol

Extend protocol to support streaming updates:

```protobuf
message TestBattleUpdate {
    enum UpdateType {
        SETUP_COMPLETE = 0;
        ROUND_START = 1;
        AI_DECISION = 2;
        ROUND_END = 3;
        BATTLE_END = 4;
    }

    UpdateType type = 1;
    int32 current_round = 2;

    // For AI_DECISION updates
    int32 player_id = 3;
    string command_type = 4;  // "MOVE", "MELEE", "ARCHERY", etc.

    // Optional: Include AI metrics as secondary data
    AIDecisionMetrics ai_metrics = 5;

    // For BATTLE_END updates
    TestBattleResponse final_result = 6;
}

message AIDecisionMetrics {
    int32 depth_achieved = 1;
    int32 commands_evaluated = 2;
    int64 time_spent_ms = 3;
}

// Add streaming RPC:
rpc StartTestBattleStreaming(TestBattleRequest) returns (stream TestBattleUpdate) {}
```

**Benefits**:
- Real-time battle monitoring
- Can log/replay interesting decisions
- Still captures proxy metrics as secondary data (if desired)
- Enables debugging of specific scenarios

**Estimated effort**: 3 days

#### 2.2 Go Client Updates

Add streaming support to Go client:

```go
func runTestBattleStreaming(client pb.ShardokInternalInterfaceClient, scenario ScenarioConfig) (*pb.TestBattleResponse, error) {
    ctx := context.Background()
    stream, err := client.StartTestBattleStreaming(ctx, req)
    if err != nil {
        return nil, err
    }

    for {
        update, err := stream.Recv()
        if err == io.EOF {
            break
        }
        if err != nil {
            return nil, err
        }

        switch update.Type {
        case pb.TestBattleUpdate_AI_DECISION:
            fmt.Printf("  Round %d: Player %d chose %s\n",
                update.CurrentRound, update.PlayerId, update.CommandType)
        case pb.TestBattleUpdate_BATTLE_END:
            return update.FinalResult, nil
        }
    }
}
```

**Estimated effort**: 2 days

### Phase 3: Unity Client Integration (2 weeks)

**Goal**: Enable Unity clients to watch AI battles in real-time

#### 3.1 Route Through Eagle

Extend Eagle server to accept test battle requests and create Shardok games:

```scala
// In Eagle server
def startTestBattle(request: TestBattleRequest): Unit = {
  // 1. Create game state in Eagle
  // 2. Send to Shardok via existing protocol
  // 3. Mark both players as AI
  // 4. Allow Unity clients to connect and watch
}
```

**Benefits**:
- Tests complete production stack (Eagle + Shardok)
- Unity clients can connect as spectators
- Most realistic integration test possible

**Estimated effort**: 1 week

#### 3.2 Unity Spectator Mode

Add spectator mode to Unity client:

```csharp
// In Unity client
public class AIBattleSpectator : MonoBehaviour {
    public void ConnectToBattle(string gameId) {
        // Connect to Eagle as spectator
        // Receive battle updates
        // Render on screen
    }
}
```

**Estimated effort**: 1 week

## Usage Examples

### Basic Effectiveness Testing

```bash
# Start Shardok server
bazel run //src/main/cpp/net/eagle0/shardok:shardok-server

# Run effectiveness tests
bazel run //src/main/go/net/eagle0/shardok/ai_effectiveness_runner:ai_effectiveness_runner -- \
    --server=localhost:50051 \
    --scenarios=test_scenarios.json \
    --iterations=20

# Output:
# === Testing Scenario: mcts_vs_id_balanced ===
#   Battle 1/20: Winner=0, Rounds=47, Survivors=3
#   Battle 2/20: Winner=1, Rounds=52, Survivors=2
#   ...
#
# === Summary ===
# Scenario: mcts_vs_id_balanced
#   MCTS (attacker) wins: 15/20 (75%)
#   Iterative Deepening (defender) wins: 5/20 (25%)
#   Average rounds: 48.3
#   Average survivors: 2.8
```

### Comparing AI Improvements

```bash
# Before optimization
git checkout main
bazel run //src/main/go/net/eagle0/shardok/ai_effectiveness_runner:ai_effectiveness_runner -- \
    --scenarios=regression_suite.json --iterations=50 > baseline_results.txt

# After optimization
git checkout my-ai-improvement
bazel run //src/main/go/net/eagle0/shardok/ai_effectiveness_runner:ai_effectiveness_runner -- \
    --scenarios=regression_suite.json --iterations=50 > improved_results.txt

# Compare
diff baseline_results.txt improved_results.txt
# Shows: MCTS win rate improved from 60% to 75%! ✅
```

### Watching Battles in Unity

```bash
# Terminal 1: Eagle server
bazel run //src/main/scala/net/eagle0/eagle:eagle_server

# Terminal 2: Shardok server
bazel run //src/main/cpp/net/eagle0/shardok:shardok-server

# Terminal 3: Start test battle
bazel run //src/main/go/net/eagle0/shardok/ai_effectiveness_runner:ai_effectiveness_runner -- \
    --server=localhost:40032 \
    --scenarios=interesting_scenario.json \
    --stream

# Terminal 4: Unity client
# Open Unity, connect as spectator to watch battle unfold
```

## Success Metrics

### Immediate (Phase 1)
- ✅ Can run 20+ battle scenarios against server
- ✅ Measures win rates, rounds, survivors
- ✅ Tests real production Shardok server code paths
- ✅ Reproducible results

### Medium-term (Phase 2)
- ✅ Streaming battle updates work
- ✅ Can capture and replay interesting battles
- ✅ Logs include both effectiveness metrics and proxy metrics

### Long-term (Phase 3)
- ✅ Unity clients can watch AI battles
- ✅ Tests complete Eagle + Shardok stack
- ✅ Community can watch AI improvements

## Migration Strategy

### Keep Existing Tools

**AI Battle Simulator** (in-process, keep for development):
- Fast iteration during development
- Easy debugging (direct access to internals)
- Use case: "Does this change work at all?"

**AI Effectiveness Runner** (server-based, new primary tool):
- Tests production code paths
- Measures real effectiveness
- Use case: "Is this change actually better?"

### Deprecate Performance Runner

The current `ai_performance_runner` measures proxy metrics. Recommend:
1. Keep it temporarily for comparison
2. After Phase 2 (streaming + AI metrics), deprecate it
3. Streaming effectiveness runner includes proxy metrics as secondary data

## Open Questions

1. **Server Resource Management**: Should we limit concurrent test battles on the server?
   - Proposal: Add `--max-concurrent-battles` flag to Go client

2. **Scenario Versioning**: How do we ensure scenarios remain valid as game evolves?
   - Proposal: Version scenarios in git, validate against server on load

3. **Metrics Storage**: Should we store historical effectiveness metrics?
   - Proposal: Phase 4 (future) - add database for tracking AI effectiveness over time

4. **Randomness Control**: How do we handle dice roll randomness in battles?
   - Current: Protocol supports `roll` override, but battles have many rolls
   - Proposal: Add `random_seed` to TestBattleRequest for reproducibility

## Timeline

- **Phase 1**: 1 week (core framework)
- **Phase 2**: 1 week (streaming + observability)
- **Phase 3**: 2 weeks (Unity integration)

**Total**: 4 weeks for complete vision

**Minimal viable**: Phase 1 only (1 week) provides immediate value

## Next Steps

1. Review and approve this proposal
2. Merge current PR (#4518) - AI testing infrastructure refactor
3. Begin Phase 1 implementation:
   - Protocol extensions (1 day)
   - Server implementation (2 days)
   - Go client (2 days)
   - Test scenarios (1 day)
4. Validate with initial test runs
5. Iterate based on findings
6. Plan Phase 2 based on Phase 1 learnings

## Conclusion

This proposal shifts AI testing from measuring **proxies** (search depth) to measuring **reality** (win rates). By running battles through the production server, we:

- Test what matters: actual intelligence
- Validate real code paths: gRPC, threading, server logic
- Enable future capabilities: Unity viewing, distributed testing
- Measure improvements objectively: win rate changes

The infrastructure mostly exists - ShardokGameController already handles AI vs AI battles. We just need to expose it via protocol and build a client to drive it.

**Recommendation**: Approve and implement Phase 1 (1 week) to immediately gain better AI effectiveness measurement.
