# AI Testing Guide

This guide explains how to use the two AI testing tools in the Eagle0 codebase for evaluating and improving AI performance.

## Overview

The codebase provides two complementary tools for AI testing:

1. **AI Performance Runner** - Benchmarks AI decision-making quality over specific turns
2. **AI Battle Simulator** - Tests AI effectiveness by running complete battles

Both tools support the Iterative Deepening and MCTS AI algorithms.

---

## AI Performance Runner

**Location**: `src/main/cpp/net/eagle0/shardok/ai_performance_runner/`

### Purpose

Measures AI decision-making performance to:
- Identify performance regressions when making AI changes
- Understand search depth achieved within time budgets
- Analyze command evaluation patterns at each depth
- Profile AI bottlenecks

### Building

```bash
bazel build //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner
```

### Usage

```bash
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
    --map=MAP_NAME \
    --turns=N \
    [--defender=true|false] \
    [--verbose]
```

### Command-Line Arguments

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `--map` | Yes | - | Map name (e.g., "FourWay", "Bridge") |
| `--turns` | Yes | - | Number of turns to run AI for |
| `--defender` | No | false | If true, test defender AI; if false, test attacker AI |
| `--verbose` | No | false | Enable verbose logging of AI decisions |

### Output Format

For each turn, outputs:
```
Turn 1:
  Depth achieved: 3
  Commands evaluated:
    Depth 1: 45/45 commands (100%)
    Depth 2: 127/234 commands (54%)
    Depth 3: 23/891 commands (3%)
  Completion reason: TIME_LIMIT
  Time elapsed: 5.2s
```

### Performance Metrics Explained

- **Depth achieved**: Maximum lookahead depth the AI reached
- **Commands evaluated**: At each depth, shows commands fully evaluated vs total available
  - Higher depth evaluations are more valuable (depth 3 > depth 2 > depth 1)
  - Completion rates show how thoroughly the AI searched each depth
- **Completion reason**: Why the search stopped
  - `TIME_LIMIT`: Ran out of time budget (normal)
  - `COMPLETE`: Fully evaluated all possibilities (rare, usually only in simple endgames)
  - `DEPTH_LIMIT`: Hit maximum configured depth

### Example Workflow: Testing Performance Improvements

```bash
# 1. Commit your baseline changes
git checkout -b my-performance-improvement
git add . && git commit -m "Baseline before optimization"

# 2. Run performance tests multiple times (reduce noise)
for i in 1 2 3; do
    echo "=== Baseline Run $i ==="
    bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
        --map=FourWay --turns=5 --defender=true | grep -A 10 "Turn"
done
# Save the results

# 3. Make your optimization changes
# ... edit code ...

# 4. Run tests again and compare
for i in 1 2 3; do
    echo "=== Optimized Run $i ==="
    bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
        --map=FourWay --turns=5 --defender=true | grep -A 10 "Turn"
done

# 5. Compare the results
# Look for: increased depth, higher completion rates, more commands evaluated at deeper levels
```

### When to Use Performance Runner

- ✅ Making changes to AI search algorithms
- ✅ Optimizing performance-critical code paths
- ✅ Identifying regressions in decision quality
- ✅ Understanding where the AI spends its time
- ❌ Testing which AI strategy wins more battles (use Battle Simulator instead)

---

## AI Battle Simulator

**Location**: `src/main/cpp/net/eagle0/shardok/ai_battle_simulator/`

### Purpose

Tests AI effectiveness by:
- Running complete AI vs AI battles from start to finish
- Measuring win rates across different AI configurations
- Testing AI behavior with various unit compositions
- Comparing different AI algorithms (MCTS vs Iterative Deepening)

### Building

```bash
bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main
```

### Usage

The battle simulator works with JSON configuration files.

#### Step 1: Generate a Sample Configuration

```bash
bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main -- \
    --generate-config --output=my_battle_config.json
```

This creates a sample configuration file you can customize.

#### Step 2: Customize the Configuration

Edit the generated JSON file:

```json
{
  "map_name": "Alah",
  "month": 4,
  "max_rounds": 31,
  "random_seed": 1,
  "use_configured_placements": false,
  "apply_forced_commands": true,
  "attacker": {
    "ai_algorithm": "ITERATIVE_DEEPENING",
    "scoring_calculator": "STANDARD",
    "units": [
      {
        "profession": 1,
        "battalion_type_id": 4,
        "starting_position_index": 0,
        "can_flee": true,
        "battalion": {
          "size": 1000,
          "armament": 102,
          "training": 102,
          "morale": 76
        }
      }
    ]
  },
  "defender": {
    "ai_algorithm": "MCTS",
    "scoring_calculator": "MCTS_OPTIMIZED",
    "mcts_config": {
      "exploration_constant": 1.41421356237,
      "max_simulation_depth": 20,
      "max_tree_depth": 8,
      "use_multithreading": false,
      "num_threads": 1,
      "simulation_policy": "WEIGHTED_HEURISTIC",
      "backpropagation_policy": "MINIMAX",
      "max_player_flips": 4,
      "max_simulation_flips": 8,
      "random_seed": 1
    },
    "units": [
      {
        "profession": 14,
        "battalion_type_id": 0,
        "starting_position_index": -1
      }
    ]
  }
}
```

#### Step 3: Run the Battle

```bash
bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main -- \
    --config=my_battle_config.json
```

For compact benchmark output, pass one or more configs and request CSV:

```bash
bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main
```

```bash
bazel-bin/src/main/cpp/net/eagle0/shardok/ai_battle_simulator/ai_battle_simulator_main \
    --config=src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/id_standard_seed1.json \
    --summary=csv \
    --quiet
```

Running the built binary directly keeps Bazel progress output out of the CSV stream.
If you need strict CSV on stdout, redirect stderr as well; one static initialization message is
emitted before `--quiet` can take effect.

You can also start from a serialized Shardok `GameState` flatbuffer instead of a generated simulator
setup. This is useful for saved-game positions:

```bash
bazel-bin/src/main/cpp/net/eagle0/shardok/ai_battle_simulator/ai_battle_simulator_main \
    --config=src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/id_standard_seed1.json \
    --state-file=/tmp/shardok_state.fb \
    --summary=csv \
    --quiet
```

Saved `game.db` files store the latest Shardok state for each battle in `shardok_state.game_state`.
The SQLite CLI can export those blobs directly:

```bash
sqlite3 /path/to/game.db "select writefile('/tmp/' || shardok_game_id || '.fb', game_state) from shardok_state"
```

If the exported state is already terminal, the simulator will report the saved outcome and casualties
with zero AI commands. Nonterminal saved states can be used as real-world decision benchmarks.

### Configuration Format

#### Top-Level Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `map_name` | string | Yes | Name of the map to use (e.g., "FourWay", "Bridge") |
| `month` | integer | No | Month index used for initial game state setup |
| `max_rounds` | integer | No | Maximum rounds before the defender wins by timeout; generated configs default to `31` |
| `random_seed` | integer | No | Seed for reproducible engine randomness; `0` uses system randomness |
| `use_configured_placements` | boolean | No | If true, units with `has_fixed_position` are placed at their configured row/column during setup |
| `apply_forced_commands` | boolean | No | If true, forced single-option commands are posted after each AI command and included in command totals |
| `attacker` | object | Yes | Attacker configuration (see below) |
| `defender` | object | Yes | Defender configuration (see below) |

#### Player Configuration (attacker/defender)

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ai_algorithm` | string | Yes | "ITERATIVE_DEEPENING" or "MCTS" |
| `units` | array | Yes | List of unit configurations (see below) |
| `scoring_calculator` | string | No | "STANDARD" or "MCTS_OPTIMIZED"; defaults to "STANDARD" |
| `mcts_config` | object | No | MCTS-only search settings (see below) |

#### Unit Configuration

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `profession` | integer | Yes | - | Hero profession id; use `14` for a hero with no profession |
| `battalion_type_id` | integer | Yes | - | Battalion type id from Shardok battalion type data |
| `starting_position_index` | integer | Yes | - | Setup placement slot used by setup AI (`0` for attacker examples, `-1` for defender examples) |
| `battalion` | object | No | Client-like defaults | Battalion stats: `size`, `armament`, `training`, `morale` |
| `hero` | object | No | Client-like defaults | Hero stats for the attached hero; `profession: 14` is a valid no-profession hero |
| `can_flee` | boolean | No | true | Whether this unit can flee. Real-battle extraction sets this from the source army's flee province. |
| `has_fixed_position` | boolean | No | false | Enables direct row/column placement when `use_configured_placements` is true |
| `row` | integer | No | - | Fixed placement row |
| `column` | integer | No | - | Fixed placement column |

#### Hero Configuration

Hero stats are numeric fields: `strength`, `agility`, `wisdom`, `charisma`, `constitution`, `bravery`, `integrity`, `ambition`, and `vigor`.

#### MCTS Configuration

| Field | Type | Description |
|-------|------|-------------|
| `exploration_constant` | number | UCT exploration constant |
| `max_simulation_depth` | integer | Maximum rollout depth |
| `max_tree_depth` | integer | Maximum tree expansion depth |
| `use_multithreading` | boolean | Enables threaded MCTS search |
| `num_threads` | integer | Thread count when multithreading is enabled |
| `simulation_policy` | string | "RANDOM", "FILTERED_RANDOM", "BEST_IMMEDIATE", "WEIGHTED_BEST_IMMEDIATE", or "WEIGHTED_HEURISTIC" |
| `backpropagation_policy` | string | "AVERAGING" or "MINIMAX" |
| `max_player_flips` | integer | Maximum player perspective flips during tree search |
| `max_simulation_flips` | integer | Maximum player perspective flips during rollouts |
| `random_seed` | integer | MCTS-specific seed; overrides the battle seed for that player's MCTS search |

### Output Format

Human output is the default. For benchmark runs, `--summary=csv --quiet` emits one row per config:

```csv
config,winner,end_reason,result,total_rounds,total_commands,attacker_ai_commands,defender_ai_commands,attacker_avg_depth,defender_avg_depth,attacker_surviving_units,defender_surviving_units,attacker_surviving_troops,defender_surviving_troops
```

`--summary=json --quiet` emits newline-delimited JSON objects with the same fields.

After running a battle, outputs:

```
Battle Result:
  Winner: ATTACKER
  Total Rounds: 47
  Total Commands: 2,341
  Attacker AI Commands: 1,102
  Defender AI Commands: 1,097
  Attacker Avg Depth: 3.8
  Defender Avg Depth: 2.6

Attacker Survivors: 3 units
  - Heavy Infantry at (4,5): 723 strength
  - Archers at (3,6): 412 strength
  - Cavalry at (5,4): 891 strength

Defender Survivors: 0 units

Battle Duration: 14.2 seconds
```

### Example Workflow: Testing AI Effectiveness

```bash
# 1. Run the checked-in seed scenarios on the baseline commit
bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main
```

```bash
bazel-bin/src/main/cpp/net/eagle0/shardok/ai_battle_simulator/ai_battle_simulator_main \
    --config=src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/id_standard_seed1.json \
    --summary=csv \
    --quiet
```

```text
# 2. Make the scoring change

# 3. Run the same configs again and compare winner, rounds, command counts,
# survivor troop totals, and achieved depth.
```

For scoring work, prefer fixed `random_seed` values and compare the same configs before and after
the code change. To compare two scoring calculators that both exist in the same build, pit baseline
against candidate by assigning different `scoring_calculator` values to attacker and defender, then
run the mirrored scenario with sides swapped to control for map and side bias.

Real-world battle data from a game database can feed this same workflow by exporting each observed
battle setup into the simulator JSON format. Keep the original battle id in the config filename so
CSV rows can be traced back to the source battle.

### Advanced: Testing Specific Scenarios

The battle simulator is ideal for testing:

**Scenario 1: Hero Ability Usage**
```json
{
  "attacker": {
    "units": [
      {
        "profession": 2,
        "battalion_type_id": 0,
        "starting_position_index": 0,
        "hero": {
          "agility": 120,
          "wisdom": 95,
          "vigor": 100
        }
      }
    ]
  }
}
```
Tests whether AI properly uses Ranger stealth abilities.

**Scenario 2: Terrain Advantage**
```json
{
  "map_name": "BridgeChoke",
  "use_configured_placements": true,
  "defender": {
    "units": [
      {
        "profession": 14,
        "battalion_type_id": 0,
        "starting_position_index": -1,
        "has_fixed_position": true,
        "row": 5,
        "column": 4
      }
    ]
  }
}
```
Tests whether AI exploits defensive terrain.

**Scenario 3: Numerical Superiority**
```json
{
  "attacker": {
    "units": [
      {"profession": 1, "battalion_type_id": 4, "starting_position_index": 0},
      {"profession": 2, "battalion_type_id": 4, "starting_position_index": 0},
      {"profession": 3, "battalion_type_id": 4, "starting_position_index": 0}
    ]
  },
  "defender": {
    "units": [
      {"profession": 14, "battalion_type_id": 0, "starting_position_index": -1}
    ]
  }
}
```
Tests whether AI properly coordinates multiple units.

### When to Use Battle Simulator

- ✅ Testing which AI strategy wins more battles
- ✅ Measuring AI effectiveness with different unit types
- ✅ Comparing MCTS vs Iterative Deepening algorithms
- ✅ Validating AI behavior in specific tactical scenarios
- ✅ Regression testing: ensuring AI changes don't reduce win rates
- ❌ Profiling performance bottlenecks (use Performance Runner instead)

---

## Combining Both Tools

For comprehensive AI evaluation:

1. **Make AI changes** to improve decision-making
2. **Run Performance Runner** to verify the AI searches deeper or evaluates more commands
3. **Run Battle Simulator** to verify the changes actually improve win rates
4. **Iterate** if performance improves but win rate doesn't (or vice versa)

### Example: Evaluating a New Heuristic

```bash
# 1. Baseline performance
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
    --map=FourWay --turns=3 > baseline_perf.txt

# 2. Baseline effectiveness
for i in {1..10}; do
    bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main -- \
        --config=test_scenario.json | grep "Winner"
done > baseline_wins.txt

# 3. Make changes to heuristic
# ... edit code ...

# 4. New performance
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
    --map=FourWay --turns=3 > new_perf.txt

# 5. New effectiveness
for i in {1..10}; do
    bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main -- \
        --config=test_scenario.json | grep "Winner"
done > new_wins.txt

# 6. Compare results
diff baseline_perf.txt new_perf.txt
wc -l < baseline_wins.txt | grep ATTACKER  # Count baseline wins
wc -l < new_wins.txt | grep ATTACKER       # Count new wins
```

---

## Available Maps

Common maps for testing:
- **FourWay**: Open terrain with multiple approach paths
- **Bridge**: Chokepoint scenario testing tactical positioning
- **Forest**: Tests unit behavior in hiding terrain
- **Mountain**: Tests pathfinding around impassable terrain

Find all available maps in: `src/main/resources/net/eagle0/shardok/maps/`

---

## Troubleshooting

### Performance Runner Issues

**Issue**: "Map not found"
```bash
# Solution: Use exact map name without .e0mj extension
# ✅ Correct:
--map=FourWay
# ❌ Incorrect:
--map=FourWay.e0mj
```

**Issue**: AI completes instantly
```bash
# Likely cause: Too few turns or already-won scenario
# Solution: Increase --turns or use more complex map
--turns=10  # Instead of --turns=1
```

### Battle Simulator Issues

**Issue**: "Invalid unit position"
```bash
# Solution: Ensure positions are within map bounds and not overlapping
# Check map size first, then place units accordingly
```

**Issue**: "Battle ends immediately"
```bash
# Cause: Units placed too close or in invalid starting positions
# Solution:
# - Attacker units should start on attacker side (low row numbers)
# - Defender units should start on defender side (high row numbers)
# - Leave space between forces for tactical maneuvering
```

**Issue**: Battle runs extremely slowly
```bash
# Cause: Too many units or max_rounds too high
# Solution: Start with 2-3 units per side, max_rounds=31
# Scale up once basic scenario works
```

---

## Best Practices

### For Performance Testing
1. **Run multiple iterations** (3-5) to reduce variance
2. **Test on consistent maps** to enable before/after comparison
3. **Focus on depth and completion rates** rather than raw time
4. **Test both attacker and defender** AIs (they use different strategies)

### For Effectiveness Testing
1. **Run 10-20 battles** minimum for statistical significance
2. **Use symmetric scenarios** (equal forces) to isolate AI quality
3. **Test multiple maps** to avoid overfitting to one scenario
4. **Document unit compositions** so tests are reproducible
5. **Version control configs** alongside code changes

### For Both
1. **Commit before testing** so you can easily revert
2. **Document unexpected results** (AI might be correct, your intuition wrong)
3. **Test edge cases** (one unit, heroes only, terrain heavy)
4. **Compare algorithms** (MCTS vs Iterative Deepening) regularly
