# AI Testing Guide

This guide explains how to use the two AI testing tools in the Eagle0 codebase for evaluating and improving AI performance.

## Overview

The codebase provides two complementary tools for AI testing:

1. **AI Performance Runner** - Benchmarks AI decision-making quality over specific turns
2. **AI Battle Simulator** - Tests AI effectiveness by running complete battles

Both tools support the Iterative Deepening and MCTS AI algorithms.

---

## AI Performance Runner

**Location**: `src/main/cpp/net/eagle0/shardok/ai_performance_runner/`

### Purpose

Measures AI decision-making performance to:
- Identify performance regressions when making AI changes
- Understand search depth achieved within time budgets
- Analyze command evaluation patterns at each depth
- Profile AI bottlenecks

### Building

```bash
bazel build //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner
```

### Usage

```bash
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
    --map=MAP_NAME \
    --turns=N \
    [--defender=true|false] \
    [--verbose]
```

### Command-Line Arguments

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `--map` | Yes | - | Map name (e.g., "FourWay", "Bridge") |
| `--turns` | Yes | - | Number of turns to run AI for |
| `--defender` | No | false | If true, test defender AI; if false, test attacker AI |
| `--verbose` | No | false | Enable verbose logging of AI decisions |

### Output Format

For each turn, outputs:
```
Turn 1:
  Depth achieved: 3
  Commands evaluated:
    Depth 1: 45/45 commands (100%)
    Depth 2: 127/234 commands (54%)
    Depth 3: 23/891 commands (3%)
  Completion reason: TIME_LIMIT
  Time elapsed: 5.2s
```

### Performance Metrics Explained

- **Depth achieved**: Maximum lookahead depth the AI reached
- **Commands evaluated**: At each depth, shows commands fully evaluated vs total available
  - Higher depth evaluations are more valuable (depth 3 > depth 2 > depth 1)
  - Completion rates show how thoroughly the AI searched each depth
- **Completion reason**: Why the search stopped
  - `TIME_LIMIT`: Ran out of time budget (normal)
  - `COMPLETE`: Fully evaluated all possibilities (rare, usually only in simple endgames)
  - `DEPTH_LIMIT`: Hit maximum configured depth

### Example Workflow: Testing Performance Improvements

```bash
# 1. Commit your baseline changes
git checkout -b my-performance-improvement
git add . && git commit -m "Baseline before optimization"

# 2. Run performance tests multiple times (reduce noise)
for i in 1 2 3; do
    echo "=== Baseline Run $i ==="
    bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
        --map=FourWay --turns=5 --defender=true | grep -A 10 "Turn"
done
# Save the results

# 3. Make your optimization changes
# ... edit code ...

# 4. Run tests again and compare
for i in 1 2 3; do
    echo "=== Optimized Run $i ==="
    bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
        --map=FourWay --turns=5 --defender=true | grep -A 10 "Turn"
done

# 5. Compare the results
# Look for: increased depth, higher completion rates, more commands evaluated at deeper levels
```

### When to Use Performance Runner

- ✅ Making changes to AI search algorithms
- ✅ Optimizing performance-critical code paths
- ✅ Identifying regressions in decision quality
- ✅ Understanding where the AI spends its time
- ❌ Testing which AI strategy wins more battles (use Battle Simulator instead)

---

## AI Battle Simulator

**Location**: `src/main/cpp/net/eagle0/shardok/ai_battle_simulator/`

### Purpose

Tests AI effectiveness by:
- Running complete AI vs AI battles from start to finish
- Measuring win rates across different AI configurations
- Testing AI behavior with various unit compositions
- Comparing different AI algorithms (MCTS vs Iterative Deepening)

### Building

```bash
bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator
```

### Usage

The battle simulator works with JSON configuration files.

#### Step 1: Generate a Sample Configuration

```bash
bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
    --generate-config=my_battle_config.json
```

This creates a sample configuration file you can customize.

#### Step 2: Customize the Configuration

Edit the generated JSON file:

```json
{
  "map_name": "FourWay",
  "max_rounds": 100,
  "attacker": {
    "ai_algorithm": "ITERATIVE_DEEPENING",
    "units": [
      {
        "battalion_type": "HEAVY_INFANTRY",
        "row": 2,
        "column": 3,
        "strength": 1000,
        "has_hero": true,
        "hero_profession": "WARRIOR",
        "hero_level": 5
      },
      {
        "battalion_type": "ARCHERS",
        "row": 2,
        "column": 4,
        "strength": 800,
        "has_hero": false
      }
    ]
  },
  "defender": {
    "ai_algorithm": "MCTS",
    "units": [
      {
        "battalion_type": "HEAVY_INFANTRY",
        "row": 8,
        "column": 3,
        "strength": 1000,
        "has_hero": true,
        "hero_profession": "WARRIOR",
        "hero_level": 5
      }
    ]
  }
}
```

#### Step 3: Run the Battle

```bash
bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
    --config=my_battle_config.json
```

### Configuration Format

#### Top-Level Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `map_name` | string | Yes | Name of the map to use (e.g., "FourWay", "Bridge") |
| `max_rounds` | integer | Yes | Maximum rounds before declaring a draw |
| `attacker` | object | Yes | Attacker configuration (see below) |
| `defender` | object | Yes | Defender configuration (see below) |

#### Player Configuration (attacker/defender)

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ai_algorithm` | string | Yes | "ITERATIVE_DEEPENING" or "MCTS" |
| `units` | array | Yes | List of unit configurations (see below) |

#### Unit Configuration

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `battalion_type` | string | Yes | - | "HEAVY_INFANTRY", "LIGHT_INFANTRY", "ARCHERS", "CAVALRY", etc. |
| `row` | integer | Yes | - | Starting row position (0-indexed) |
| `column` | integer | Yes | - | Starting column position (0-indexed) |
| `strength` | integer | No | 1000 | Unit strength (max 1000) |
| `has_hero` | boolean | No | false | Whether unit has an attached hero |
| `hero_profession` | string | No* | - | "WARRIOR", "RANGER", "MAGE" (*required if has_hero=true) |
| `hero_level` | integer | No | 1 | Hero level (1-10) |
| `hero_vigor` | integer | No | 100 | Hero vigor (0-100) |

### Output Format

After running a battle, outputs:

```
Battle Result:
  Winner: ATTACKER
  Total Rounds: 47
  Total Commands: 2,341

Attacker Survivors: 3 units
  - Heavy Infantry at (4,5): 723 strength
  - Archers at (3,6): 412 strength
  - Cavalry at (5,4): 891 strength

Defender Survivors: 0 units

Battle Duration: 14.2 seconds
```

### Example Workflow: Testing AI Effectiveness

```bash
# 1. Create a baseline configuration
bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
    --generate-config=baseline.json

# 2. Edit baseline.json to set up your test scenario
# Set both players to ITERATIVE_DEEPENING

# 3. Run multiple battles to get win rate
for i in {1..20}; do
    echo "Battle $i:"
    bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
        --config=baseline.json | grep "Winner"
done

# 4. Create a comparison configuration
cp baseline.json mcts_comparison.json
# Edit mcts_comparison.json: change attacker to MCTS

# 5. Run battles with MCTS attacker
for i in {1..20}; do
    echo "Battle $i:"
    bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
        --config=mcts_comparison.json | grep "Winner"
done

# 6. Compare win rates
# Count ATTACKER wins in each set to see if MCTS performs better/worse
```

### Advanced: Testing Specific Scenarios

The battle simulator is ideal for testing:

**Scenario 1: Hero Ability Usage**
```json
{
  "attacker": {
    "units": [
      {
        "battalion_type": "LIGHT_INFANTRY",
        "has_hero": true,
        "hero_profession": "RANGER",
        "hero_level": 8,
        "row": 2, "column": 3
      }
    ]
  }
}
```
Tests whether AI properly uses Ranger stealth abilities.

**Scenario 2: Terrain Advantage**
```json
{
  "map_name": "BridgeChoke",
  "defender": {
    "units": [
      {"battalion_type": "HEAVY_INFANTRY", "row": 5, "column": 4}
    ]
  }
}
```
Tests whether AI exploits defensive terrain.

**Scenario 3: Numerical Superiority**
```json
{
  "attacker": {
    "units": [
      {"battalion_type": "ARCHERS", "row": 2, "column": 2},
      {"battalion_type": "ARCHERS", "row": 2, "column": 3},
      {"battalion_type": "ARCHERS", "row": 2, "column": 4}
    ]
  },
  "defender": {
    "units": [
      {"battalion_type": "CAVALRY", "row": 8, "column": 3}
    ]
  }
}
```
Tests whether AI properly coordinates multiple units.

### When to Use Battle Simulator

- ✅ Testing which AI strategy wins more battles
- ✅ Measuring AI effectiveness with different unit types
- ✅ Comparing MCTS vs Iterative Deepening algorithms
- ✅ Validating AI behavior in specific tactical scenarios
- ✅ Regression testing: ensuring AI changes don't reduce win rates
- ❌ Profiling performance bottlenecks (use Performance Runner instead)

---

## Combining Both Tools

For comprehensive AI evaluation:

1. **Make AI changes** to improve decision-making
2. **Run Performance Runner** to verify the AI searches deeper or evaluates more commands
3. **Run Battle Simulator** to verify the changes actually improve win rates
4. **Iterate** if performance improves but win rate doesn't (or vice versa)

### Example: Evaluating a New Heuristic

```bash
# 1. Baseline performance
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
    --map=FourWay --turns=3 > baseline_perf.txt

# 2. Baseline effectiveness
for i in {1..10}; do
    bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
        --config=test_scenario.json | grep "Winner"
done > baseline_wins.txt

# 3. Make changes to heuristic
# ... edit code ...

# 4. New performance
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- \
    --map=FourWay --turns=3 > new_perf.txt

# 5. New effectiveness
for i in {1..10}; do
    bazel run //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator -- \
        --config=test_scenario.json | grep "Winner"
done > new_wins.txt

# 6. Compare results
diff baseline_perf.txt new_perf.txt
wc -l < baseline_wins.txt | grep ATTACKER  # Count baseline wins
wc -l < new_wins.txt | grep ATTACKER       # Count new wins
```

---

## Available Maps

Common maps for testing:
- **FourWay**: Open terrain with multiple approach paths
- **Bridge**: Chokepoint scenario testing tactical positioning
- **Forest**: Tests unit behavior in hiding terrain
- **Mountain**: Tests pathfinding around impassable terrain

Find all available maps in: `src/main/resources/net/eagle0/shardok/maps/`

---

## Troubleshooting

### Performance Runner Issues

**Issue**: "Map not found"
```bash
# Solution: Use exact map name without .e0mj extension
# ✅ Correct:
--map=FourWay
# ❌ Incorrect:
--map=FourWay.e0mj
```

**Issue**: AI completes instantly
```bash
# Likely cause: Too few turns or already-won scenario
# Solution: Increase --turns or use more complex map
--turns=10  # Instead of --turns=1
```

### Battle Simulator Issues

**Issue**: "Invalid unit position"
```bash
# Solution: Ensure positions are within map bounds and not overlapping
# Check map size first, then place units accordingly
```

**Issue**: "Battle ends immediately"
```bash
# Cause: Units placed too close or in invalid starting positions
# Solution:
# - Attacker units should start on attacker side (low row numbers)
# - Defender units should start on defender side (high row numbers)
# - Leave space between forces for tactical maneuvering
```

**Issue**: Battle runs extremely slowly
```bash
# Cause: Too many units or max_rounds too high
# Solution: Start with 2-3 units per side, max_rounds=50
# Scale up once basic scenario works
```

---

## Best Practices

### For Performance Testing
1. **Run multiple iterations** (3-5) to reduce variance
2. **Test on consistent maps** to enable before/after comparison
3. **Focus on depth and completion rates** rather than raw time
4. **Test both attacker and defender** AIs (they use different strategies)

### For Effectiveness Testing
1. **Run 10-20 battles** minimum for statistical significance
2. **Use symmetric scenarios** (equal forces) to isolate AI quality
3. **Test multiple maps** to avoid overfitting to one scenario
4. **Document unit compositions** so tests are reproducible
5. **Version control configs** alongside code changes

### For Both
1. **Commit before testing** so you can easily revert
2. **Document unexpected results** (AI might be correct, your intuition wrong)
3. **Test edge cases** (one unit, heroes only, terrain heavy)
4. **Compare algorithms** (MCTS vs Iterative Deepening) regularly
