# AI Performance Runner Implementation Plan

## Overview

This document outlines the implementation plan for an automated AI performance testing tool for Shardok. The tool will replicate the manual performance testing currently done through the Unity client's "Custom Battle" interface, providing reproducible and automated performance measurements.

## Goals

1. **Automate Performance Testing**: Eliminate the need for manual Unity client interaction
2. **Reproducible Results**: Ensure consistent test conditions across runs
3. **Detailed Metrics**: Capture the same metrics currently observed manually (commands evaluated at each depth)
4. **Clean Architecture**: Maintain proper dependency boundaries (no src/test dependencies in src/main)

## Directory Structure

```
src/main/cpp/net/eagle0/shardok/ai_performance_runner/
├── AIPerformanceRunner.cpp              # Main binary entry point
├── AIPerformanceRunner.hpp              # Performance metrics structs and helpers
├── PerformanceTestGameStateBuilder.cpp  # Game state setup utilities
├── PerformanceTestGameStateBuilder.hpp  # Game state builder interface
├── BUILD.bazel                          # Build configuration
└── README.md                            # Usage documentation
```

## Implementation Details

### 1. Performance Metrics Structure

```cpp
struct AIPerformanceMetrics {
    int commandNumber;
    int depthAchieved;
    std::map<int, int> commandsEvaluatedAtDepth;  // depth -> count
    std::chrono::milliseconds timeUsed;
    bool minimumDepthCompleted;
    bool searchCompleted;
    std::string selectedCommandType;
};

struct PerformanceTestResults {
    std::string mapName;
    int totalTurns;
    std::vector<AIPerformanceMetrics> commandMetrics;
    double averageDepth;
    double completionRate;
    std::chrono::milliseconds totalTime;
};
```

### 2. Test Configuration

The default configuration replicates the Unity client's "Perf" button:

- **Map**: "Alah" 
- **AI Player**: 6 units with professions 1-6, all battalion type 4 (Heavy Infantry)
- **Human Player**: 6 units (no specific configuration needed since AI will control)
- **Defender Toggle**: Configurable (affects starting positions)

### 3. Key Components

#### AIPerformanceRunner.cpp
- Main entry point with command-line argument parsing
- Test execution loop
- Results formatting and output
- Integration with ShardokEngine and IterativeDeepeningAI

#### PerformanceTestGameStateBuilder.cpp
- Game state creation utilities (migrated from test code)
- Map loading helpers
- Unit placement logic
- Player setup functions

### 4. Build Configuration

```python
load("//tools:copts.bzl", "COPTS")

cc_binary(
    name = "ai_performance_runner",
    srcs = ["AIPerformanceRunner.cpp"],
    copts = COPTS,
    data = [
        "//src/main/resources/net/eagle0/shardok:battalion_types",
        "//src/main/resources/net/eagle0/shardok:settings",
        "//src/main/resources/net/eagle0/shardok/maps",
    ],
    deps = [
        ":performance_test_game_state_builder",
        "//src/main/cpp/net/eagle0/common:time_utils",
        "//src/main/cpp/net/eagle0/shardok/ai:ai_iterative_deepening",
        "//src/main/cpp/net/eagle0/shardok/ai:ai_attacker_strategy_selector",
        "//src/main/cpp/net/eagle0/shardok/ai:ai_defender_strategy_selector",
        "//src/main/cpp/net/eagle0/shardok/ai:ai_score_calculator",
        "//src/main/cpp/net/eagle0/shardok/ai:ai_time_budget",
        "//src/main/cpp/net/eagle0/shardok/ai:ai_water_crossing_command_chooser",
        "//src/main/cpp/net/eagle0/shardok/library:engine",
        "//src/main/cpp/net/eagle0/shardok/library:game_state_w",
        "//src/main/cpp/net/eagle0/shardok/library/settings:game_settings",
        "//src/main/cpp/net/eagle0/shardok/util:battalion_type_registrar",
        "//src/main/cpp/net/eagle0/shardok/util:map_loader",
    ],
)

cc_library(
    name = "performance_test_game_state_builder",
    srcs = ["PerformanceTestGameStateBuilder.cpp"],
    hdrs = [
        "AIPerformanceRunner.hpp",
        "PerformanceTestGameStateBuilder.hpp",
    ],
    copts = COPTS,
    deps = [
        "//src/main/cpp/net/eagle0/common:filesystem_utils",
        "//src/main/cpp/net/eagle0/common:tsv_parser",
        "//src/main/cpp/net/eagle0/shardok/library:game_state_w",
        "//src/main/cpp/net/eagle0/shardok/library/settings:game_settings",
        "//src/main/cpp/net/eagle0/shardok/util:map_loader",
        "//src/main/flatbuffer/net/eagle0/shardok/storage:game_state_cc_fbs",
        "//src/main/flatbuffer/net/eagle0/shardok/storage:player_info_cc_fbs",
        "//src/main/flatbuffer/net/eagle0/shardok/storage:unit_cc_fbs",
    ],
)
```

### 5. Command-Line Interface

```bash
# Run default performance test (Alah map, 6v6 units)
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner

# Run with specific number of turns
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- --turns=10

# Run with defender configuration
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- --defender=true

# Run with verbose output
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- --verbose

# Run with specific map
bazel run //src/main/cpp/net/eagle0/shardok/ai_performance_runner:ai_performance_runner -- --map=Chipingia
```

### 6. Expected Output Format

```
Shardok AI Performance Test
===========================
Map: Alah
Configuration: 6v6 units (AI as attacker)
Time Budget: Dynamic (proximity-based)

Turn 1:
  Command 1: Depth 2, evaluated 140/280 commands, time: 1250ms [MoveCommand]
  Command 2: Depth 2, evaluated ALL commands, time: 1180ms [MeleeCommand]
  Command 3: Depth 3, evaluated 21/156 commands, time: 1300ms [ArcheryCommand]
  Command 4: Depth 3, evaluated 78/312 commands, time: 1290ms [MoveCommand]
  Turn Summary: Avg depth 2.5, Total time: 5020ms

Overall Results:
  Total Turns: 5
  Average Depth Achieved: 2.4
  Commands Completed at Target Depth: 85%
  Total Time: 25.1s
  Average Time per Command: 1255ms
```

### 7. Implementation Phases

#### Phase 1: Basic Infrastructure
1. Create directory structure and BUILD.bazel
2. Implement PerformanceTestGameStateBuilder with minimal game state creation
3. Create basic AIPerformanceRunner that can load a map and create players

#### Phase 2: AI Integration
1. Integrate IterativeDeepeningAI
2. Implement performance metric collection
3. Add basic output formatting

#### Phase 3: Full Feature Set
1. Add command-line argument parsing
2. Implement multiple test configurations (Perf, Rivers, Custom)
3. Add detailed performance metrics and analysis

#### Phase 4: Polish and Documentation
1. Create comprehensive README.md
2. Add error handling and validation
3. Implement baseline comparison features

## Success Criteria

1. **Functional**: Tool successfully runs AI turns and captures performance metrics
2. **Accurate**: Results match manually observed performance within reasonable variance
3. **Reproducible**: Multiple runs produce consistent results
4. **Maintainable**: Clean code structure with no dependencies on src/test
5. **Usable**: Clear command-line interface and helpful output

## Future Enhancements

- JSON output format for automated analysis
- Performance regression detection
- Integration with CI/CD pipeline
- Configurable test scenarios beyond "Perf" and "Rivers"
- Multi-threaded performance testing