# Shardok AI Experiment Tooling Plan

## Summary

Build a reusable workflow for Shardok AI battle baselines and scoring experiments. The goal is to
turn the current one-off benchmark process into repeatable commands that extract real battles, run
baseline and candidate configurations, capture detailed per-command traces, and write comparable
reports.

This work starts with the target-aware archery scoring experiment: attacker `EXPERIMENTAL` against
defender `STANDARD`.

## Key Changes

- Add simulator trace support to `ai_battle_simulator_main`.
  - New flag: `--trace-jsonl=<path>`.
  - Emit one JSONL row per AI-selected command.
  - Include run label, config path, state file, phase, sequence, round, player, side, command type,
    actor unit, target coords, chosen index, available command count, search depth, commands
    evaluated, completion reason, forced commands posted afterward, and pre/post side troop totals.
  - Keep existing summary output unchanged.

- Add reusable benchmark tooling under
  `src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/`.
  - `real_battle_suite.py`: shared helpers for reading `game.db`, materializing battle/state/result
    blobs, extracting configs, running simulator binaries, parsing summaries, and collecting
    real-game metadata.
  - `run_real_battle_experiment.py`: CLI orchestration for baseline/candidate runs.
  - Outputs: `summary.csv`, `comparison.csv`, `report.md`, generated configs, and per-run trace
    JSONL files.

- Add a clean EXPERIMENTAL switch for target-aware archery value.
  - Use a new experiment id.
  - Change only archery-availability scoring.
  - Replace the flat archery-possible value with a target-sensitive value based on the best enemy
    unit value.
  - Do not include the previous fear-value change from experiment `27`.

## Execution Plan

- Create a feature branch from fetched `origin/main`.
- Commit this plan document first.
- Implement simulator trace output and reusable benchmark tooling.
- Add the target-aware archery-only EXPERIMENTAL switch.
- Run the new tool first on battle `6248`, then on the all-40 saved-battle suite if local runtime is
  acceptable.
- Compare baseline attacker `STANDARD` / defender `STANDARD` with candidate attacker
  `EXPERIMENTAL` / defender `STANDARD`.
- Record the results under the existing benchmark results area and call out battle `6248` in the
  generated report.
- Push the branch and create a PR after local validation.

## Tests

- Build:
  - `bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main`
  - `bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:real_battle_config_extractor`
  - `bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:real_battle_smoke_metadata`
- Unit/integration:
  - `bazel test //src/test/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_test`
  - Add or extend a simulator test that verifies trace output is created and contains required
    command fields.
  - `python3 -m py_compile` on the new benchmark scripts.
- Validation:
  - Run the new tool on battle `6248`.
  - Run the new tool on all 40 battles if feasible locally.
  - Run `bazel test //src/test/cpp/...` before push if local time allows; otherwise open the PR
    promptly and continue validation while CI starts.

## Assumptions

- "Experiment 3" means target-aware archery value from the previous recommendation list.
- The first experiment compares attacker-only candidate behavior against unchanged defender
  `STANDARD`.
- JSONL is the trace format because command diagnostics will evolve over time.
- Python benchmark scripts follow the existing executable-script pattern rather than adding Python
  Bazel targets.