# Shardok AI Scoring Benchmarks

The current STANDARD scoring experiment results are recorded in
`src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/experimental_results/`.
They use 40 saved Shardok battles extracted from `game_389ffb1901903118.zip`.

The corrected all-40 run uses:

- `max_rounds = 31`
- `random_seed = 1`
- iterative-deepening timeout propagation fix from PR #6923
- AI setup placement
- forced single-option commands
- defender scoring fixed to `STANDARD` for every row
- archery/start-fire capability derived like production setup
- per-unit flee capability preserved by real-battle extraction
- zero-troop combat heroes preserved as live units
- `NO_PROFESSION` treated as a valid hero profession state, not as "no hero"

Shardok siege battles do not have a draw result. If the attacker has not won after 31 rounds, the
defender wins. The simulator records those cutoffs as `max_rounds_reached` defender wins.

All 40 saved battle payloads now produce valid simulator configs. A battle side can have zero
remaining troops as long as it has a hero; this is common for `NO_PROFESSION` heroes with no
commanded battalion.

An all-40 real-vs-sim comparison records whether the saved-game attacker/defender were human or AI
players. Factions `3` and `4` are human players; all other factions in this save are AI players.
The comparison uses broad winner side from the saved Shardok result payload, because exact replay
is not available from the compact result rows. Real end-of-battle troop totals come from the
all-factions Shardok `ActionResultView` stream's final `changed_player_totals`.

An AI-vs-AI smoke check also reruns a subset where neither side used faction `3` or `4`.

## Current Results

These artifacts are the current baseline for future scoring comparisons.

| Dataset | Valid battles | Invalid rows | Attacker wins | Defender wins | Notes |
| --- | ---: | ---: | ---: | ---: | --- |
| All-40 timeout-fix STANDARD baseline | 40 | 0 | 18 | 22 | `standard_scoring_all40_post_bugfix_baseline.*`; regenerated after fixing iterative-deepening timeout score propagation |
| All-40 real saved game | 40 known | 0 unknown | 19 | 21 | `standard_scoring_all40_real_vs_sim.*`; sim matches 39/40 known winners |
| AI-vs-AI smoke | 31 | 0 | 10 | 21 | 31/31 winner-side matches |

The remaining real-vs-sim winner-side miss is `11754` (real attacker, sim defender), which had an
AI attacker but a human defender in the saved game. The real attacker won with `2119` troops
remaining; the STANDARD-vs-STANDARD simulation replaces the human defender with AI and the defender
wins with `1000` troops remaining, so this row is not a comparable AI-vs-AI smoke miss. Battle
`6248`, previously the remaining AI-vs-AI miss, now matches as an attacker win after expected-impact
archery scoring was promoted to STANDARD.

The closest six valid baseline rows are used for the scalar and behavior experiment batches. The
single-run harness still has some nondeterminism even with `random_seed = 1`, so near-margin changes
should be repeated before being promoted.

## Experiment Tooling

`src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/run_real_battle_experiment.py`
is the reusable real-battle experiment runner. It can read a saved-game zip or extracted `game.db`,
generate baseline/candidate simulator configs, run both sides, and write `summary.csv`,
`comparison.csv`, `report.md`, generated configs, and per-command trace JSONL files.

The repository keeps the current STANDARD baseline artifacts, but omits generated configs/traces
for abandoned experiment batches. Regenerate those locally with the runner when a new comparison
needs command-level traces.

`src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/run_standard_baselines.py`
regenerates the tracked STANDARD-vs-STANDARD baseline artifacts from a saved-game zip or extracted
`game.db`, including the all-40 baseline, real-vs-sim comparison, and AI-vs-AI smoke files.

The simulator supports `--trace-jsonl=<path>` and `--run-label=<label>` for detailed command traces.
Each row records the selected command, search depth, forced command count, and pre/post side troop
totals.

Expected-impact archery replaced the old flat archery-position value in STANDARD. A unit that can
shoot now gets half of the expected volley impact; a unit that is merely positioned for a future
volley gets one quarter. Promoting that behavior flips `6248` to match the real AI-vs-AI attacker
win without introducing an AI-vs-AI winner-side miss in the current saved-battle baseline.

The latest baseline refresh was generated after the iterative-deepening evaluator stopped letting
timeout fallback scores compete with completed command evaluations. Future STANDARD-vs-EXPERIMENTAL
comparisons should compare against these checked-in timeout-fix baseline artifacts.
