eagle0/docs/SHARDOK_AI_EXPERIMENT_TOOLING_PLAN.md

Shardok AI Experiment Tooling Plan

Summary

Build a reusable workflow for Shardok AI battle baselines and scoring experiments. The goal is to turn the current one-off benchmark process into repeatable commands that extract real battles, run baseline and candidate configurations, capture detailed per-command traces, and write comparable reports.

This work starts with the target-aware archery scoring experiment: attacker EXPERIMENTAL against defender STANDARD.

Key Changes

• Add simulator trace support to ai_battle_simulator_main.

  • New flag: --trace-jsonl=<path>.
  • Emit one JSONL row per AI-selected command.
  • Include run label, config path, state file, phase, sequence, round, player, side, command type, actor unit, target coords, chosen index, available command count, search depth, commands evaluated, completion reason, forced commands posted afterward, and pre/post side troop totals.
  • Keep existing summary output unchanged.

• Add reusable benchmark tooling under src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/.

  • real_battle_suite.py: shared helpers for reading game.db, materializing battle/state/result blobs, extracting configs, running simulator binaries, parsing summaries, and collecting real-game metadata.
  • run_real_battle_experiment.py: CLI orchestration for baseline/candidate runs.
  • Outputs: summary.csv, comparison.csv, report.md, generated configs, and per-run trace JSONL files.

• Add a clean EXPERIMENTAL switch for target-aware archery value.

  • Use a new experiment id.
  • Change only archery-availability scoring.
  • Replace the flat archery-possible value with a target-sensitive value based on the best enemy unit value.
  • Do not include the previous fear-value change from experiment 27.

Execution Plan

• Create a feature branch from fetched origin/main.
• Commit this plan document first.
• Implement simulator trace output and reusable benchmark tooling.
• Add the target-aware archery-only EXPERIMENTAL switch.
• Run the new tool first on battle 6248, then on the all-40 saved-battle suite if local runtime is acceptable.
• Compare baseline attacker STANDARD / defender STANDARD with candidate attacker EXPERIMENTAL / defender STANDARD.
• Record the results under the existing benchmark results area and call out battle 6248 in the generated report.
• Push the branch and create a PR after local validation.

Tests

• Build:
  • bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main
  • bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:real_battle_config_extractor
  • bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:real_battle_smoke_metadata
• Unit/integration:
  • bazel test //src/test/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_test
  • Add or extend a simulator test that verifies trace output is created and contains required command fields.
  • python3 -m py_compile on the new benchmark scripts.
• Validation:
  • Run the new tool on battle 6248.
  • Run the new tool on all 40 battles if feasible locally.
  • Run bazel test //src/test/cpp/... before push if local time allows; otherwise open the PR promptly and continue validation while CI starts.

Assumptions

• "Experiment 3" means target-aware archery value from the previous recommendation list.
• The first experiment compares attacker-only candidate behavior against unchanged defender STANDARD.
• JSONL is the trace format because command diagnostics will evolve over time.
• Python benchmark scripts follow the existing executable-script pattern rather than adding Python Bazel targets.