mirror of
https://github.com/nolen777/eagle0.git
synced 2026-07-28 21:35:42 +00:00
78 lines
3.7 KiB
Markdown
78 lines
3.7 KiB
Markdown
# Shardok AI Experiment Tooling Plan
|
|
|
|
## Summary
|
|
|
|
Build a reusable workflow for Shardok AI battle baselines and scoring experiments. The goal is to
|
|
turn the current one-off benchmark process into repeatable commands that extract real battles, run
|
|
baseline and candidate configurations, capture detailed per-command traces, and write comparable
|
|
reports.
|
|
|
|
This work starts with the target-aware archery scoring experiment: attacker `EXPERIMENTAL` against
|
|
defender `STANDARD`.
|
|
|
|
## Key Changes
|
|
|
|
- Add simulator trace support to `ai_battle_simulator_main`.
|
|
- New flag: `--trace-jsonl=<path>`.
|
|
- Emit one JSONL row per AI-selected command.
|
|
- Include run label, config path, state file, phase, sequence, round, player, side, command type,
|
|
actor unit, target coords, chosen index, available command count, search depth, commands
|
|
evaluated, completion reason, forced commands posted afterward, and pre/post side troop totals.
|
|
- Keep existing summary output unchanged.
|
|
|
|
- Add reusable benchmark tooling under
|
|
`src/main/cpp/net/eagle0/shardok/ai_battle_simulator/benchmarks/`.
|
|
- `real_battle_suite.py`: shared helpers for reading `game.db`, materializing battle/state/result
|
|
blobs, extracting configs, running simulator binaries, parsing summaries, and collecting
|
|
real-game metadata.
|
|
- `run_real_battle_experiment.py`: CLI orchestration for baseline/candidate runs.
|
|
- Outputs: `summary.csv`, `comparison.csv`, `report.md`, generated configs, and per-run trace
|
|
JSONL files.
|
|
|
|
- Add a clean EXPERIMENTAL switch for target-aware archery value.
|
|
- Use a new experiment id.
|
|
- Change only archery-availability scoring.
|
|
- Replace the flat archery-possible value with a target-sensitive value based on the best enemy
|
|
unit value.
|
|
- Do not include the previous fear-value change from experiment `27`.
|
|
|
|
## Execution Plan
|
|
|
|
- Create a feature branch from fetched `origin/main`.
|
|
- Commit this plan document first.
|
|
- Implement simulator trace output and reusable benchmark tooling.
|
|
- Add the target-aware archery-only EXPERIMENTAL switch.
|
|
- Run the new tool first on battle `6248`, then on the all-40 saved-battle suite if local runtime is
|
|
acceptable.
|
|
- Compare baseline attacker `STANDARD` / defender `STANDARD` with candidate attacker
|
|
`EXPERIMENTAL` / defender `STANDARD`.
|
|
- Record the results under the existing benchmark results area and call out battle `6248` in the
|
|
generated report.
|
|
- Push the branch and create a PR after local validation.
|
|
|
|
## Tests
|
|
|
|
- Build:
|
|
- `bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_main`
|
|
- `bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:real_battle_config_extractor`
|
|
- `bazel build //src/main/cpp/net/eagle0/shardok/ai_battle_simulator:real_battle_smoke_metadata`
|
|
- Unit/integration:
|
|
- `bazel test //src/test/cpp/net/eagle0/shardok/ai_battle_simulator:ai_battle_simulator_test`
|
|
- Add or extend a simulator test that verifies trace output is created and contains required
|
|
command fields.
|
|
- `python3 -m py_compile` on the new benchmark scripts.
|
|
- Validation:
|
|
- Run the new tool on battle `6248`.
|
|
- Run the new tool on all 40 battles if feasible locally.
|
|
- Run `bazel test //src/test/cpp/...` before push if local time allows; otherwise open the PR
|
|
promptly and continue validation while CI starts.
|
|
|
|
## Assumptions
|
|
|
|
- "Experiment 3" means target-aware archery value from the previous recommendation list.
|
|
- The first experiment compares attacker-only candidate behavior against unchanged defender
|
|
`STANDARD`.
|
|
- JSONL is the trace format because command diagnostics will evolve over time.
|
|
- Python benchmark scripts follow the existing executable-script pattern rather than adding Python
|
|
Bazel targets.
|