# Chance Nodes in MCTS for Shardok

## Problem Statement

### Current Behavior
The current MCTS implementation uses a fixed roll (50th percentile) for all probabilistic outcomes during simulation. This creates several issues:

1. **Binary success actions overvalued**: A START_FIRE command with 51% success is treated as always succeeding, making it appear better than it actually is.
2. **Discontinuity at 50%**: Actions with 49% vs 51% success have dramatically different evaluations, when they should be similar.
3. **Variable-outcome actions simplified**: Melee/archery attacks with damage ranges are evaluated at a single point rather than their full distribution.

### Example Issue
```
START_FIRE with 51% success:
- Current MCTS: Assumes always succeeds (roll = 50)
- Reality: Succeeds 51% of time, fails 49% of time
- Result: AI overvalues this action
```

### How Iterative Deepening Solves This
The iterative deepening AI (see `AICommandEvaluator.cpp:352-393`) handles randomness correctly:

```cpp
// For actions with odds (binary success/fail):
// 1. Evaluate success outcome with representative roll
auto [successScore, successLookahead] = EvaluateWithRandomness(
    ...,
    std::make_shared<SequenceRandomGenerator>(std::vector{1.0 - successChance / 2.0})
);

// 2. Evaluate failure outcome with representative roll
auto [failureScore, failureLookahead] = EvaluateWithRandomness(
    ...,
    std::make_shared<SequenceRandomGenerator>(std::vector{(1.0 - successChance) / 2.0})
);

// 3. Compute weighted average (expected value)
immediateScore = std::lerp(failureScore, successScore, successChance);
lookaheadScore = std::lerp(failureLookahead.get(), successLookahead.get(), successChance);
```

This is essentially an implicit form of chance nodes - evaluating both outcomes and weighting by probability.

## Chance Nodes Concept

### Classic MCTS with Chance Nodes

In games with randomness (e.g., backgammon), MCTS uses two types of nodes:

1. **Decision Nodes**: Player chooses an action
   - Selection uses UCB formula (exploration/exploitation tradeoff)
   - One child per legal action

2. **Chance Nodes**: Nature determines outcome
   - Selection uses expectation (weighted by probability)
   - One child per possible outcome

```
Decision Node (Player to move)
├─ Action A
│  └─ Chance Node
│     ├─ Outcome 1 (prob 0.3) → Game State
│     ├─ Outcome 2 (prob 0.5) → Game State
│     └─ Outcome 3 (prob 0.2) → Game State
└─ Action B
   └─ Deterministic → Game State
```

### Example: START_FIRE in Shardok

**Current approach:**
```
State S
└─ START_FIRE (roll=50)
   └─ State S' (fire always starts)
```

**With chance nodes:**
```
State S
└─ START_FIRE action
   └─ Chance Node
      ├─ Success (51%) → State S_success (fire started)
      └─ Failure (49%) → State S_failure (no fire, vigor spent)
```

### Value Propagation

**Decision nodes:** Maximize/minimize over children (depending on player)
**Chance nodes:** Expected value over children (weighted by probability)

```cpp
// Decision node value (max for current player)
value = max(child.value for child in children)

// Chance node value (expectation)
value = sum(prob[i] * child[i].value for i in outcomes)
```

## Implementation Approaches

### Option 1: Explicit Chance Nodes (Full Implementation)

Modify the MCTS tree structure to explicitly represent chance nodes.

**Pros:**
- Theoretically sound
- Handles arbitrary outcome distributions
- Clear separation of decision vs chance

**Cons:**
- Significant code changes
- Larger tree (more memory)
- More complex tree traversal

**Tree Structure:**
```cpp
enum class NodeType { DECISION, CHANCE };

struct MCTSNode {
    NodeType type;

    // For decision nodes
    MCTSPlayerId player;
    std::vector<std::unique_ptr<MCTSAction>> actions;
    std::vector<std::unique_ptr<MCTSNode>> children;  // One per action

    // For chance nodes
    std::vector<double> probabilities;  // One per outcome
    std::vector<std::unique_ptr<MCTSNode>> outcomes;  // One per outcome

    double visits;
    double totalReward;
};
```

**Selection Phase:**
```cpp
MCTSNode* select(MCTSNode* node) {
    while (!node->isLeaf()) {
        if (node->type == DECISION) {
            // Use UCB to select action
            node = selectChildUCB(node);
        } else {  // CHANCE node
            // Use probability-weighted selection
            node = selectOutcomeByProbability(node);
        }
    }
    return node;
}
```

**Backpropagation:**
```cpp
void backpropagate(MCTSNode* node, double reward) {
    while (node != nullptr) {
        node->visits++;
        if (node->type == DECISION) {
            node->totalReward += reward;  // Sum for averaging
        } else {  // CHANCE node
            node->totalReward += reward;  // Still sum, but averaged differently
        }
        node = node->parent;
    }
}
```

### Option 2: Implicit Chance Nodes (Hybrid Approach)

Keep the current tree structure but sample outcomes during expansion/simulation.

**Pros:**
- Smaller code changes
- More memory efficient
- Easier to implement incrementally

**Cons:**
- Less theoretically pure
- May need more visits to converge
- Sampling introduces variance

**Approach:**
```cpp
// During expansion
std::unique_ptr<MCTSGameState> expand(
    const MCTSGameState& state,
    const MCTSAction& action
) {
    if (action.isDeterministic()) {
        return applyActionDeterministic(state, action);
    } else {
        // Sample an outcome based on probabilities
        auto outcome = sampleOutcome(action);
        return applyActionWithOutcome(state, action, outcome);
    }
}
```

**For binary actions (e.g., START_FIRE):**
```cpp
// Expand creates one of two children based on sampling
if (random() < successProbability) {
    return applySuccess(state, action);
} else {
    return applyFailure(state, action);
}

// Over many visits, visit ratio will approach probability ratio
// E.g., 51% success action will have ~51% success children, 49% failure children
```

### Option 3: Determinized Sampling (Simplest)

Pre-sample all random outcomes at the start of each simulation rollout.

**Pros:**
- Minimal code changes
- Easy to understand
- Works with existing tree structure

**Cons:**
- May converge slowly
- Doesn't explicitly represent probability
- Can waste simulations on unlikely outcomes

**Approach:**
```cpp
// At start of each simulation
std::vector<double> rollSequence = generateRollSequence(maxDepth);

// Use sequence during simulation
auto state = rootState;
for (int depth = 0; depth < maxDepth; depth++) {
    auto action = selectAction(state);
    state = applyAction(state, action, rollSequence[depth]);
}
```

## Recommended Approach: Progressive Enhancement

Implement in phases to manage complexity:

### Phase 1: Binary Chance Nodes (Explicit)

Start with actions that have clear success/failure outcomes (e.g., START_FIRE, EXTINGUISH_FIRE, RAISE_DEAD):

1. Identify binary actions (commands with `HasOdds()`)
2. Add chance node support for these actions only
3. Modify tree expansion to create chance nodes
4. Update selection/backpropagation for chance nodes

**Implementation:**
```cpp
// In ShardokGameEngine::getLegalActions()
// Mark which actions require chance nodes
struct ActionMetadata {
    std::unique_ptr<MCTSAction> action;
    bool requiresChanceNode;
    double successProbability;  // If requiresChanceNode = true
};
```

```cpp
// In tree expansion
if (action.requiresChanceNode) {
    // Create chance node with two children
    auto chanceNode = std::make_unique<MCTSNode>(CHANCE);
    chanceNode->probabilities = {successProb, 1.0 - successProb};

    // Expand both outcomes
    chanceNode->outcomes.push_back(applySuccess(state, action));
    chanceNode->outcomes.push_back(applyFailure(state, action));

    return chanceNode;
} else {
    // Normal deterministic expansion
    return applyAction(state, action);
}
```

### Phase 2: Multi-Outcome Actions

Extend to actions with multiple outcomes (e.g., melee damage ranges):

1. Discretize continuous distributions into buckets
2. For melee/archery, use 3-5 representative damage values (min, low, avg, high, max)
3. Compute probabilities for each bucket
4. Create chance nodes with multiple children

**Example: Melee Attack**
```cpp
// Instead of sampling full damage distribution,
// use representative values
struct DamageBucket {
    int damageValue;  // Representative damage
    double probability;  // Probability of this range
};

// For a melee attack that can deal 10-20 damage
std::vector<DamageBucket> buckets = {
    {10, 0.1},  // Min damage (unlucky)
    {13, 0.2},  // Low damage
    {15, 0.4},  // Average damage
    {17, 0.2},  // High damage
    {20, 0.1}   // Max damage (lucky)
};
```

### Phase 3: Optimization

Once chance nodes work correctly:

1. Add transposition table support for chance nodes
2. Optimize memory layout
3. Consider progressive widening (start with 2 outcomes, expand to more if visited often)
4. Profile and tune

## Design Decisions

### How to Represent Outcomes?

**Option A: Explicit state copies**
```cpp
struct ChanceNode {
    std::vector<std::unique_ptr<MCTSGameState>> outcomeStates;
    std::vector<double> probabilities;
};
```

**Option B: Lazy evaluation**
```cpp
struct ChanceNode {
    MCTSGameState baseState;
    MCTSAction action;
    std::vector<int> outcomeRolls;  // Roll values for each outcome
    std::vector<double> probabilities;

    // Compute state on-demand
    MCTSGameState getOutcome(size_t index) {
        return applyActionWithRoll(baseState, action, outcomeRolls[index]);
    }
};
```

**Recommendation:** Option B - lazy evaluation. Only materialize states when visited.

### How Many Outcomes per Action?

**Binary actions (START_FIRE, etc.):**
- Exactly 2 outcomes (success/fail)
- Use exact probabilities from `GetOddsPercentile()`

**Damage actions (MELEE, ARCHERY):**
- Start with 3 outcomes (low/med/high)
- Can expand to 5 if needed for accuracy
- Use representative rolls: 10th, 50th, 90th percentile

**Complex actions (METEOR):**
- Consider 2-3 outcomes initially
- Can model as "hits N enemies" for N in {0, 1, 2, 3+}

### How to Handle Transposition Table?

**Challenge:** Same state can be reached via different chance outcomes

**Solution:**
- Hash based on game state only (not the path taken)
- When looking up, return cached evaluation if state matches
- This is already how transposition tables work!

```cpp
// Current approach works fine:
auto hash = computeHash(gameState);  // Doesn't include how we got here
if (auto cached = transpositionTable.lookup(hash)) {
    return cached->value;
}
```

### Selection at Chance Nodes

**During tree traversal:**
```cpp
size_t selectOutcome(const ChanceNode& node) {
    // Option 1: Sample by probability (introduces variance)
    double r = random();
    double cumulative = 0.0;
    for (size_t i = 0; i < node.probabilities.size(); i++) {
        cumulative += node.probabilities[i];
        if (r < cumulative) return i;
    }

    // Option 2: Round-robin weighted by visit count vs probability
    // (Explore under-visited outcomes more)
    size_t leastVisited = findMostUnderExploredOutcome(node);
    return leastVisited;
}
```

**Recommendation:** Use Option 2 to ensure all outcomes get explored proportionally.

## Integration Points

### Modified Functions

1. **`ShardokGameEngine::getLegalActions()`**
   - Add metadata about which actions need chance nodes
   - Return action + probability information

2. **`ShardokGameEngine::applyAction()`**
   - For binary actions, return both possible outcomes
   - Or: take an explicit outcome index parameter

3. **`AbstractMCTSAI::selection()`**
   - Handle chance nodes differently from decision nodes
   - Use probability-weighted selection instead of UCB

4. **`AbstractMCTSAI::expand()`**
   - Create chance node children for probabilistic actions
   - May create multiple child nodes per action

5. **`AbstractMCTSAI::backpropagate()`**
   - Update all nodes in path (both decision and chance)
   - Value calculation already handles this correctly (just averages)

### New Functions Needed

```cpp
// In ShardokGameEngine
struct ChanceOutcome {
    int roll;  // The dice roll that produces this outcome
    double probability;  // Probability of this outcome
};

std::vector<ChanceOutcome> getChanceOutcomes(const MCTSAction& action) const;
```

```cpp
// In MCTSNode
bool isChanceNode() const;
const std::vector<double>& getOutcomeProbabilities() const;
```

## Testing Strategy

### Unit Tests

1. **Binary action correctness**
   ```cpp
   TEST(ChanceNodes, BinaryActionExpectedValue) {
       // START_FIRE with 60% success
       // Run MCTS with chance nodes
       // Verify: visits to success ~= 60%, visits to failure ~= 40%
       // Verify: expected value matches manual calculation
   }
   ```

2. **Comparison with iterative deepening**
   ```cpp
   TEST(ChanceNodes, MatchesIterativeDeepening) {
       // Same position, both AIs
       // Should choose same action
       // Scores should be similar (within variance)
   }
   ```

3. **Transposition table with chance**
   ```cpp
   TEST(ChanceNodes, TranspositionConsistency) {
       // Two paths to same state via different chance outcomes
       // Should reuse cached evaluation
   }
   ```

### Integration Tests

1. Compare MCTS with/without chance nodes on test positions
2. Verify that chance nodes reduce overvaluation of marginal actions
3. Performance test: measure slowdown (expect 1.5-2x for binary actions)

### Real-World Validation

Run the problematic START_FIRE scenario:
- With current MCTS: Should overvalue START_FIRE
- With chance nodes: Should correctly weight success/failure
- Expected: END_TURN should get significantly more visits

## Performance Considerations

### Memory Overhead

**Per chance node:**
- Probability vector: `N * sizeof(double)` (N = number of outcomes)
- Outcome children: `N * sizeof(unique_ptr)`
- For binary: ~32 bytes per chance node

**Estimate:**
- Current tree: ~100K nodes per search
- With chance nodes: ~150K nodes (50% actions are probabilistic)
- Extra memory: ~50K * 32 bytes = ~1.6 MB
- **Acceptable overhead**

### Computational Overhead

**Per simulation:**
- Current: 1 path through tree
- With chance nodes: Still 1 path, but more nodes
- Overhead: ~20-30% (more node visits)

**Mitigation:**
- Transposition table helps (same states via different paths)
- Progressive widening (start with 2 outcomes, expand if visited often)
- Lazy state evaluation (don't materialize until needed)

### Convergence Speed

Chance nodes may require more visits to converge because:
- More children per action (branching factor increases)
- Outcomes need proportional exploration

**Mitigation:**
- Use visit count thresholds before expanding chance nodes
- Consider progressive widening (UCT-ProgressiveWidening)

## Migration Path

### Step 1: Infrastructure (1-2 days)
- Add `NodeType` enum and metadata to MCTSNode
- Implement chance node creation (without using them yet)
- Add unit tests for chance node structure

### Step 2: Binary Actions (2-3 days)
- Identify all binary success/fail actions
- Modify expansion to create chance nodes for these
- Update selection/backpropagation
- Test on START_FIRE scenario

### Step 3: Integration Testing (1 day)
- Run full MCTS tests with chance nodes enabled
- Compare with iterative deepening on test positions
- Validate that it fixes the START_FIRE overvaluation

### Step 4: Multi-Outcome Actions (2-3 days)
- Implement damage bucketing for MELEE/ARCHERY
- Create chance nodes with 3-5 outcomes
- Test on combat scenarios

### Step 5: Optimization (1-2 days)
- Profile performance
- Add progressive widening if needed
- Tune outcome granularity

### Step 6: Documentation & Cleanup (1 day)
- Document the new approach
- Clean up code
- Add comprehensive tests

## Alternative: Simpler Hybrid Approach

If full chance nodes are too complex, consider a hybrid:

1. **Keep current tree structure** (no explicit chance nodes)
2. **During expansion:** Sample outcome and create one child
3. **Over many simulations:** Statistics converge to correct probabilities
4. **Add outcome tracking:** Store "which outcome" in edge/node metadata

**Example:**
```cpp
// Expansion samples an outcome
auto expand(state, action) {
    if (action.hasBinaryOutcome()) {
        // Sample once
        bool success = (random() < successProb);
        // Store which outcome this edge represents
        edge.metadata.outcome = success ? OUTCOME_SUCCESS : OUTCOME_FAILURE;
        return applyWithOutcome(state, action, success);
    }
}

// Selection prioritizes under-explored outcomes
auto selectChild(node) {
    // Find action where outcome distribution is unbalanced
    // E.g., 60% success action should have ~60% success children
    // If we have 80% success children, prefer exploring failure
}
```

This is simpler but less theoretically sound. It's a reasonable starting point if full chance nodes prove too complex.

## Comparison: Chance Nodes vs Open-Loop MCTS

### What is Open-Loop MCTS?

**Open-loop MCTS** (also called "determinization MCTS" or "information set MCTS") is an alternative approach to handling randomness:

1. At the **start of each simulation**, sample all random outcomes needed for that simulation
2. Play out the entire simulation using those fixed random values
3. Different simulations use different random seeds
4. The tree structure doesn't explicitly model randomness - it's all in the rollouts

**Example implementation:**
```cpp
// At start of simulation
std::vector<double> rollSequence = sampleRolls(maxDepth);  // Pre-sample all rolls

// During simulation
MCTSNode* node = root;
for (int depth = 0; depth < maxDepth; depth++) {
    Action action = selectAction(node);
    node = applyAction(node, action, rollSequence[depth]);  // Use pre-sampled roll
}
```

### Open-Loop MCTS for Shardok

**How it would work:**
```cpp
// Each simulation samples a "possible world"
void simulate(MCTSNode* root) {
    // Sample random rolls for this simulation
    auto rolls = generateRollSequence();  // e.g., {0.45, 0.78, 0.23, ...}

    // Play out simulation using these fixed rolls
    auto state = root->state;
    for (int depth = 0; depth < maxDepth; depth++) {
        auto action = selectAction(state);
        state = applyAction(state, action, rolls[depth]);
    }

    double reward = evaluate(state);
    backpropagate(root, reward);
}
```

**Would this fix the START_FIRE issue?**

**Yes** - partially. Different simulations would see different outcomes:
- Some simulations: START_FIRE succeeds (roll < 0.51)
- Some simulations: START_FIRE fails (roll >= 0.51)
- Over many simulations, the action's value would approach the expected value

**However**, it's less efficient than chance nodes because:
- Needs MORE simulations to converge
- Wastes effort exploring unlikely scenarios equally with likely ones
- Doesn't explicitly guide exploration based on probability

### Detailed Comparison

| Aspect | Chance Nodes (Closed-Loop) | Open-Loop MCTS | Current (Fixed Roll) |
|--------|---------------------------|----------------|----------------------|
| **Randomness Handling** | Explicit in tree structure | Implicit in simulation sampling | Fixed roll=50 |
| **Convergence Speed** | Fast - probabilities guide search | Slower - needs more samples | N/A (wrong answer) |
| **Memory Usage** | Higher (more nodes) | Lower (no extra nodes) | Lowest |
| **Implementation Complexity** | High (tree structure changes) | Medium (sampling layer) | Low (current) |
| **Theoretical Soundness** | Highest (models true game tree) | Medium (approximation via sampling) | Low (assumes fixed outcome) |
| **START_FIRE Fix** | ✅ Yes, accurately | ✅ Yes, eventually | ❌ No |
| **Efficiency** | Most efficient per simulation | Less efficient (wasted samples) | Efficient but wrong |
| **Handles Hidden Information** | Poor | Excellent | N/A |

### When to Prefer Each Approach

**Prefer Chance Nodes when:**
- Randomness outcomes are discrete and enumerable (e.g., binary success/fail)
- Probabilities are known precisely
- You want fastest convergence to correct answer
- Game tree is the primary concern (no hidden information)
- **This is Shardok's situation** ✅

**Prefer Open-Loop when:**
- Randomness is continuous and high-dimensional
- Hidden information or imperfect information is present
- Simplicity is paramount
- You can afford many simulations
- Used in games like poker, bridge, Skat

### Why Chance Nodes are Better for Shardok

1. **Discrete outcomes**: Most Shardok randomness is binary (success/fail) or small discrete sets (damage ranges)
   - START_FIRE: 2 outcomes (success/fail)
   - MELEE: Can bucket into 3-5 damage ranges
   - Not continuous - perfect fit for chance nodes

2. **Known probabilities**: We have exact probabilities from `GetOddsPercentile()`
   - Chance nodes can use exact probabilities
   - Open-loop just samples blindly

3. **No hidden information**: Shardok is perfect information (all units visible to AI)
   - Chance nodes' main weakness doesn't apply
   - Open-loop's main strength doesn't help

4. **Convergence matters**: Limited simulation budget
   - Need to converge quickly
   - Chance nodes achieve this better

5. **Existing infrastructure**: We already have deterministic state transitions
   - Adding chance nodes builds on what we have
   - Open-loop would need different rollout structure

### Performance Analysis

**Chance Nodes:**
```
Time per simulation: 1.3x current
Simulations needed: 10,000 to converge
Total time: 13,000x units

Memory: 1.5x current (extra chance nodes)
```

**Open-Loop:**
```
Time per simulation: 1.0x current (same as now)
Simulations needed: 30,000 to converge (more variance)
Total time: 30,000x units

Memory: 1.0x current (no extra nodes)
```

**Result:** Chance nodes are **2.3x faster overall** despite being slower per simulation, because they converge with fewer simulations.

### Hybrid Approach: Best of Both Worlds?

Could we combine them?

**Idea:** Use chance nodes for high-probability branches, open-loop for rare events
```cpp
if (probability > 0.1 && outcomeCount <= 5) {
    // Use explicit chance node
    createChanceNode(outcomes, probabilities);
} else {
    // Use open-loop sampling
    sampleOutcome();
}
```

**Verdict:** Probably not worth the complexity. Shardok's randomness is simple enough that chance nodes handle everything well.

### Recommendation for Shardok

**Use Chance Nodes**, specifically:

1. **Phase 1:** Binary actions (START_FIRE, RAISE_DEAD, etc.)
   - 2 outcomes, exact probabilities
   - Biggest bang for buck

2. **Phase 2:** Damage ranges (MELEE, ARCHERY)
   - 3-5 buckets
   - Still manageable

3. **If needed:** Could fall back to open-loop for complex actions
   - E.g., METEOR with many possible outcomes
   - But likely unnecessary

### Why Not Open-Loop?

While open-loop would eventually fix the START_FIRE issue, it has significant downsides for Shardok:

1. **Slower convergence**: Needs 2-3x more simulations
2. **Doesn't leverage known probabilities**: We have exact odds, why ignore them?
3. **Less interpretable**: Harder to debug why AI chose an action
4. **Doesn't align with iterative deepening**: We want MCTS to match the proven algorithm

The only advantage of open-loop (simplicity) is outweighed by chance nodes' efficiency and correctness.

### Could We Use Current Approach + Better Sampling?

**Idea:** Keep fixed rolls but use different rolls per simulation?

```cpp
// Instead of always roll=50
double roll = random();  // Different each simulation
```

**Problem:** This is essentially open-loop without the tree!
- Even slower to converge
- Tree doesn't learn the outcome probabilities
- Worst of both worlds

**Verdict:** No, this doesn't help. If we're going to sample, do it properly (open-loop). Otherwise, use chance nodes.

### Final Verdict

**For Shardok, chance nodes are clearly superior:**

- ✅ Faster convergence (2-3x vs open-loop)
- ✅ Leverages exact probabilities
- ✅ Perfect fit for discrete outcomes
- ✅ Aligns with iterative deepening approach
- ✅ Better debuggability and interpretability
- ❌ More complex implementation (but manageable)

Open-loop would be a fallback if chance nodes prove too difficult, but given the benefits and the bounded complexity (only binary and small discrete outcomes), chance nodes are the right choice.

## Conclusion

Implementing chance nodes will fix the overvaluation of marginal probabilistic actions like START_FIRE with 51% success. The recommended approach is:

1. Start with **explicit chance nodes for binary actions**
2. Use **lazy state evaluation** to minimize memory
3. **Progressive enhancement** - binary first, then multi-outcome
4. Compare with iterative deepening to validate correctness

Expected benefits:
- More accurate action evaluation
- Better handling of probabilistic outcomes
- Closer alignment with theoretical MCTS
- Fixes the START_FIRE issue without tuning heuristics

Expected costs:
- ~20-30% slower per simulation (more nodes)
- ~1-2MB extra memory
- ~1-2 weeks development time

The benefits significantly outweigh the costs for a more theoretically sound and accurate AI.
