# LLM Model Comparison

This document compares streaming latency (time-to-first-token) and pricing across OpenAI, Anthropic (Claude), and Google (Gemini) models for use in Eagle's narrative text generation.

## Test Methodology

All tests were performed locally using curl with streaming enabled. Each model was tested 3 times with the same prompt:

> "Write a short paragraph about a brave knight who discovers a hidden cave. Make it vivid and descriptive."

Time-to-first-token (TTFT) was measured from request initiation to the first text content appearing in the stream.

## Streaming Latency Results (January 2026)

| Model | Run 1 | Run 2 | Run 3 | Average TTFT |
|-------|-------|-------|-------|--------------|
| **Gemini 2.5 Flash-Lite** | 0.76s | 0.54s | 0.53s | **~0.6s** |
| **gpt-4.1-mini** | 1.65s | 1.72s | 1.68s | **~1.7s** |
| **claude-3-5-haiku** | 1.85s | 1.92s | 1.88s | **~1.9s** |
| gpt-5.2 | 3.25s | 3.38s | 3.32s | **~3.3s** |
| gpt-5-mini | 2.52s | 5.82s | 3.12s | **~3.8s** (high variance) |
| Gemini 3 Flash Preview | 4.11s | 4.77s | 4.28s | **~4.4s** |
| claude-sonnet-4 | 4.89s | 5.12s | 4.98s | **~5.0s** |
| Gemini 2.5 Flash | 5.60s | 7.00s | 7.79s | **~6.8s** |

## Pricing Comparison (per 1M tokens)

| Model | Input Price | Output Price | Notes |
|-------|-------------|--------------|-------|
| **Gemini 2.5 Flash-Lite** | $0.10 | $0.40 | Cheapest and fastest |
| Gemini 2.5 Flash | $0.15 | $0.60 | |
| gpt-5-mini | $0.25 | $2.00 | |
| **gpt-4.1-mini** | $0.40 | $1.60 | Best OpenAI value |
| Gemini 3 Flash Preview | $0.50 | $3.00 | Includes thinking tokens |
| **claude-3-5-haiku** | $0.80 | $4.00 | Best Anthropic value |
| gpt-5.2 | ~$1.00 | ~$10.00 | Full reasoning model |
| Gemini 2.5 Pro | $1.25 | $10.00 | |
| Gemini 3 Pro Preview | $2.00 | $12.00 | ≤200K context |
| claude-sonnet-4 | $3.00 | $15.00 | |

## Recommendations

### For Narrative Text Generation (Default)

**Gemini 3.1 Flash-Lite** is recommended as the default:
- Priced at $0.25/$1.50 per 1M input/output tokens
- Significantly faster than 2.5 Flash-Lite on throughput and TTFT
- Meaningfully smarter than 2.5 Flash-Lite, while remaining one of the cheapest options
- Quality is more than sufficient for short narrative snippets

### Alternative Options

| Priority | Model | When to Use |
|----------|-------|-------------|
| Speed + Cost | Gemini 3.1 Flash-Lite | Default for most use cases |
| Speed + Quality | gpt-4.1-mini | When you need OpenAI quality with good speed |
| Instruction Following | claude-3-5-haiku | Complex multi-step prompts, consistent tone |
| Maximum Quality | claude-sonnet-4 or gpt-5.2 | When output quality is paramount |

### Quality Trade-offs

For short narrative snippets (1-3 paragraphs):
- **Flash-Lite vs Haiku/4.1-mini**: Minor quality difference, significant speed gain
- **Haiku vs Sonnet**: Noticeable quality difference in creative writing variety
- **gpt-4.1-mini vs gpt-5.2**: Moderate quality difference, significant cost savings

## Configuration

LLM settings can be changed at runtime via the admin console:

1. Navigate to Admin Console → Settings
2. Change `LlmProvider` to select vendor (gemini, openai, claude)
3. Change the corresponding model name setting:
   - `GeminiModelName` (default: gemini-3.1-flash-lite)
   - `OpenAiModelName` (default: gpt-4.1-mini)
   - `ClaudeModelName` (default: claude-3-5-haiku-20241022)

Changes take effect on the next LLM request.

## Environment Variables

For production deployment, ensure API keys are set:

```bash
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=AIza...
```

## Notes

- **gpt-5-mini** showed high latency variance (2.5s - 5.8s) in testing
- **Gemini 2.5 Flash** was surprisingly slower than Flash-Lite, possibly due to internal reasoning overhead
- **Gemini 3 Flash** is a frontier model with better quality but higher latency than 2.5 Flash-Lite
- All Gemini models have a generous free tier (up to 1,000 daily requests)