adminandGitHub
1215b5044c
* Remove retired Gemini fallback models * Prefer cheaper Gemini fallbacks first
3.9 KiB
LLM Model Comparison
This document compares streaming latency (time-to-first-token) and pricing across OpenAI, Anthropic (Claude), and Google (Gemini) models for use in Eagle's narrative text generation.
Test Methodology
All tests were performed locally using curl with streaming enabled. Each model was tested 3 times with the same prompt:
"Write a short paragraph about a brave knight who discovers a hidden cave. Make it vivid and descriptive."
Time-to-first-token (TTFT) was measured from request initiation to the first text content appearing in the stream.
Streaming Latency Results (January 2026)
| Model | Run 1 | Run 2 | Run 3 | Average TTFT |
|---|---|---|---|---|
| Gemini 2.5 Flash-Lite | 0.76s | 0.54s | 0.53s | ~0.6s |
| gpt-4.1-mini | 1.65s | 1.72s | 1.68s | ~1.7s |
| claude-3-5-haiku | 1.85s | 1.92s | 1.88s | ~1.9s |
| gpt-5.2 | 3.25s | 3.38s | 3.32s | ~3.3s |
| gpt-5-mini | 2.52s | 5.82s | 3.12s | ~3.8s (high variance) |
| Gemini 3 Flash Preview | 4.11s | 4.77s | 4.28s | ~4.4s |
| claude-sonnet-4 | 4.89s | 5.12s | 4.98s | ~5.0s |
| Gemini 2.5 Flash | 5.60s | 7.00s | 7.79s | ~6.8s |
Pricing Comparison (per 1M tokens)
| Model | Input Price | Output Price | Notes |
|---|---|---|---|
| Gemini 2.5 Flash-Lite | $0.10 | $0.40 | Cheapest and fastest |
| Gemini 2.5 Flash | $0.15 | $0.60 | |
| gpt-5-mini | $0.25 | $2.00 | |
| gpt-4.1-mini | $0.40 | $1.60 | Best OpenAI value |
| Gemini 3 Flash Preview | $0.50 | $3.00 | Includes thinking tokens |
| claude-3-5-haiku | $0.80 | $4.00 | Best Anthropic value |
| gpt-5.2 | ~$1.00 | ~$10.00 | Full reasoning model |
| Gemini 2.5 Pro | $1.25 | $10.00 | |
| Gemini 3 Pro Preview | $2.00 | $12.00 | ≤200K context |
| claude-sonnet-4 | $3.00 | $15.00 |
Recommendations
For Narrative Text Generation (Default)
Gemini 3.1 Flash-Lite is recommended as the default:
- Priced at $0.25/$1.50 per 1M input/output tokens
- Significantly faster than 2.5 Flash-Lite on throughput and TTFT
- Meaningfully smarter than 2.5 Flash-Lite, while remaining one of the cheapest options
- Quality is more than sufficient for short narrative snippets
Alternative Options
| Priority | Model | When to Use |
|---|---|---|
| Speed + Cost | Gemini 3.1 Flash-Lite | Default for most use cases |
| Speed + Quality | gpt-4.1-mini | When you need OpenAI quality with good speed |
| Instruction Following | claude-3-5-haiku | Complex multi-step prompts, consistent tone |
| Maximum Quality | claude-sonnet-4 or gpt-5.2 | When output quality is paramount |
Quality Trade-offs
For short narrative snippets (1-3 paragraphs):
- Flash-Lite vs Haiku/4.1-mini: Minor quality difference, significant speed gain
- Haiku vs Sonnet: Noticeable quality difference in creative writing variety
- gpt-4.1-mini vs gpt-5.2: Moderate quality difference, significant cost savings
Configuration
LLM settings can be changed at runtime via the admin console:
- Navigate to Admin Console → Settings
- Change
LlmProviderto select vendor (gemini, openai, claude) - Change the corresponding model name setting:
GeminiModelName(default: gemini-3.1-flash-lite)OpenAiModelName(default: gpt-4.1-mini)ClaudeModelName(default: claude-3-5-haiku-20241022)
Changes take effect on the next LLM request.
Environment Variables
For production deployment, ensure API keys are set:
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=AIza...
Notes
- gpt-5-mini showed high latency variance (2.5s - 5.8s) in testing
- Gemini 2.5 Flash was surprisingly slower than Flash-Lite, possibly due to internal reasoning overhead
- Gemini 3 Flash is a frontier model with better quality but higher latency than 2.5 Flash-Lite
- All Gemini models have a generous free tier (up to 1,000 daily requests)