mirror of
https://github.com/nolen777/eagle0.git
synced 2026-07-28 21:35:42 +00:00
adminandGitHub
1215b5044c
* Remove retired Gemini fallback models * Prefer cheaper Gemini fallbacks first
96 lines
3.9 KiB
Markdown
96 lines
3.9 KiB
Markdown
# LLM Model Comparison
|
|
|
|
This document compares streaming latency (time-to-first-token) and pricing across OpenAI, Anthropic (Claude), and Google (Gemini) models for use in Eagle's narrative text generation.
|
|
|
|
## Test Methodology
|
|
|
|
All tests were performed locally using curl with streaming enabled. Each model was tested 3 times with the same prompt:
|
|
|
|
> "Write a short paragraph about a brave knight who discovers a hidden cave. Make it vivid and descriptive."
|
|
|
|
Time-to-first-token (TTFT) was measured from request initiation to the first text content appearing in the stream.
|
|
|
|
## Streaming Latency Results (January 2026)
|
|
|
|
| Model | Run 1 | Run 2 | Run 3 | Average TTFT |
|
|
|-------|-------|-------|-------|--------------|
|
|
| **Gemini 2.5 Flash-Lite** | 0.76s | 0.54s | 0.53s | **~0.6s** |
|
|
| **gpt-4.1-mini** | 1.65s | 1.72s | 1.68s | **~1.7s** |
|
|
| **claude-3-5-haiku** | 1.85s | 1.92s | 1.88s | **~1.9s** |
|
|
| gpt-5.2 | 3.25s | 3.38s | 3.32s | **~3.3s** |
|
|
| gpt-5-mini | 2.52s | 5.82s | 3.12s | **~3.8s** (high variance) |
|
|
| Gemini 3 Flash Preview | 4.11s | 4.77s | 4.28s | **~4.4s** |
|
|
| claude-sonnet-4 | 4.89s | 5.12s | 4.98s | **~5.0s** |
|
|
| Gemini 2.5 Flash | 5.60s | 7.00s | 7.79s | **~6.8s** |
|
|
|
|
## Pricing Comparison (per 1M tokens)
|
|
|
|
| Model | Input Price | Output Price | Notes |
|
|
|-------|-------------|--------------|-------|
|
|
| **Gemini 2.5 Flash-Lite** | $0.10 | $0.40 | Cheapest and fastest |
|
|
| Gemini 2.5 Flash | $0.15 | $0.60 | |
|
|
| gpt-5-mini | $0.25 | $2.00 | |
|
|
| **gpt-4.1-mini** | $0.40 | $1.60 | Best OpenAI value |
|
|
| Gemini 3 Flash Preview | $0.50 | $3.00 | Includes thinking tokens |
|
|
| **claude-3-5-haiku** | $0.80 | $4.00 | Best Anthropic value |
|
|
| gpt-5.2 | ~$1.00 | ~$10.00 | Full reasoning model |
|
|
| Gemini 2.5 Pro | $1.25 | $10.00 | |
|
|
| Gemini 3 Pro Preview | $2.00 | $12.00 | ≤200K context |
|
|
| claude-sonnet-4 | $3.00 | $15.00 | |
|
|
|
|
## Recommendations
|
|
|
|
### For Narrative Text Generation (Default)
|
|
|
|
**Gemini 3.1 Flash-Lite** is recommended as the default:
|
|
- Priced at $0.25/$1.50 per 1M input/output tokens
|
|
- Significantly faster than 2.5 Flash-Lite on throughput and TTFT
|
|
- Meaningfully smarter than 2.5 Flash-Lite, while remaining one of the cheapest options
|
|
- Quality is more than sufficient for short narrative snippets
|
|
|
|
### Alternative Options
|
|
|
|
| Priority | Model | When to Use |
|
|
|----------|-------|-------------|
|
|
| Speed + Cost | Gemini 3.1 Flash-Lite | Default for most use cases |
|
|
| Speed + Quality | gpt-4.1-mini | When you need OpenAI quality with good speed |
|
|
| Instruction Following | claude-3-5-haiku | Complex multi-step prompts, consistent tone |
|
|
| Maximum Quality | claude-sonnet-4 or gpt-5.2 | When output quality is paramount |
|
|
|
|
### Quality Trade-offs
|
|
|
|
For short narrative snippets (1-3 paragraphs):
|
|
- **Flash-Lite vs Haiku/4.1-mini**: Minor quality difference, significant speed gain
|
|
- **Haiku vs Sonnet**: Noticeable quality difference in creative writing variety
|
|
- **gpt-4.1-mini vs gpt-5.2**: Moderate quality difference, significant cost savings
|
|
|
|
## Configuration
|
|
|
|
LLM settings can be changed at runtime via the admin console:
|
|
|
|
1. Navigate to Admin Console → Settings
|
|
2. Change `LlmProvider` to select vendor (gemini, openai, claude)
|
|
3. Change the corresponding model name setting:
|
|
- `GeminiModelName` (default: gemini-3.1-flash-lite)
|
|
- `OpenAiModelName` (default: gpt-4.1-mini)
|
|
- `ClaudeModelName` (default: claude-3-5-haiku-20241022)
|
|
|
|
Changes take effect on the next LLM request.
|
|
|
|
## Environment Variables
|
|
|
|
For production deployment, ensure API keys are set:
|
|
|
|
```bash
|
|
OPENAI_API_KEY=sk-...
|
|
ANTHROPIC_API_KEY=sk-ant-...
|
|
GEMINI_API_KEY=AIza...
|
|
```
|
|
|
|
## Notes
|
|
|
|
- **gpt-5-mini** showed high latency variance (2.5s - 5.8s) in testing
|
|
- **Gemini 2.5 Flash** was surprisingly slower than Flash-Lite, possibly due to internal reasoning overhead
|
|
- **Gemini 3 Flash** is a frontier model with better quality but higher latency than 2.5 Flash-Lite
|
|
- All Gemini models have a generous free tier (up to 1,000 daily requests)
|