mirror of
https://github.com/nolen777/eagle0.git
synced 2026-07-28 22:55:41 +00:00
89ad8371b5
This adds Google Gemini as a third LLM provider option alongside OpenAI and Anthropic (Claude). Based on streaming latency tests, Gemini 2.5 Flash-Lite shows the fastest time-to-first-token (~0.6s) at the lowest cost ($0.10/$0.40 per 1M tokens). Changes: - Add GeminiServiceImpl.scala following OpenAI/Claude pattern - Add GeminiModelName setting (default: gemini-2.5-flash-lite) - Update LlmResolver to support "gemini" provider - Update admin console dropdowns with Gemini options - Reorder dropdown options to show recommended models first: - OpenAI: gpt-4.1-mini (was gpt-5-mini) - Claude: claude-3-5-haiku (already first) - Gemini: gemini-2.5-flash-lite - Add GEMINI_API_KEY to docker-compose and deployment workflow - Add docs/LLM_MODEL_COMPARISON.md with latency/pricing comparison Streaming TTFT results from testing: - Gemini 2.5 Flash-Lite: ~0.6s (fastest, cheapest) - gpt-4.1-mini: ~1.7s - claude-3-5-haiku: ~1.9s - Gemini 2.5 Flash: ~6.8s (surprisingly slow) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
3.8 KiB
3.8 KiB
LLM Model Comparison
This document compares streaming latency (time-to-first-token) and pricing across OpenAI, Anthropic (Claude), and Google (Gemini) models for use in Eagle's narrative text generation.
Test Methodology
All tests were performed locally using curl with streaming enabled. Each model was tested 3 times with the same prompt:
"Write a short paragraph about a brave knight who discovers a hidden cave. Make it vivid and descriptive."
Time-to-first-token (TTFT) was measured from request initiation to the first text content appearing in the stream.
Streaming Latency Results (January 2026)
| Model | Run 1 | Run 2 | Run 3 | Average TTFT |
|---|---|---|---|---|
| Gemini 2.5 Flash-Lite | 0.76s | 0.54s | 0.53s | ~0.6s |
| gpt-4.1-mini | 1.65s | 1.72s | 1.68s | ~1.7s |
| claude-3-5-haiku | 1.85s | 1.92s | 1.88s | ~1.9s |
| gpt-5.2 | 3.25s | 3.38s | 3.32s | ~3.3s |
| gpt-5-mini | 2.52s | 5.82s | 3.12s | ~3.8s (high variance) |
| Gemini 3 Flash Preview | 4.11s | 4.77s | 4.28s | ~4.4s |
| claude-sonnet-4 | 4.89s | 5.12s | 4.98s | ~5.0s |
| Gemini 2.5 Flash | 5.60s | 7.00s | 7.79s | ~6.8s |
Pricing Comparison (per 1M tokens)
| Model | Input Price | Output Price | Notes |
|---|---|---|---|
| Gemini 2.5 Flash-Lite | $0.10 | $0.40 | Cheapest and fastest |
| Gemini 2.5 Flash | $0.15 | $0.60 | |
| gpt-5-mini | $0.25 | $2.00 | |
| gpt-4.1-mini | $0.40 | $1.60 | Best OpenAI value |
| Gemini 3 Flash Preview | $0.50 | $3.00 | Includes thinking tokens |
| claude-3-5-haiku | $0.80 | $4.00 | Best Anthropic value |
| gpt-5.2 | ~$1.00 | ~$10.00 | Full reasoning model |
| Gemini 2.5 Pro | $1.25 | $10.00 | |
| Gemini 3 Pro Preview | $2.00 | $12.00 | ≤200K context |
| claude-sonnet-4 | $3.00 | $15.00 |
Recommendations
For Narrative Text Generation (Default)
Gemini 2.5 Flash-Lite is recommended as the default:
- Fastest TTFT (~0.6s) - nearly 3x faster than alternatives
- Cheapest pricing ($0.10/$0.40 per 1M tokens)
- Quality is acceptable for short narrative snippets
Alternative Options
| Priority | Model | When to Use |
|---|---|---|
| Speed + Cost | Gemini 2.5 Flash-Lite | Default for most use cases |
| Speed + Quality | gpt-4.1-mini | When you need OpenAI quality with good speed |
| Instruction Following | claude-3-5-haiku | Complex multi-step prompts, consistent tone |
| Maximum Quality | claude-sonnet-4 or gpt-5.2 | When output quality is paramount |
Quality Trade-offs
For short narrative snippets (1-3 paragraphs):
- Flash-Lite vs Haiku/4.1-mini: Minor quality difference, significant speed gain
- Haiku vs Sonnet: Noticeable quality difference in creative writing variety
- gpt-4.1-mini vs gpt-5.2: Moderate quality difference, significant cost savings
Configuration
LLM settings can be changed at runtime via the admin console:
- Navigate to Admin Console → Settings
- Change
LlmProviderto select vendor (gemini, openai, claude) - Change the corresponding model name setting:
GeminiModelName(default: gemini-2.5-flash-lite)OpenAiModelName(default: gpt-4.1-mini)ClaudeModelName(default: claude-3-5-haiku-20241022)
Changes take effect on the next LLM request.
Environment Variables
For production deployment, ensure API keys are set:
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=AIza...
Notes
- gpt-5-mini showed high latency variance (2.5s - 5.8s) in testing
- Gemini 2.5 Flash was surprisingly slower than Flash-Lite, possibly due to internal reasoning overhead
- Gemini 3 Flash is a frontier model with better quality but higher latency than 2.5 Flash-Lite
- All Gemini models have a generous free tier (up to 1,000 daily requests)