How this workload behaves
Long-document Q&A workloads are input-heavy due to the need to ingest large volumes of text (e.g., legal contracts, research papers, or technical manuals) before generating concise answers. The cost driver here is the input token volume, which scales linearly with document length, while output tokens remain relatively small per query. Retrieval-augmented generation (RAG) or agentic workflows that repeatedly reference the same documents exacerbate costs, as they re-process the same context across multiple interactions. Even with a 2M-token context window, inefficient prompt engineering or lack of context compression can inflate input costs disproportionately.
Compare against 5 alternatives
| Model | Vendor | Monthly cost | vs primary | Δ |
|---|---|---|---|---|
| Gemini 2.0 Flash | $1.35 | 0.06× | $21.15 | |
| Gemini 1.5 Flash | $1.35 | 0.06× | $21.15 | |
| Mistral Small | Mistral | $3.20 | 0.14× | $19.30 |
| GPT-4o mini | OpenAI | $2.70 | 0.12× | $19.80 |
| Claude Haiku 4.5 | Anthropic | $20.00 | 0.89× | $2.50 |
Recommendation
Use **Gemini 2.0 Flash** or **Gemini 1.5 Flash** for this workload. Both models deliver identical cost savings (94% reduction vs. Gemini 1.5 Pro) while maintaining strong performance on long-context tasks. The $21.15 monthly savings per user or endpoint is substantial, especially at scale, and the 2M-token context window ensures compatibility with large documents. The trade-off in intelligence is minimal for Q&A use cases, where precision and conciseness matter more than deep reasoning.
Cost-saving tactics for this workload
- Enable **prompt caching** for repeated queries over the same document to avoid re-processing identical context tokens.
- Use **context compression** (e.g., summarization or chunking) to reduce input token volume before sending to the LLM.
- Batch API calls for multiple queries to amortize fixed costs (e.g., process 10 Q&A pairs in a single request).
- Implement a **routing layer** with a smaller model (e.g., Mistral Small) for simple queries and escalate only complex or ambiguous cases to the primary model.
- Cache frequent answers locally to avoid redundant LLM calls for identical questions.