How this workload behaves
This is a multi-step agent loop where the LLM repeatedly processes input, generates intermediate outputs, and triggers further calls until a final answer is produced. Each loop iteration consumes input tokens (prompt + prior context) and generates output tokens (intermediate steps, tool calls, or final responses). Costs are dominated by output tokens since each step produces new text, and by the cumulative input tokens from repeated context accumulation. The high output cost is driven by the agent's chatty nature—frequent tool-use justifications, step-by-step reasoning, and verbose responses.
Compare against 6 alternatives
| Model | Vendor | Monthly cost | vs primary | Δ |
|---|---|---|---|---|
| GPT-4o | OpenAI | $45.00 | 0.28× | $115.00 |
| GPT-4o mini | OpenAI | $2.70 | 0.02× | $157.30 |
| GPT-3.5 Turbo | OpenAI | $8.00 | 0.05× | $152.00 |
| Mistral Small | Mistral | $3.20 | 0.02× | $156.80 |
| Claude Haiku 4.5 | Anthropic | $20.00 | 0.13× | $140.00 |
| Gemini 1.5 Flash | $1.35 | 0.01× | $158.65 |
Recommendation
Use **Gemini 1.5 Flash** as the primary model for this agent loop. It achieves the lowest total cost ($1.35 vs. $160 for GPT-4 Turbo) while maintaining sufficient reasoning capability for multi-step workflows. The 99.2% cost reduction ($158.65 saved monthly) is substantial, and its long context window (1M tokens) helps avoid truncation or excessive summarization overhead. While GPT-4o mini is cheaper ($2.70), the 2x cost difference is negligible compared to the potential intelligence gap in complex agent tasks.
Cost-saving tactics for this workload
- Implement **prompt caching** for repeated system prompts or static instructions across agent iterations to reduce input token costs by up to 90%.
- Use **structured outputs** (JSON/Tool Use) to minimize verbose intermediate steps, cutting output token usage by 30-50% per loop.
- Batch **parallelize independent agent steps** where possible to amortize context setup costs across multiple calls.
- Switch to **Gemini 1.5 Flash** for routing/lightweight steps and reserve GPT-4o mini for critical reasoning paths to balance cost and quality.
- Enable **automatic context compression** (e.g., summarizing prior steps) after 3-5 iterations to cap input token growth without losing coherence.