CheapestInference | Blog

DeepSeek V3.2 vs Claude Opus for coding: when to use which

Wed, 15 Apr 2026 00:00:00 GMT

The question isn’t which model is “better” at coding. It’s which model is better for the coding task you’re doing right now.

Claude Opus 4.6 is the highest-scoring model on most coding benchmarks. DeepSeek V3.2 costs 55x less. The quality gap is real but narrow — and for many tasks, it doesn’t matter.

We ran both models through five categories of coding tasks and measured quality, speed, and cost. Here’s what we found.

Benchmark scores

Benchmark	Claude Opus 4.6	DeepSeek V3.2	Gap
SWE-bench Verified	72.5%	68.2%	-4.3
HumanEval+	93.2%	91.8%	-1.4
LiveCodeBench (Q1 2026)	48.5%	43.1%	-5.4
Aider polyglot	68.1%	65.3%	-2.8

Opus wins every benchmark. But the gap ranges from 1.4 to 5.4 points. The question is whether that gap justifies a 55x price difference.

Task-by-task comparison

Greenfield code generation

“Write an Express middleware that validates JWTs and attaches the user to the request.”

Both models produce correct, well-structured code. Opus tends to add more edge-case handling (expired tokens, malformed headers, missing claims). DeepSeek produces cleaner, shorter code that handles the happy path and common errors.

Winner: Opus by a small margin. The extra edge-case handling is genuinely useful. Does it justify 55x cost? No. A 2-minute code review catches what DeepSeek misses.

Debugging

“This test fails with ‘expected 3, got 4’. Here’s the test and the implementation.”

Both models identify the off-by-one error correctly. Opus explains the root cause more clearly and suggests a fix with a regression test. DeepSeek identifies and fixes the bug but doesn’t suggest the test.

Winner: Opus. Better explanations help prevent similar bugs. Does it justify 55x cost? For isolated bugs, no. For debugging sessions with complex context, maybe.

Refactoring

“Extract this 200-line function into smaller, testable functions.”

Opus excels here. It identifies logical boundaries, names functions well, maintains the original behavior, and adds type annotations. DeepSeek produces correct refactoring but sometimes picks awkward function boundaries or generic names.

Winner: Opus. Refactoring quality matters for maintainability. Does it justify 55x cost? For critical production code, yes. For internal tools, no.

Code review

“Review this PR for bugs, security issues, and style.”

Both models catch obvious bugs and security issues (SQL injection, missing auth checks). Opus catches more subtle issues — race conditions, edge cases in error handling, potential memory leaks. DeepSeek focuses on the most impactful issues and misses some subtle ones.

Winner: Opus, particularly for security-sensitive code. Does it justify 55x cost? For security reviews, yes. For routine PR reviews, no.

Boilerplate and scaffolding

“Create a CRUD API with Prisma, Express, and TypeScript for a blog platform.”

Both models produce identical-quality boilerplate. This is the category where the quality gap is zero. There’s no creative problem-solving involved — just pattern application.

Winner: Tie. Does it justify 55x cost? Absolutely not. Use the cheapest model available.

The cost math

For a developer using an AI coding assistant throughout the day:

Claude Opus (all tasks)

~$3,000/mo

Mixed (Opus + DeepSeek)

~$540/mo

DeepSeek V3.2 (all tasks)

~$53/mo

CheapestInference

from $39/mo

The “mixed” approach — using Opus for refactoring and security reviews, DeepSeek for everything else — captures 90% of Opus’s value at 18% of the cost.

The practical recommendation

Use Opus for:

Security-critical code reviews
Complex refactoring of production systems
Debugging subtle concurrency or memory issues
Architectural decisions that need thorough reasoning

Use DeepSeek V3.2 for:

Greenfield code generation
Boilerplate and scaffolding
Simple bug fixes
Test writing
Documentation generation
Any task where “correct” is sufficient and “polished” isn’t required

Use a small model (Llama 8B, Qwen 35B) for:

Code formatting
Simple find-and-replace refactoring
Generating repetitive test cases
Explaining code (reading comprehension, not generation)

The right model depends on the task, not on a blanket preference. A multi-model architecture that routes by task complexity gives you the best of both worlds.

Route by task through one API

The same task-routing logic applies to open-weight models. CheapestInference serves Kimi K2.6, GLM 5.2, and MiniMax M3 — all strong on coding — through a single OpenAI- and Anthropic-compatible endpoint, so you can pick the right model per task without juggling accounts:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.cheapestinference.com/v1",
    api_key="sk-your-key"
)

# Reach for the strongest model on the hard stuff
review = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[{"role": "user", "content": f"Review this PR for security issues:\n{diff}"}]
)

# A cheaper-to-run model for the routine work
code = client.chat.completions.create(
    model="glm-5.2",
    messages=[{"role": "user", "content": "Write a CRUD API for blog posts"}]
)

Same SDK, same key, different model per task. The routing decision is yours — or your agent’s.

CheapestInference serves Kimi K2.6, GLM 5.2, and MiniMax M3 through one OpenAI- and Anthropic-compatible API. Unlimited time-block subscriptions start at $39/month — reserve the hours you work and run without budget caps. Get started or see the models.

LLM API pricing in 2026: the complete comparison

Wed, 15 Apr 2026 00:00:00 GMT

LLM pricing changes every few weeks. A model that cost $60/M output tokens last year costs $10 today. New providers undercut each other constantly. This page is our attempt to keep a single, updated reference.

Last updated: April 2026.

Frontier models

The most capable models from each provider:

Model	Input $/M	Output $/M	Context
Claude Opus 4.6	$15.00	$75.00	200K
Claude Sonnet 4.6	$3.00	$15.00	200K
GPT-5.4	$2.50	$10.00	128K
Gemini 2.5 Pro	$1.25	$10.00	1M
DeepSeek V3.2	$0.27	$1.10	128K
Qwen 3.5 397B	$0.40	$1.20	128K
Mistral Large 3	$2.00	$6.00	128K

The price spread is 55x between the cheapest (DeepSeek V3.2) and most expensive (Claude Opus 4.6) frontier model. The quality spread on MMLU-Pro is 6.5 points. That’s the opportunity.

Cost-efficient models

The sweet spot — models that handle 80% of tasks at a fraction of frontier prices:

Model	Input $/M	Output $/M	Context
Claude Haiku 4.5	$0.80	$4.00	200K
GPT-4.1 mini	$0.40	$1.60	1M
Gemini 2.5 Flash	$0.15	$0.60	1M
Qwen 3.5 35B	$0.06	$0.12	128K
Llama 3.1 8B	$0.02	$0.05	128K

Llama 3.1 8B at $0.02/M input is 750x cheaper than Claude Opus. It won’t write your authentication system, but it’ll classify intents, extract entities, and route requests just fine.

Real workload cost comparison

Pricing per million tokens is hard to reason about. Here’s what actual workloads cost monthly:

Chatbot (50 conversations/day, ~2K tokens each)

Claude Opus 4.6

$270/mo

GPT-5.4

$100/mo

DeepSeek V3.2

$10/mo

CheapestInference

from $39/mo

Agent workload (20 tasks/day, ~500K tokens each)

Claude Opus 4.6

$5,508/mo

GPT-5.4

$2,838/mo

DeepSeek V3.2

$96/mo

CheapestInference

from $39/mo

The gap widens dramatically with agent workloads because context accumulation multiplies the per-token cost. Flat-rate pricing eliminates this entirely.

Per-token vs. flat-rate: when each makes sense

Per-token is better when:

Your usage is low and predictable (< $20/month)
You’re prototyping and don’t know your volume yet
You need a specific model not available on flat-rate platforms

Flat-rate is better when:

You run agents with unpredictable token consumption
Your monthly token bill exceeds the flat-rate plan cost
You want cost certainty for budgeting
You run multiple agents that need independent rate limits

The breakeven for flat-rate vs. per-token on DeepSeek V3.2 is roughly 40M tokens/month. An active agent does that in a week.

How to switch without rewriting code

Every provider listed in this article supports the OpenAI API format. Switching is a config change:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.cheapestinference.com/v1",  # or any provider
    api_key="sk-your-key"
)

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[{"role": "user", "content": "Hello"}]
)

Same SDK. Same methods. Same response format. Different price.

CheapestInference serves Kimi K2.6, GLM 5.2, and MiniMax M3 through a single OpenAI- and Anthropic-compatible API. Unlimited time-block subscriptions start at $39/month — reserve the hours you work and run without budget caps. Compare plans or see the models.

OpenAI API alternatives in 2026: price, speed, and quality compared

Wed, 15 Apr 2026 00:00:00 GMT

Every team that builds on GPT-5.4 eventually asks the same question: is there something cheaper that works just as well?

The answer is yes — but “cheaper” means different things depending on your workload. A chatbot that sends 50 messages/day has different economics than an agent framework burning 2M tokens per hour. This guide compares the real alternatives, with numbers.

What you’re actually paying for with OpenAI

OpenAI’s pricing for GPT-5.4:

Input: $2.50/M tokens
Output: $10.00/M tokens
Cached input: $1.25/M tokens

For a typical API integration doing 1M input + 200K output tokens per day, that’s $4.50/day or $135/month. For an agent workload doing 10M input + 1M output per day, it’s $35/day or $1,050/month.

The question isn’t whether GPT-5.4 is good. It is. The question is whether you need GPT-5.4 for every request.

The alternatives

1. Use a cheaper OpenAI model

Before switching providers, check if a smaller OpenAI model works:

Model	Input $/M	Output $/M	Quality (MMLU-Pro)
GPT-5.4	$2.50	$10.00	88.5%
GPT-4.1 mini	$0.40	$1.60	81.2%
GPT-4.1 nano	$0.10	$0.40	73.8%

GPT-4.1 mini is 6x cheaper than GPT-5.4 with a 7-point quality drop. For classification, extraction, and simple Q&A, that’s a good trade.

But if you need frontier quality at lower cost, you need to look beyond OpenAI.

2. Open-source models via inference providers

The real price disruption comes from open-source models. DeepSeek V3.2, Qwen 3.5, and Kimi K2.5 score within 4 points of GPT-5.4 on most benchmarks — at 5–50x less cost.

Provider	DeepSeek V3.2 Input	DeepSeek V3.2 Output	Models
DeepSeek (direct)	$0.27	$1.10	4
Together AI	$0.30	$0.90	100+
Fireworks	$0.20	$0.80	50+
Groq	$0.10	$0.30	15+
OpenRouter	varies	varies	200+
CheapestInference	flat-rate	flat-rate	3

All of these are OpenAI-compatible — change base_url and api_key, keep the rest of your code.

The hidden cost: per-token pricing on agent workloads

Per-token pricing works well for predictable workloads — chatbots, single-shot completions, classification. You can estimate monthly cost from your traffic.

It doesn’t work well for agents. Agent workloads have:

Unpredictable token consumption — a simple task might take 10 steps, a complex one might take 60
Context accumulation — each step re-sends everything, so cost grows quadratically with steps
Retry storms — errors trigger retries that consume tokens without producing output

We broke this down in detail in OpenClaw is free. Running it is not. The short version: a single OpenClaw task consumes ~525K tokens. On pay-per-token, that’s $0.16–$9.18 depending on the model.

On flat-rate, it’s included. Context accumulation, retries, and overhead don’t increase your bill.

Switching from OpenAI: what actually changes

If your code uses the OpenAI SDK, switching to any OpenAI-compatible provider is a two-line change:

from openai import OpenAI

# Before
client = OpenAI(api_key="sk-openai-...")

# After — any compatible provider
client = OpenAI(
    base_url="https://api.cheapestinference.com/v1",
    api_key="sk-your-key"
)

What stays the same:

client.chat.completions.create() — same API
Streaming — same stream=True pattern
Tool calling — same tools parameter
Response format — same JSON structure

What might change:

Model names — gpt-5.4 becomes deepseek/deepseek-chat-v3-0324 or qwen/qwen3.5-397b
Rate limits — each provider has different RPM/TPM limits
Latency — varies by provider and model size
Feature support — not all providers support vision, function calling, or JSON mode on all models

Test with your actual prompts before switching production traffic. Benchmarks measure general capability — your specific use case might have different results.

When to use which alternative

You need the highest quality and cost doesn’t matter: Stay with GPT-5.4 or Claude Opus 4.6 directly.

You want GPT-5.4 quality at lower cost: Use OpenRouter to access GPT-5.4 at discounted rates, or switch to open-weight models within a few points on most benchmarks — CheapestInference serves Kimi K2.6, GLM 5.2, and MiniMax M3 on flat-rate plans.

You run agents: Flat-rate pricing eliminates the unpredictability of agent workloads. You reserve time blocks and the agent runs unlimited during those hours, no token counting.

You need the fastest inference: Groq’s LPU hardware delivers the lowest latency for supported models. If your model is on Groq, it’s hard to beat on speed.

You want one API for everything: OpenRouter gives you access to multiple providers through a single endpoint with the largest catalog. If a few strong open-weight models cover your needs, CheapestInference offers flat-rate pricing on Kimi K2.6, GLM 5.2, and MiniMax M3.

The bottom line

OpenAI built the best developer experience in AI. But being the best product doesn’t mean being the best price. The API landscape in 2026 has enough competition that you can get 95% of the quality at 10–50% of the cost — or eliminate cost uncertainty entirely with flat-rate pricing.

The switch is two lines of code. The savings compound every month.

CheapestInference serves Kimi K2.6, GLM 5.2, and MiniMax M3 through a single OpenAI- and Anthropic-compatible endpoint. Unlimited time-block subscriptions start at $39/month — reserve 1–3 daily 8-hour blocks for unlimited usage during those hours. Get started or compare plans.

OpenRouter alternatives in 2026: unified LLM APIs compared

Wed, 15 Apr 2026 00:00:00 GMT

OpenRouter solved a real problem: one API key, hundreds of models, no separate accounts per provider. You point your code at openrouter.ai/api/v1 and pick any model from any provider.

But OpenRouter isn’t the only unified API anymore. And depending on your workload, it might not be the cheapest or fastest option. Here’s how the alternatives compare.

What OpenRouter does well

Credit where it’s due:

Model coverage: 200+ models from dozens of providers. If a model exists, OpenRouter probably has it.
Automatic routing: openrouter/auto picks a model for you based on your prompt. Useful for prototyping.
Fallback: If one provider is down, OpenRouter routes to another. You don’t handle failover yourself.
Single billing: One account, one API key, one invoice. No managing 8 provider accounts.

For developers who want access to everything and don’t want to manage multiple integrations, OpenRouter is a good default.

Where OpenRouter gets expensive

OpenRouter adds a margin on top of each provider’s per-token price. This is how they make money — they’re a reseller. The markup varies by model but is typically 5–20% above the direct provider price.

For low-volume usage, the convenience premium is negligible. For high-volume or agent workloads, it compounds:

Model	Direct price (input)	OpenRouter price	Markup
Claude Sonnet 4.6	$3.00/M	$3.00/M	0%
DeepSeek V3.2	$0.27/M	$0.30/M	+11%
Llama 3.1 70B	$0.13/M	$0.16/M	+23%
Qwen 3.5 397B	$0.40/M	$0.48/M	+20%

The markup is smallest on premium models (where the provider’s price already includes healthy margin) and largest on cheap open-source models (where OpenRouter’s fixed costs are a bigger percentage).

For an agent consuming 10M tokens/day on DeepSeek V3.2, the markup adds $9/month. Not a lot. But on a team of 10 with multiple agents each, it adds up — and the per-token model itself is the real problem for agent workloads.

The alternatives

Together AI

Best for: Fastest open-source model inference.

Together runs their own GPU clusters optimized for open-source models. No reselling — they serve the models directly. This means lower latency and often lower prices than OpenRouter for the same model.

100+ models
Own infrastructure (not reselling)
Competitive pricing on open-source models
Dedicated endpoints for production workloads
Per-token pricing only

Together doesn’t carry proprietary models (no Claude, no GPT). If you need Anthropic or OpenAI alongside open-source, you need a second integration.

Fireworks

Best for: Low-latency inference with custom model support.

Fireworks focuses on speed. Their custom serving infrastructure delivers lower latency than most providers, especially for open-source models. They also support fine-tuned model deployment.

50+ models
Very low latency
Fine-tuned model hosting
Serverless and dedicated options
Per-token pricing only

Like Together, Fireworks doesn’t carry proprietary models natively.

Groq

Best for: Absolute lowest latency.

Groq’s custom LPU hardware delivers the fastest inference in the market for supported models. If your use case is latency-sensitive (real-time chat, voice agents), Groq is hard to beat.

15+ models (smaller catalog)
Sub-second TTFT on most models
Free tier available
Per-token pricing

Limited model selection. No Claude, no GPT. But what they have is fast.

CheapestInference

Best for: Agent workloads and cost certainty.

Full disclosure — this is us. Here’s what we do differently:

Time-block subscriptions: Reserve one or more daily 8-hour blocks on a model pool — Asia-Pacific ($39/mo), Europe ($49/mo), or Americas ($45/mo). Reserve all three for full 24/7 coverage. From $39/month, annual ~15% off. No per-token billing.
Unlimited during your hours: During your reserved block, requests are unlimited with no budget cap — two concurrent requests per key. Pay by card (Stripe) or USDC on Base.
A focused lineup: Kimi K2.6, GLM 5.2, and MiniMax M3 — strong open-weight models through one endpoint.
x402 pay-per-request: No account needed — agents pay with USDC on Base L2 per request. Credit top-ups from $10 also available.

The trade-off: a small, curated model catalog instead of OpenRouter’s breadth, no proprietary models, and no automatic routing between providers.

Side-by-side comparison

	OpenRouter	Together	Fireworks	Groq	CheapestInf.
Models	200+	100+	50+	15+	3 (curated)
Proprietary models	Yes	No	No	No	No
Pricing model	Per-token	Per-token	Per-token	Per-token	Time-block flat-rate
Unlimited in reserved hours	No	No	No	No	Yes
Auto routing	Yes	No	No	No	No
API format	OpenAI	OpenAI	OpenAI	OpenAI	OpenAI

Every provider on this list is OpenAI-compatible. Switching between them is a base_url change.

Cost comparison for real workloads

Light usage (chatbot, ~3M tokens/month)

OpenRouter

$4.20/mo

Together AI

$3.60/mo

CheapestInference

from $39/mo

At low volume, per-token wins. A time-block subscription only pays off once your per-token spend during those hours would exceed the block price.

Heavy usage (agents, ~300M tokens/month)

OpenRouter

$420/mo

Together AI

$360/mo

CheapestInference

from $39/mo

At agent-scale volume, a time-block subscription is dramatically cheaper. The gap grows with usage because per-token scales linearly and a reserved block is unlimited — it doesn’t scale at all.

When to use what

Stay on OpenRouter if: You need access to 200+ models, use auto-routing, and your monthly spend is under $50. The convenience premium is worth it at this scale.

Switch to Together/Fireworks if: You only use open-source models, care about latency, and want to avoid the reseller markup. Together and Fireworks serve models directly.

Switch to CheapestInference if: You run agents during predictable hours, want cost certainty, and the curated open-weight lineup (Kimi K2.6, GLM 5.2, MiniMax M3) covers your needs. Unlimited inference during a reserved time block beats per-token billing once your usage in those hours is heavy.

Use Groq if: Latency is your primary constraint and your model is in their catalog.

All five are OpenAI-compatible. Try each one with a base_url swap and see which fits.

CheapestInference serves a curated open-weight lineup — Kimi K2.6, GLM 5.2, MiniMax M3 — through one OpenAI- and Anthropic-compatible API. Unlimited time-block subscriptions from $39/month. See the pools or get started.

Self-hosted vs. API inference: the real cost comparison

Wed, 15 Apr 2026 00:00:00 GMT

“Why pay for an API when I can run the model myself?”

It’s a reasonable question. Open-source models are free. GPUs are available on every cloud. vLLM and Ollama make serving straightforward. The math should be simple: GPU cost per hour × hours = total cost. Done.

Except it’s not. The GPU is the minority of the cost. Here’s the full picture.

The visible costs

GPU hardware

Running DeepSeek V3.2 (671B MoE, ~130B active parameters) requires at least 4× A100 80GB or 2× H100 80GB in FP8. Qwen 3.5 397B has similar requirements.

Setup	Hourly	Monthly (24/7)	Monthly (8h/day)
4× A100 80GB (cloud)	$12.80	$9,216	$2,816
2× H100 80GB (cloud)	$8.40	$6,048	$1,848
1× A100 80GB (Llama 70B)	$3.20	$2,304	$704
1× L40S (Llama 8B)	$1.10	$792	$242

These are cloud GPU rental prices (AWS, GCP, Lambda Labs — varies by provider and availability). If you buy hardware, the upfront cost is $15K–$40K per GPU, amortized over 3–4 years, plus electricity, cooling, and data center costs.

Smaller models are cheaper — but limited

Running Llama 3.1 8B on a single L40S costs $242/month (8h/day). That’s competitive with API pricing. But 8B models can’t handle complex coding, multi-step reasoning, or nuanced analysis — the tasks where AI provides the most value.

The models worth self-hosting (70B+, MoE) require multi-GPU setups where the economics change dramatically.

The invisible costs

GPU rental is just the beginning.

1. Operations and maintenance

Someone has to:

Set up vLLM/TGI with optimal batch sizes, quantization, and memory allocation
Monitor GPU utilization and restart crashed processes
Update model weights when new versions release
Handle OOM errors, NCCL failures, and driver issues
Manage the serving infrastructure (load balancer, health checks, auto-scaling)

If this is a full-time DevOps engineer at $150K/year, that’s $12,500/month in labor. If it’s 20% of a senior engineer’s time, it’s $2,500/month. Either way, it’s more than the GPU.

2. Idle capacity

GPUs cost money whether they’re inferring or not. If your usage pattern is 8 hours of heavy use (work hours) and 16 hours of near-zero traffic, you’re paying for 24 hours and using 8.

Cloud spot instances help but introduce availability risk. Auto-scaling GPU clusters is possible but complex — model loading takes minutes, not seconds.

API pricing is purely usage-based. Zero requests = zero cost.

3. Multi-model overhead

Self-hosting one model is manageable. Self-hosting five models for different tasks — a coding model, a reasoning model, a fast classification model, an embedding model, and a vision model — requires either:

5 separate GPU instances (expensive)
Shared GPU with model swapping (slow — loading a 70B model takes 2–5 minutes)
A serving framework that handles multi-model routing (complex)

An API gives you access to many models through the same endpoint. No model loading, no GPU allocation, no routing logic.

4. Opportunity cost

Every hour your team spends on inference infrastructure is an hour not spent on your actual product. For startups, this is the most expensive cost of all — it doesn’t show up on any invoice.

Total cost of ownership

For a team of 5 developers running AI-assisted coding with a mix of DeepSeek V3.2 and smaller models:

Cost	Self-hosted	API (per-token)	API (time-block sub)
Compute/inference	$2,800	$265	$250
Ops/maintenance	$2,500	$0	$0
Idle waste (~60%)	$1,680	$0	$0
Total monthly	$6,980	$265	$250

Self-hosting costs 26x more for the same workload. The GPU is only 40% of the self-hosted cost — ops and idle waste are the majority.

When self-hosting makes sense

Self-hosting wins in specific scenarios:

Data sovereignty: If your data cannot leave your network — regulated industries, government, healthcare with strict compliance — self-hosting is the only option. No API provider can guarantee the data isolation you need.

Extreme scale: If you’re processing millions of requests per day and your GPUs are consistently at 80%+ utilization, the per-token math eventually favors owned hardware. This threshold is higher than most teams expect — typically $20K+/month in API spend before self-hosting breaks even.

Custom models: If you’ve fine-tuned a model and need to serve it, self-hosting or a dedicated inference provider (Fireworks, Together) is required. Most unified APIs don’t serve custom model weights.

Latency control: If you need guaranteed sub-100ms TTFT and your data center is co-located with your GPUs, self-hosting eliminates network hops.

For everyone else — startups, small teams, companies with variable usage patterns — the API is cheaper, faster to set up, and easier to maintain.

The migration path

Most teams don’t need to choose one forever. A practical approach:

Start with an API: Get your product working, validate demand, understand your usage patterns.
Optimize model selection: Use cheaper models for simple tasks, frontier models for hard tasks. Full guide: Multi-model architecture.
Evaluate self-hosting when: Your monthly API spend exceeds $10K, your GPU utilization would be >70%, and you have DevOps capacity to maintain it.
Hybrid: Self-host your high-volume models, use an API for long-tail models and overflow capacity.

The worst outcome is spending 3 months setting up GPU infrastructure before you’ve validated that anyone wants your product.

CheapestInference serves frontier open-weight models — Kimi K2.6, GLM 5.2, and MiniMax M3 — through a single API. No GPUs to manage, no idle costs, no ops burden. Reserve a daily 8-hour time block for unlimited usage from $39/mo (reserve all three for full 24/7), or pay as you go with credits from $10. Get started or see the pools.

The real cost of running AI agents in production

Wed, 15 Apr 2026 00:00:00 GMT

Chatbots are cheap. Agents are not.

A chatbot sends a user message, gets a response, displays it. Maybe 2,000 tokens per exchange. An agent reads files, calls tools, retries on errors, re-sends the entire conversation every step, and does this 20–60 times per task. Same API, completely different economics.

If you’re budgeting for AI agents the same way you budget for a chatbot, you’re underestimating by 10–50x.

Token consumption: chatbot vs. agent

We measured token consumption across three workload types, each running for one hour:

Coding agent (OpenClaw)

~2.1M tokens

Research agent (CrewAI)

~1.2M tokens

RAG chatbot

~200K tokens

Simple chatbot

~40K tokens

The coding agent consumed 52x more tokens than a simple chatbot in the same time period. And this is normal — the agent was doing useful work the entire time.

Why agents cost so much

Three architectural properties of agents make them expensive:

1. Context accumulation

Every agent step appends tool outputs to the conversation. The LLM re-processes the entire conversation on each step. If the agent reads a 3,000-token file at step 5, that file gets re-sent at steps 6, 7, 8… all the way to the end.

For a 40-step task, one file read costs: 3,000 tokens × 35 remaining steps = 105,000 tokens in re-transmission.

This is why agent token consumption grows quadratically, not linearly.

2. System prompt overhead

Agent frameworks use large system prompts — OpenClaw’s is ~9,600 tokens, CrewAI’s varies by agent configuration. This prompt is sent with every request. Over 40 steps, the system prompt alone costs 384,000 tokens.

3. Error retry loops

When a tool call fails, the agent retries. Each retry sends the full context plus the error message. Three retries on a 30K-token context wastes 90K tokens with no productive output.

Without a retry cap, this can run indefinitely — always bound agents with a retry cap and a maximum iteration count.

Monthly cost by model and framework

Assuming one developer running 15 agent tasks per day, 22 working days per month, ~500K tokens per task:

Model	Cost/task	Daily (×15)	Monthly
Claude Opus 4.6	$9.18	$137.70	$3,029
Claude Sonnet 4.6	$2.25	$33.75	$743
GPT-5.4	$4.73	$70.95	$1,561
DeepSeek V3.2	$0.16	$2.40	$53
Qwen 3.5 35B	$0.04	$0.60	$13
CheapestInference (full day)	—	—	from $39 flat

A team of 5 developers each running 15 tasks/day on Claude Opus spends $15,145/month. The same team on flat-rate via CheapestInference pays a fixed monthly subscription per seat (from $39 for a reserved daily time block) — no matter how many tokens those agents burn. That’s an order-of-magnitude reduction.

Four strategies to cut agent inference costs

1. Switch to open-source models

DeepSeek V3.2 and Qwen 3.5 score within 4 points of GPT-5.4 and Opus on most benchmarks. For coding tasks specifically, DeepSeek V3.2 matches Opus on HumanEval and SWE-bench. Full data: Open-source models are production-ready.

2. Route by task complexity

Not every agent step needs a frontier model. File reads, simple classifications, and formatting don’t need 685B parameters. Use a small model for easy steps and a large model for hard ones. Full guide: Building a multi-model architecture.

3. Give each agent its own key

Give each agent its own API key so one runaway agent can’t starve the others. On a time-block subscription each key gets unlimited usage during its reserved hours, so you isolate workloads without juggling per-token allocations.

4. Use flat-rate pricing

Per-token pricing penalizes the exact patterns agents use: large contexts, many steps, retries. Flat-rate pricing makes all of that free. During your reserved time blocks your agent can use the full context window and retry freely without increasing the bill — reserve all three blocks for 24/7 coverage.

The math that matters

Here’s the equation most teams miss:

Agent cost = tokens_per_step × steps × cost_per_token

Most optimization focuses on cost_per_token — switching to a cheaper model. But tokens_per_step grows with context (quadratic), and steps is unpredictable. Optimizing only one variable leaves the other two working against you.

Flat-rate pricing eliminates all three variables from your bill. The cost is the subscription. Period.

We serve Kimi K2.6, GLM 5.2, and MiniMax M3 with flat-rate, unlimited time-block subscriptions — no token counting, no budget caps during your reserved hours. Reserve 1–3 daily 8-hour blocks from $39/month and your agent’s token consumption never becomes your problem. Get started or see plans.

Qwen 3.5 vs GPT-5.4 vs Claude Opus 4.6 — same quality, fraction of the price

Thu, 26 Mar 2026 00:00:00 GMT

You asked for this. After our first benchmark post, the most requested model was Qwen 3.5. Here it is — 4 models across 5 metrics, same models in every chart:

Open-source: Qwen3.5-397B-A17B (flagship), Qwen3.5-35B-A3B (efficient) Proprietary: GPT-5.4, Claude Opus 4.6

Knowledge: MMLU-Pro (%)

GPT-5.4

88.5%

Qwen3.5 397B

87.8%

Qwen3.5 35B

85.3%

Claude Opus 4.6

82.0%

GPT-5.4 leads at 88.5%, but Qwen3.5-397B is 0.7 points behind — statistically noise. The 35B with only 3B active parameters scores 85.3%, beating Opus by 3 points. The total spread across all four models is just 6.5 points.

Qwen3.5-397B matches GPT-5.4 at 5x less cost. The 35B beats Opus at 23x less.

Reasoning: GPQA Diamond (%)

GPT-5.4

92.0%

Claude Opus 4.6

91.3%

Qwen3.5 397B

88.4%

Qwen3.5 35B

84.2%

Proprietary models lead on graduate-level reasoning. GPT-5.4 at 92% and Opus at 91.3% are strong. But Qwen3.5-397B at 88.4% is within 4 points — and costs $0.54/M vs $2.50 and $5.00. The 35B at 84.2% is still PhD-level performance for $0.22/M input.

Code: LiveCodeBench v6 (%)

GPT-5.4

84.0%

Qwen3.5 397B

83.6%

Claude Opus 4.6

76.0%

Qwen3.5 35B

74.6%

The 397B essentially ties GPT-5.4 on competitive coding — 0.4 points apart. Both beat Opus by 8+ points. The 35B at 74.6% is within 2 points of Opus, at 1/23rd the price.

For dedicated coding workloads, the open ecosystem also offers Qwen3-Coder-480B (SWE-bench Verified: 69.6%, comparable to Claude Sonnet 4).

Speed: output tokens per second

Qwen3.5 35B

178 t/s

Qwen3.5 397B

84 t/s

GPT-5.4

~78 t/s

Claude Opus 4.6

46 t/s

The 35B’s MoE architecture pays off — 178 tok/s is 2.3x faster than GPT-5.4 and 3.9x faster than Opus. Even the 397B flagship at 84 tok/s outpaces both proprietary models. This is what happens when only 3-17B parameters activate per token instead of the full model.

Speed data from Artificial Analysis. Actual speeds on our infrastructure may differ.

Price: input cost per million tokens

Qwen3.5 35B

$0.22

Qwen3.5 397B

$0.54

GPT-5.4

$2.50

Claude Opus 4.6

$5.00

This is the chart that matters. Opus costs 23x more than the 35B and 9x more than the 397B. GPT-5.4 costs 5x more than the 397B. The quality difference? Single-digit percentage points on every benchmark.

The full picture

Quality only — no price axis. GPT-5.4 (gray) has the largest shape. Opus (dashed) is strong on reasoning and code. The 397B (indigo) nearly overlaps GPT-5.4 on code and knowledge. The 35B (teal) pulls hard left on speed — 178 tok/s is 2.3x faster than anything else here. Price tells its own story in the chart above.

The scorecard

Metric	Winner	Qwen3.5 397B	GPT-5.4	Claude Opus 4.6	Gap (397B vs best)
Knowledge (MMLU-Pro)	GPT-5.4	87.8%	88.5%	82.0%	-0.7 pts
Reasoning (GPQA)	GPT-5.4	88.4%	92.0%	91.3%	-3.6 pts
Code (LiveCodeBench)	GPT-5.4	83.6%	84.0%	76.0%	-0.4 pts
Speed (tok/s)	Qwen3.5 397B	84 t/s	~78 t/s	46 t/s	1.1x faster
Price ($/M input)	Qwen3.5 397B	$0.54	$2.50	$5.00	4.6x cheaper

Same weight class, different price tag. The 397B trades 0.4–3.6 points on quality for 4.6x lower price and faster speed. It beats Opus on 4 out of 5 metrics outright.

Note: The Qwen3.5-35B-A3B ($0.22/M) scores 85.3% MMLU-Pro, 84.2% GPQA, 74.6% LiveCodeBench at 178 tok/s — beating Opus on knowledge and speed at 23x less cost. A different weight class, but worth considering if speed and price matter more than the last few quality points.

The real question: what are you paying for?

The quality gap between Qwen3.5-397B and GPT-5.4 is 0.7 points on knowledge, 0.4 points on code. The price gap is 4.6x.

Put it differently:

Model	MMLU-Pro	Cost per quality point
Qwen3.5 35B	85.3%	$0.003 per point per M tokens
Qwen3.5 397B	87.8%	$0.006 per point per M tokens
GPT-5.4	88.5%	$0.028 per point per M tokens
Claude Opus 4.6	82.0%	$0.061 per point per M tokens

Opus costs 20x more per quality point than the 35B — and scores lower. GPT-5.4 leads on quality but costs 5-10x more for single-digit advantages.

For most workloads, the last 3% of benchmark performance isn’t worth a 5x price increase. And for workloads where it is — the 397B gets you within 1 point of GPT-5.4 at a fraction of the cost.

Also worth knowing: specialized Qwen models

Beyond the general-purpose models, the Qwen family includes two notable specialists:

Qwen3-Coder-480B — SWE-bench Verified 69.6%, comparable to Claude Sonnet 4. Built for agentic coding.
Qwen3-235B-Thinking — Chain-of-thought reasoning specialist. When you need the model to show its work.

CheapestInference serves frontier open-weight models — Kimi K2.6, GLM 5.2, and MiniMax M3 — on unlimited time-block subscriptions from $39/mo, or pay-as-you-go credits from $10. See plans and try it →

Sources: Qwen3.5-397B Model Card · Qwen3.5-35B Model Card · Artificial Analysis Leaderboard · GPQA Diamond Leaderboard · OpenAI Pricing · Anthropic Pricing · LiveCodeBench Leaderboard

OpenClaw is free. Running it is not.

Tue, 24 Mar 2026 00:00:00 GMT

OpenClaw has 247,000 GitHub stars. It’s free, open-source, and runs locally. You install it, point it at an LLM, and it writes code, browses the web, queries databases, and executes files on your behalf.

The agent is free. The inference is not.

Every time OpenClaw calls a model, it re-sends the entire conversation history — every tool output, every file it read, every intermediate result. By iteration 20 of a typical task, the input context is 30,000+ tokens. By iteration 40, it’s past 100,000. And it sends this every single request.

This is not a bug. It’s how agents work. And it’s why running OpenClaw on pay-per-token APIs costs $300–600/month for active users — sometimes more.

Where the tokens go

We broke down token consumption for a typical OpenClaw coding task: “add authentication to an Express API.” The agent completed it in 38 tool calls.

Context accumulation

~280K tokens

System prompt (×38)

~156K tokens

Tool outputs (files, etc.)

~70K tokens

Agent output

~19K tokens

Total: ~525,000 tokens for a single task. The agent’s actual output — the code it wrote — was 19K tokens. The other 96% is overhead.

On Claude Opus at $15/M input + $75/M output, that single task costs $9.18. Run five tasks a day and you’re at $1,377/month.

On DeepSeek V3.2 via a pay-per-token provider at $0.27/M input + $1.10/M output, the same task costs $0.16. Better — but 20 tasks a day is still $96/month, and that’s one agent.

The three cost traps

Here’s the OpenClaw-specific version:

1. Context grows quadratically

OpenClaw reads files into context. If it reads a 2,000-token file at step 5, that file gets re-sent at steps 6, 7, 8… all the way to 38. That single file read costs 2,000 × 33 remaining steps = 66,000 tokens in re-transmission alone.

Users report session contexts at 56–58% of the 400K context window during normal use. This isn’t a failure mode — it’s the architecture working as designed.

2. System prompt is a fixed tax

OpenClaw’s system prompt is ~9,600 tokens. It gets sent with every request. Over 38 tool calls, that’s 365K tokens just in system prompts. You pay this whether the agent does useful work or not.

3. Wrong model for the job

OpenClaw defaults to a single model for everything. But not every tool call needs the same intelligence:

Reading a file and deciding what to edit? Llama 3.1 8B handles this at 200 tokens/sec.
Writing complex authentication logic? A frontier open-weight model like Kimi K2.6 is the right call.
Formatting a config file? Any 8B model is overkill but still cheaper than Opus.

We wrote a full guide on this pattern: Building a multi-model architecture. Routing agent requests to the right model can cut costs by 60–80% without reducing output quality.

The math on flat-rate vs. pay-per-token

Here’s the comparison for an OpenClaw user running ~20 tasks/day:

Provider	Cost/task	20 tasks/day	Monthly
Claude Opus (direct)	$9.18	$183.60	$5,508
GPT-5.4 (direct)	$4.73	$94.60	$2,838
DeepSeek V3.2 (per-token)	$0.16	$3.20	$96
CheapestInference	—	—	from $39/mo

Flat-rate means you don’t care about context accumulation. The 280K tokens of context overhead that makes pay-per-token expensive? Irrelevant. The system prompt tax? Doesn’t matter. Your agent can call models 24/7 and the bill is the same.

If you’re running OpenClaw, here’s the setup we see working best:

1. Use open-weight models. Frontier open-weight models like Kimi K2.6 and GLM 5.2 score within a few points of proprietary models on coding benchmarks (the data). The gap doesn’t justify a 50x cost difference.

2. Route by complexity. Don’t send file reads and simple decisions to the same model as complex code generation. A router model costs fractions of a cent per classification. Full guide: Multi-model architecture.

3. Reserve the hours you work. On CheapestInference you reserve one or more daily 8-hour time blocks (Asia-Pacific, Europe, Americas — pick 1–3, all three is full 24/7). During your reserved hours inference is unlimited with no budget cap. One API key per agent, two concurrent requests per key. Outside your window, requests return 429 until your block opens again.

4. Handle rate limits automatically. Time blocks mean your agent will hit 429s outside your reserved window — that’s expected. But OpenClaw kills the conversation when it gets a 429. The agent stops, and if you close the dashboard, that conversation is gone.

We built an OpenClaw plugin that fixes this: openclaw-ratelimit-retry. It hooks into agent_end, detects retriable 429s, parks the session on disk, and waits for the budget window to reset. Then it sends chat.send to the original session — resuming the conversation with its full transcript, as if you had typed a message.

openclaw plugins install @cheapestinference/openclaw-ratelimit-retry

~/.openclaw/config.yaml

plugins:
  ratelimit-retry:
    budgetWindowHours: 8    # matches your CheapestInference 8-hour time block
    maxRetryAttempts: 3     # give up after 3 consecutive 429s
    checkIntervalMinutes: 5 # check every 5 min for ready retries

The plugin is zero-dependency, persists across server restarts, deduplicates by session, and handles edge cases like sub-agents, queue overflow, and corrupted state files. If the retry itself hits a 429, it re-queues automatically. No tokens wasted on re-sending from scratch — the agent picks up exactly where it left off.

This turns budget caps from “your agent crashes” into “your agent naps and wakes up.” Set it up once and forget about it.

5. Consider unlimited time blocks. If your agent runs more than a few tasks per day, per-token pricing works against you. Every token of context overhead is money. With an unlimited time-block subscription, context overhead is free during your reserved hours — re-send the full window, let the agent work without a budget cap.

The irony

OpenClaw is free because the code runs on your machine. But the valuable part — the intelligence — runs on someone else’s GPUs. The agent framework is the cheap part. Inference is the expensive part.

Open-source models on flat-rate infrastructure flip this equation. The models are free. The inference is flat. The only variable cost left is your time.

Point your OpenClaw base_url at https://api.cheapestinference.com/v1 and find out what unconstrained agents actually cost: nothing more than you already budgeted.

Building a multi-model architecture: route requests to the right LLM

Thu, 19 Mar 2026 00:00:00 GMT

Using one model for everything is the simplest architecture. It’s also the most wasteful. A 685B-parameter reasoning model answering “what’s the weather?” is like hiring a PhD to sort mail.

This guide covers how to use a small, fast model to classify incoming requests and route them to the right specialist. The result: lower latency, lower cost, and often better quality — because each model handles what it’s actually good at.

The problem with single-model architectures

Most applications start with one model:

User request --> Large Model --> Response

This works, but every request — simple or complex — pays the same latency and cost penalty. When 60% of your traffic is simple classification, FAQ, or extraction, you’re burning expensive compute on tasks a small model handles equally well.

Llama 3.1 8B

~200 t/s

DeepSeek V3.2

~60 t/s

DeepSeek R1

~30 t/s

The gap between Llama 8B and R1 is nearly 7x in throughput. Routing simple requests to the small model saves that difference on every request.

The multi-model architecture

User request --> Router (Llama 8B) --> classify intent
                                          |
                  +-----------+-----------+-----------+
                  |           |           |           |
               simple      general    reasoning     code
                  |           |           |           |
            Llama 3.1 8B  DeepSeek   DeepSeek R1   Qwen3
                            V3.2                    Coder
                  |           |           |           |
                  +-----+-----+-----+-----+
                        |
                     Response

Two stages:

Classify — The router model reads the user’s message and outputs a category. A fast model returns this in a fraction of a second.
Route — Based on the category, forward the request to the appropriate specialist model.

The router adds minimal overhead (~200ms) but saves significant compute by keeping simple requests away from expensive models.

Step 1: Classify with a fast model

A fast, lightweight model makes a good router. With low TTFT and a short, single-word output, the classification step costs almost nothing and completes before the user notices.

The classification prompt is simple — you want a single-word category, not a conversation:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.cheapestinference.com/v1",
    api_key="your-api-key"
)

def classify_request(user_message: str) -> str:
    """Classify a user message into a routing category."""
    response = client.chat.completions.create(
        model="glm-5.2",
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the user's message into exactly one category. "
                    "Respond with only the category name, nothing else.\n\n"
                    "Categories:\n"
                    "- simple: greetings, FAQ, simple factual questions\n"
                    "- general: complex questions, analysis, writing, summarization\n"
                    "- reasoning: math, logic, multi-step problems, science\n"
                    "- code: code generation, debugging, refactoring, technical implementation\n"
                    "- agent: tasks requiring tool use, web search, or multi-step execution"
                )
            },
            {"role": "user", "content": user_message}
        ],
        max_tokens=10,
        temperature=0
    )
    category = response.choices[0].message.content.strip().lower()
    # Default to general if classification is unclear
    valid = {"simple", "general", "reasoning", "code", "agent"}
    return category if category in valid else "general"

The key details: max_tokens=10 because we only need one word. temperature=0 for deterministic routing. The system prompt is explicit about format — no preamble, just the category.

Step 2: Route to the specialist

Each category maps to a model optimized for that task:

# Model routing table
ROUTE_TABLE = {
    "simple":    "glm-5.2",
    "general":   "glm-5.2",
    "reasoning": "MiniMax-M3",
    "code":      "MiniMax-M3",
    "agent":     "kimi-k2.6",
}

def route_request(user_message: str, conversation_history: list) -> str:
    """Classify and route a request to the appropriate model."""
    category = classify_request(user_message)
    model = ROUTE_TABLE[category]

    response = client.chat.completions.create(
        model=model,
        messages=conversation_history + [
            {"role": "user", "content": user_message}
        ],
        stream=True
    )

    # Stream the response back
    full_response = ""
    for chunk in response:
        if chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            full_response += content
            print(content, end="", flush=True)

    return full_response

Notice that simple requests route back to GLM 5.2 — the same model that did the classification. For simple queries, the router overhead is effectively zero because the specialist is the same model and can reuse the warm connection.

Step 3: Handle edge cases

The basic router works for most traffic, but production systems need a few refinements:

def route_request_production(
    user_message: str,
    conversation_history: list,
    force_model: str = None
) -> tuple[str, str]:
    """Production router with overrides and fallback."""

    # Allow explicit model override (for power users or testing)
    if force_model:
        model = force_model
        category = "override"
    else:
        category = classify_request(user_message)
        model = ROUTE_TABLE[category]

    try:
        response = client.chat.completions.create(
            model=model,
            messages=conversation_history + [
                {"role": "user", "content": user_message}
            ]
        )
        return response.choices[0].message.content, category

    except Exception:
        # Fallback to GLM 5.2 if the specialist is unavailable
        fallback = "glm-5.2"
        response = client.chat.completions.create(
            model=fallback,
            messages=conversation_history + [
                {"role": "user", "content": user_message}
            ]
        )
        return response.choices[0].message.content, f"{category}->fallback"

Three patterns worth noting:

Force model — Let callers bypass routing when they know what they need.
Fallback — If a specialist model is down, fall back to GLM 5.2. It handles everything reasonably well.
Return the category — Log which route each request takes. You’ll need this data to tune the system.

Cost and latency comparison

Consider a workload of 1,000 requests with this distribution: 600 simple, 300 general, 70 reasoning, 30 code. Average 500 input tokens, 200 output tokens per request.

Single-model approach (everything on V3.2)

Avg latency

~4.5s

All 1000 reqs

V3.2 only

Every request waits for V3.2’s ~1.2s TTFT plus generation time at ~60 t/s. Simple questions get the same treatment as complex analysis.

Multi-model approach (routed)

Simple (600)

~1.2s (8B)

General (300)

~4.7s (V3.2)

Reasoning (70)

~9.0s (R1)

Code (30)

~3.5s (Coder)

The weighted average latency drops to approximately 2.7s — a 40% reduction. The 600 simple requests finish in ~1.2s instead of ~4.5s. That’s a 3.7x improvement for the majority of your traffic.

The 70 reasoning requests are slower individually (~9s vs ~4.5s) because R1 generates chain-of-thought tokens. But the quality on those specific requests is significantly better — R1 scores 50.2% on HLE versus V3.2’s 39.3%.

You get faster averages and better quality on the hard tail.

Real example: a support chatbot

A customer support chatbot receives three types of requests:

FAQ (60%) — “What are your business hours?” / “How do I reset my password?”
Complex support (30%) — “I was charged twice for order #12345, can you investigate?”
Technical issues (10%) — “Your API returns 500 when I send multipart form data with UTF-8 filenames”

Without routing

All requests go to DeepSeek V3.2. FAQs get correct answers but with unnecessary latency. Technical issues get decent answers but miss edge cases that a code-specialized model would catch.

With routing

SUPPORT_ROUTES = {
    "simple":    "glm-5.2",       # FAQ, greetings
    "general":   "glm-5.2",       # Complex support
    "reasoning": "glm-5.2",       # Investigations
    "code":      "glm-5.2",       # Technical issues
    "agent":     "kimi-k2.6",     # Multi-step resolution
}

FAQs resolve quickly via GLM 5.2. Complex support issues get GLM 5.2’s full analytical capability. Technical problems also route to GLM 5.2, which understands the code context well. If a support issue requires looking up order data via API, it routes to Kimi K2.6 for tool-assisted resolution.

The classification step adds ~200ms. For the 60% of requests that drop from ~4.5s to ~1.2s, that’s an invisible cost.

When NOT to use multi-model routing

Routing adds complexity. Skip it when:

All your requests are the same type. If you’re building a code editor, just use a single coding model like GLM 5.2. No routing needed.
You have fewer than 100 requests/day. The cost savings don’t justify the engineering overhead at low volume.
Latency doesn’t matter. For batch processing or async workloads, a single capable model is simpler.
Your classification accuracy is low. If the router misclassifies frequently, you get worse results than a single good model. Test the classifier on real traffic before deploying.

The sweet spot is high-volume applications with diverse request types — chatbots, API gateways, developer tools, and customer-facing products where response time directly affects user experience.

Implementation checklist

Log your traffic. Before building a router, understand your request distribution. What percentage is simple? Complex? Code?
Start with two tiers. A fast, lighter model for simple requests, and a stronger model like MiniMax M3 for everything that needs deep reasoning, code, or long context. Add specialists only when you have data showing they help.
Measure classification accuracy. Sample 100 requests, manually label them, compare against the router’s output. Target >90% accuracy.
Add fallback. Every specialist route should fall back to GLM 5.2 if the specialist is unavailable.
Monitor per-route metrics. Track latency, cost, and quality per category. This tells you where to optimize next.

All models in this guide are available through a single OpenAI-compatible API with no configuration changes between models. If you’re building a platform that needs LLM access for your users, see how per-key plans work.

Sources: Artificial Analysis Leaderboard · DeepSeek V3.2 · HLE Leaderboard · Kimi K2.5 Benchmarks

How to choose the right open-source model for your task

Thu, 19 Mar 2026 00:00:00 GMT

Most teams default to the biggest model available and call it a day. That works — until latency spikes, costs climb, and you realize a 8B-parameter model would have handled 60% of your requests just fine.

This guide maps common use cases to specific models, with real throughput numbers from our infrastructure. No theory — just which model to pick and why.

Quick decision table

Use case	Model	Why
General chat / assistants	DeepSeek V3.2	Best all-rounder. 85% MMLU-Pro, 73% SWE-bench, 60 t/s.
Complex reasoning	DeepSeek R1	50.2% on Humanity’s Last Exam. Chain-of-thought built in.
Code generation	Qwen3 Coder	Purpose-built for code. Strong on completions, refactoring, and debugging.
Agentic workflows	Kimi K2.5	334 t/s output, native tool use, 50.2% HLE with tools. Built for agents.
Vision / multimodal	Llama 4 Scout	17 active experts, 109B params, native image understanding.
Fast classification	Llama 3.1 8B	~200 t/s, 0.2s TTFT. Small enough for routing, tagging, extraction.
General (budget)	GLM 5.2	Fast inference, competitive quality. Good when V3.2 is overkill.
Long context chat	MiniMax M3	1M-token context window. Handles very large documents and codebases.
Large general + reasoning	Qwen3 235B	235B MoE. Strong across benchmarks when you need maximum capability.
Embeddings	BGE Large	MTEB-tested. Solid retrieval quality for RAG pipelines.

General chat and assistants

Pick: DeepSeek V3.2

DeepSeek V3.2 is the default choice for most workloads. It scores 85% on MMLU-Pro (beating Claude Opus 4.6’s 82%), 73% on SWE-bench Verified, and runs at ~60 tokens/second on our infrastructure.

Kimi K2.5

334 t/s

Llama 3.1 8B

~200 t/s

DeepSeek V3.2

~60 t/s

DeepSeek R1

~30 t/s

Good at: Broad knowledge, instruction following, multilingual, structured output. Not ideal for: Tasks that need step-by-step reasoning chains (use R1) or sub-100ms latency (use Llama 8B). Pick over alternatives when: You need a reliable general-purpose model that handles most tasks without specialization.

Complex reasoning

Pick: DeepSeek R1

R1 is a reasoning-first model. It produces explicit chain-of-thought tokens before its final answer. On Humanity’s Last Exam — a benchmark designed to be unsolvable by current models — R1 scores 50.2%, beating GPT-5.4 (41.6%) and Claude Opus 4.6 (40%).

The tradeoff is speed. At ~30 t/s, R1 is the slowest model in our lineup. That’s expected — it’s generating reasoning tokens that never appear in the final output.

Good at: Math, science, logic puzzles, multi-step problems, anything where “thinking” helps. Not ideal for: Simple Q&A, classification, or latency-sensitive applications. Pick over alternatives when: The task requires multi-step deduction. If a human would need to “think through it,” R1 will outperform faster models.

Code generation

Pick: Qwen3 Coder

Qwen3 Coder is purpose-built for software engineering tasks — code completion, refactoring, debugging, and generation across languages. It’s trained specifically on code-heavy data and optimized for developer workflows.

Good at: Code completion, bug fixing, refactoring, test generation, multi-file edits. Not ideal for: General conversation or non-code tasks (use V3.2). Pick over alternatives when: Code quality matters more than general knowledge. For mixed code-and-chat workflows, V3.2 or Kimi K2.5 may be more versatile.

Agentic workflows

Pick: Kimi K2.5

Kimi K2.5 was designed for agentic use. It has native tool-calling support, runs at 334 t/s (the fastest model we serve), and scores 50.2% on HLE when using tools — matching R1’s reasoning-only score.

The speed matters for agents. Each tool call is a round trip: the model generates a function call, the tool executes, the result goes back to the model. At 334 t/s and 0.31s TTFT, Kimi completes multi-step agent loops in seconds where slower models take minutes.

Good at: Tool use, function calling, multi-step task execution, fast iteration loops. Not ideal for: Pure reasoning without tools (R1 is better). Code-only tasks (Qwen3 Coder is more specialized). Pick over alternatives when: Your application involves tool calling, API interactions, or multi-step agent orchestration where speed compounds.

Vision and multimodal

Pick: Llama 4 Scout

Llama 4 Scout is Meta’s mixture-of-experts multimodal model — 109B total parameters with 17 active experts. It handles text and images natively, making it the pick for tasks that require visual understanding alongside language.

Good at: Image description, visual Q&A, document understanding, chart interpretation. Not ideal for: Text-only tasks where you’re paying for vision capability you don’t use (use V3.2). Pick over alternatives when: Your input includes images. For text-only workloads, other models are more efficient.

Fast classification and routing

Pick: Llama 3.1 8B

At 8 billion parameters, Llama 3.1 8B runs at ~200 t/s with approximately 0.2s time to first token. It’s the right choice for tasks where speed matters more than depth: intent classification, sentiment analysis, entity extraction, content filtering, and request routing.

Good at: Classification, tagging, extraction, routing decisions, simple Q&A, content moderation. Not ideal for: Complex reasoning, long-form generation, or tasks requiring deep world knowledge. Pick over alternatives when: You need results in under a second and the task is well-defined. Also ideal as the router model in a multi-model architecture.

Budget general use

Pick: GLM 5.2

GLM 5.2 delivers competitive quality at fast inference speeds. When DeepSeek V3.2 is more capability than you need — simple conversations, basic summarization, FAQ bots — GLM 5.2 gets the job done efficiently.

Good at: Simple chat, summarization, translation, basic Q&A. Not ideal for: Complex reasoning or tasks where benchmark-leading quality matters. Pick over alternatives when: You want good-enough quality with better speed and lower cost than the largest models.

Long context

Pick: MiniMax M3

MiniMax M3 ships a 1M-token (1,048,576) context window — the largest in our lineup. For workloads that involve ingesting large documents, long conversation histories, or extensive codebases, M3 maintains coherence across the full context. It’s a frontier multimodal coding, agentic, and reasoning model, so the quality holds up across that long context rather than degrading.

Good at: Document analysis, long conversations, large-context summarization, whole-repo code reasoning. Not ideal for: Short, simple tasks where context length is irrelevant and you’d rather pay less (use Llama 8B or GLM Flash). Pick over alternatives when: Your input regularly exceeds what smaller-context models handle well, or you need frontier reasoning over a very large context.

Maximum capability

Pick: Qwen3 235B

Qwen3 235B is a large mixture-of-experts model that competes across the full benchmark spectrum. When you need the highest possible quality and latency is not the primary constraint, Qwen3 235B delivers.

Good at: Broad capability across reasoning, knowledge, and generation. Strong multilingual support. Not ideal for: Latency-sensitive applications (large model, slower inference). Pick over alternatives when: You need top-tier quality and can tolerate higher latency. Good for batch processing and offline tasks.

Embeddings

Pick: BGE Large

BGE Large (BAAI General Embedding) is a well-tested embedding model for retrieval-augmented generation. It performs well on MTEB benchmarks and produces dense vectors suitable for semantic search, document retrieval, and clustering.

Good at: Semantic search, RAG pipelines, document similarity, clustering. Not ideal for: Generative tasks (it’s an embedding model, not a chat model). Pick over alternatives when: You need vector embeddings for search or retrieval. Pair it with a generative model for the full RAG pipeline.

The decision tree

What's your task?
|
+-- Need to understand images?
|   YES --> Llama 4 Scout
|
+-- Need step-by-step reasoning? (math, logic, science)
|   YES --> DeepSeek R1 (~30 t/s, but highest reasoning quality)
|
+-- Need tool calling / agent loops?
|   YES --> Kimi K2.5 (334 t/s, native tool use)
|
+-- Need code generation / editing?
|   YES --> Qwen3 Coder (purpose-built for code)
|
+-- Need embeddings for search/RAG?
|   YES --> BGE Large
|
+-- Need sub-200ms response?
|   YES --> Llama 3.1 8B (~200 t/s, 0.2s TTFT)
|
+-- Need long context (large documents)?
|   YES --> MiniMax M3 (1M-token context)
|
+-- Need maximum quality, latency flexible?
|   YES --> Qwen3 235B
|
+-- General purpose, good balance?
    YES --> DeepSeek V3.2 (default choice)

The 80/20 rule

You don’t need ten models to cover most workloads.

Llama 3.1 8B handles 60% of requests. Classification, routing, simple Q&A, extraction, content filtering. Fast and cheap.

DeepSeek V3.2 handles 30%. General chat, complex instructions, knowledge-intensive tasks. The reliable all-rounder.

Specialized models handle the last 10%. R1 for hard reasoning. Kimi K2.5 for agent loops. Qwen3 Coder for code. BGE Large for embeddings.

Start with Llama 8B + V3.2. Add specialists only when you have evidence that general models aren’t performing on specific task categories. Measure first, specialize second.

This guide is provider-agnostic. CheapestInference serves a focused lineup — Kimi K2.6, GLM 5.2, and MiniMax M3 — through a single OpenAI- and Anthropic-compatible API. If you want unlimited inference during your reserved hours, see how time-block pools work.

Sources: Artificial Analysis Leaderboard · SWE-bench Leaderboard · Kimi K2.5 Benchmarks · DeepSeek V3.2 · HLE Leaderboard · MMLU-Pro Leaderboard · MTEB Leaderboard

Open-source models are production-ready. Here's the proof.

Thu, 19 Mar 2026 00:00:00 GMT

There’s a persistent assumption in the industry: open-source models are fine for experimentation, but production workloads need GPT-5 or Claude Opus. We run open-source models in production every day. Here’s what the benchmarks actually say.

We’re comparing 5 models across 5 metrics — the same models in every chart, no cherry-picking:

Open-source: DeepSeek V3.2, DeepSeek R1, Kimi K2.5 Proprietary (reference): Claude Opus 4.6, GPT-5.4

(We serve the latest open-weight frontier models — currently Kimi K2.6, GLM 5.2, and MiniMax M3 — through a single API. The benchmark models below are industry reference points; the conclusions about open-source readiness carry directly to the current generation.)

Code quality: SWE-bench Verified (% resolved)

Claude Opus 4.6

80.8%

GPT-5.4

~80.0%

Kimi K2.5

76.8%

DeepSeek V3.2

73.0%

DeepSeek R1

57.6%

Proprietary models lead here. Opus 4.6 and GPT-5.4 are within a point of each other at ~80%. Kimi K2.5 is 4 points behind at 76.8% — competitive but not leading. R1 is a reasoning model, not optimized for code.

Reasoning: Humanity’s Last Exam (%)

Kimi K2.5 *

50.2%

DeepSeek R1

50.2%

GPT-5.4

41.6%

Claude Opus 4.6

40.0%

DeepSeek V3.2

39.3%

Open-source wins decisively. R1 hits 50.2% and Kimi K2.5 matches it with tool-use enabled (*without tools: 31.5%). Both beat Opus 4.6 (40%) and GPT-5.4 (41.6%). V3.2 is roughly at Opus level — it’s a general model, not a reasoning specialist.

*Kimi K2.5’s HLE score uses its agentic mode with tool access. This is how the model is designed to be used in production.

Knowledge: MMLU-Pro (%)

GPT-5.4

88.5%

Kimi K2.5

87.1%

DeepSeek V3.2

85.0%

DeepSeek R1

84.0%

Claude Opus 4.6

82.0%

GPT-5.4 leads narrowly at 88.5%, but Kimi K2.5 is 1.4 points behind and all three open-source models beat Opus 4.6. The gap across all 5 models is only 6.5 points — this benchmark is nearly saturated.

Speed: output tokens per second

Kimi K2.5

334 t/s

GPT-5.4

~78 t/s

DeepSeek V3.2

~60 t/s

Claude Opus 4.6

46 t/s

DeepSeek R1

~30 t/s

Kimi K2.5 at 334 tok/s is in a different league — 4x faster than GPT-5.4, 7x faster than Opus 4.6. R1 is the slowest (expected — it’s a reasoning model producing chain-of-thought tokens).

Latency: time to first token (seconds)

Kimi K2.5

0.31s

GPT-5.4

~0.95s

DeepSeek V3.2

1.18s

DeepSeek R1

~2.0s

Claude Opus 4.6

2.48s

Lower is better. Kimi K2.5 responds 8x faster than Opus 4.6 and 3x faster than GPT-5.4. Even V3.2 beats both proprietary models. Opus 4.6 is the slowest model in this comparison.

Speed and TTFT measured on our production infrastructure. Claude and GPT-5.4 data from Artificial Analysis.

The full picture

The scorecard

Metric	Winner	Open-source	Proprietary	Gap
Code (SWE-bench)	Opus 4.6	Kimi 76.8%	Opus 80.8%	-4 pts
Reasoning (HLE)	R1	R1 50.2%	GPT-5.4 41.6%	+8.6 pts
Knowledge (MMLU-Pro)	GPT-5.4	Kimi 87.1%	GPT-5.4 88.5%	-1.4 pts
Speed (tok/s)	Kimi K2.5	334 t/s	GPT-5.4 78 t/s	4.3x faster
Latency (TTFT)	Kimi K2.5	0.31s	GPT-5.4 0.95s	3x faster

Open-source wins 3 out of 5. Proprietary models lead on Code (by 4 points) and Knowledge (by 1.4 points). Open-source leads on Reasoning (by 8.6 points), Speed (by 4.3x), and Latency (by 3x).

Note: Kimi K2.5’s HLE score (50.2%) uses tool-augmented mode. Without tools it scores 31.5%. DeepSeek R1’s 50.2% is pure chain-of-thought reasoning without tools.

What “production-ready” actually means

Reliable enough. Consistent quality across thousands of requests.
Fast enough. Kimi K2.5 at 334 tok/s and 0.31s TTFT. That’s real-time.
Capable enough. Within 4 points of the best proprietary model on code, ahead on reasoning.
Predictable. Versioned models that don’t change without warning.

The real advantage: control

Proprietary models change under you. Fine one day, different behavior the next. No changelog, no warning. Open-source models are versioned — DeepSeek V3.2 behaves the same tomorrow as today. You choose when to upgrade.

For production workloads, that predictability is worth more than a marginal quality edge on any single benchmark.

We serve frontier open-weight models — Kimi K2.6, GLM 5.2, and MiniMax M3 — through a single OpenAI- and Anthropic-compatible API. Unlimited-usage time-block subscriptions from $39/mo, plus pay-as-you-go credits from $10. See the pools.

Sources: Artificial Analysis Leaderboard · SWE-bench Leaderboard · Kimi K2.5 Benchmarks · DeepSeek V3.2 · OpenAI Pricing · Anthropic Pricing · HLE Leaderboard · MMLU-Pro Leaderboard

What it takes to build your own LLM inference platform

Thu, 19 Mar 2026 00:00:00 GMT

If you’re building a SaaS that needs to give users access to LLMs, you have two options: build the infrastructure yourself, or use a platform that does it for you. Here’s what “build it yourself” actually looks like.

This isn’t theoretical. We built this. Here’s every component, what it does, and what alternatives exist.

0. Model access — the first problem

Before you write a single line of code, you need access to models.

Self-host on your own hardware: Buy GPUs, rent datacenter space, run the models yourself. Full control, best unit economics at scale — but massive upfront cost and you’re limited to the models you can afford to deploy. Running DeepSeek V3.2 requires multiple high-end GPUs. Running dozens of models? You’d need a data center.

Rent infrastructure: Use GPU clouds like Vast.ai, AWS, Hetzner, CoreWeave, or Lambda. No hardware to buy, but you still manage deployments, scaling, and failover. Costs add up fast — a single H100 runs $2-4/hr.

Use an inference provider: Sign agreements with DeepInfra, Together.ai, Fireworks, etc. who already have the models deployed. Pay per token, no GPU management. But you depend on their availability, pricing, and terms. If they change prices or drop a model, you need a plan B.

Mix: Most serious platforms end up here. Own hardware for high-volume models where the unit economics justify it, rented GPUs for burst capacity, and provider agreements for the long tail of models nobody runs enough to self-host.

Self-hosting dozens of models on your own is economically unrealistic. The real question is where to draw the line between own infra, rented compute, and providers.

1. Serving engine

If you self-host or rent GPUs, you need software to serve the models:

vLLM — most popular, good throughput, active community
TGI (Text Generation Inference) — Hugging Face’s solution, solid for single-model deployments
TensorRT-LLM — NVIDIA’s optimized engine, best raw performance but harder to set up
SGLang — newer, fast, good for structured generation

You’ll also need to handle model weights, quantization, scaling across GPUs, and failover when a node goes down. This is a full-time ops job.

2. API proxy layer

Your users shouldn’t hit the inference backend directly. You need a proxy that:

Translates between API formats (OpenAI, Anthropic)
Routes requests to the right model/provider
Injects authentication
Handles retries and failover
Strips provider headers so users don’t know your backend

Options:

Build from scratch with Express/Fastify + http-proxy-middleware
Use an open-source gateway: LiteLLM, Portkey, Kong AI Gateway, MLflow Gateway
Use a managed gateway: Helicone, Braintrust, Promptlayer

Each has trade-offs. Open-source gateways give you control but you manage the deployment. Managed gateways are easier but add latency and cost.

3. Authentication

Two layers:

User auth (dashboard login)

Firebase Auth, Auth0, Clerk, Supabase Auth, or roll your own
Supports email, Google, GitHub, wallet signatures

API key auth (inference requests)

Generate API keys per user
Validate on every request before proxying
Store key metadata (plan, rate limits, owner)

This is where it gets interesting for platforms. You need per-key plans — each key with its own rate limits and usage tracking. Most auth solutions don’t do this out of the box. You’ll need a custom key management layer.

4. Rate limiting

Per-key rate limiting with at least:

RPM (requests per minute)
TPM (tokens per minute)
Budget caps (dollar amount per time window)

This needs to be enforced at the proxy layer, before the request hits the inference backend. Otherwise a single user can exhaust your GPU allocation.

Options:

Redis-based counters (most common)
Token bucket algorithms
Proxy-level enforcement (some gateways include this)

If you’re using per-key plans, each key needs its own set of limits. Not one global limit — individual limits per key.

5. Usage tracking and billing

You need to know:

How many tokens each key consumed (input + output)
What model was used
Cost per request
Aggregate usage per user, per day, per billing period

For subscription billing:

Stripe for card payments
Budget windows (e.g., $X per 8-hour period)
Automatic key revocation when subscription expires

For pay-as-you-go:

Credit balance per user
Deduct per request based on token count × model price
Top-up flow (Stripe, crypto, etc.)

For crypto payments:

USDC on a supported chain
On-chain transaction verification
Wallet connector in the dashboard (wagmi, viem, etc.)

This is a significant amount of code. Usage tracking alone requires intercepting every response to count tokens, calculating cost based on the model’s pricing, and storing it per key.

6. Dashboard

Your users need a web UI to:

Create and manage API keys
View usage per key (tokens, requests, cost)
Subscribe to plans or top up credits
See available models and pricing

Tech stack typically:

React/Next.js/Vue frontend
REST API backend
Real-time usage updates

For platforms (your users creating keys for their users), you also need a management API — programmatic key creation, plan assignment, usage queries.

7. Model catalog management

Models change. New ones come out weekly. You need:

A catalog of which models you serve
Pricing per model (input/output cost per token)
Sync mechanism to update prices when providers change them
Display names, categories, tags for the dashboard
Cache pricing metadata (some models support prompt caching discounts)

This is an ongoing operational burden, not a one-time setup.

8. Documentation

Your users need:

API reference (endpoints, request/response formats)
SDK examples (Python, Node.js, at minimum)
Authentication guide
Billing/usage documentation
Quick start guide

This is easily 20-30 pages of documentation that needs to stay current.

9. Monitoring and reliability

Health checks on the inference backend
Status page for users
Alerting when latency spikes or errors increase
Logging (but not logging prompt content — privacy)
Graceful degradation when a model or provider is down

10. Compliance and privacy

Privacy policy
Data handling documentation
GDPR compliance if you serve EU users
Decision: do you store prompts? (You shouldn’t)
SOC 2 / ISO 27001 if targeting enterprise

The full stack

Component	Ongoing maintenance
Inference backend	High — scaling, failover, model updates
API proxy	Medium — format changes, new providers
Auth + key management	Low
Per-key rate limiting	Low
Usage tracking + billing	Medium — edge cases, reconciliation
Dashboard	Medium — new features, UX
Model catalog	High — weekly model updates
Documentation	Medium — keep current
Monitoring	Low
Privacy/compliance	Low

What breaks in production

Building is the easy part. The hard part is what breaks with real users:

A provider changes their API format without warning. Your proxy returns 500s for 2 hours until you notice.
A model gets deprecated. Your users’ hardcoded model IDs stop working overnight.
Token counting has an off-by-one bug. You’ve been undercharging for 3 weeks. Your margin is gone.
A user finds a way to exceed rate limits through concurrent requests. Your inference bill spikes 10x in one afternoon.
Stripe webhook fails silently. A user’s subscription expired but their API key still works. Free inference for a month.
You push a billing update and break the usage tracking. Three days of missing data. Users open tickets.

Each of these has happened to us. We fixed them. The question is whether you want to fix them yourself, with your users waiting, or use a platform that already has.

Or

You use an inference platform that already has all of this, create API keys for your users, and ship your product this week.

We built all of the above so you don’t have to. See how per-key plans work.