guides

5 posts with the tag “guides”

Combined keys: stack your subscriptions into one credential

Jul 9, 2026

On an Unlimited subscription, the thing you’re actually shaping isn’t tokens — it’s capacity over time. Each subscription gives you a daily coverage window (the 8-hour blocks you reserve) and, within it, some number of requests you can run in parallel. Your real capacity is those two dimensions multiplied: parallel slots × hours.

Most people grow along both axes. You buy a second block to cover more of the day, or a second subscription to run more requests at once during your busy hours. The awkward part used to be the bookkeeping: every subscription minted its own key, so a serious setup meant three or four keys, each live at different times, each with its own capacity — and you juggling which one to paste where.

Combined keys remove the juggling. A combined key (it looks like sk-ci-meta-…) folds two or more of your subscriptions into a single credential, and it behaves like the union of everything underneath it.

Two rules, and that’s it

Coverage adds up. A combined key is live whenever any of its subscriptions has an open window. Reserve the Europe block on one subscription and the Americas block on another, combine them, and the one key covers both stretches of the day.

Overlap stacks parallel capacity. Where two subscriptions cover the same hour, their parallel slots add together. That’s the lever for concurrency: if you need to run more requests side by side during your peak hours, buy a second subscription over those hours and combine it in.

A concrete example. Say you hold one full-day (24h) subscription and add a second subscription on just the Europe block. The combined key gives you:

Double the parallel capacity during the Europe hours — the two subscriptions overlap there, so their slots stack.
Baseline capacity the rest of the day — only the 24h subscription is covering those hours.

Coverage is 24/7 (from the full-day subscription); parallel capacity is shaped to peak exactly when you work.

Allowances stay separate

Combining is about capacity, not billing. Each subscription keeps its own monthly allowance and its own renewal date — nothing is pooled or co-mingled. The practical upside is resilience: if one subscription lapses or you let it cancel, it simply drops out of the union. The combined key keeps working with whatever subscriptions remain — no dead key, no scramble to re-issue credentials.

It works across pools

Requests route by the model you ask for. So a combined key can span subscriptions on different pools — a Core Pool subscription and a Frontier Pool subscription under one key — and each request lands wherever its model lives. Ask for deepseek-v4-flash and it’s served from your Core subscription; ask for kimi-k2.7 and it’s served from your Frontier one. GET /v1/models on a combined key lists every model you can reach across all of them.

One credential per subscription

Combining a subscription into a key removes that subscription’s standalone key — each subscription has exactly one credential at a time. That’s deliberate: it means there’s never ambiguity about what capacity a given key carries. A key’s coverage and parallel capacity are always the exact sum of the subscriptions it holds, nothing more, nothing hidden. The dashboard walks you through it when you combine.

Creating one

In the Keys page, choose Create API Key and multi-select the subscriptions you want to combine. Before you commit, a preview shows the resulting daily coverage and peak parallel capacity, so you can see the shape you’re buying into. Prefer the API? The Management API does the same thing programmatically.

Full walkthrough in the combined keys guide. If you’re still deciding which blocks to reserve, the live menu and per-block prices are on the pools page.

CheapestInference serves frontier open-weights models — Kimi K2.7, Kimi K2.6, GLM 5.2, MiniMax M3 (Frontier Pool) and DeepSeek V4 Flash, MiMo v2.5 (Core Pool) — through one OpenAI- and Anthropic-compatible API on unlimited time-block subscriptions. See the pools or get started.

"Claude usage limit reached": how to auto-resume Claude Code

Jul 3, 2026

It’s 11pm. Claude Code is 40 minutes into a refactor, tests are half-green, and then the terminal prints:

5-hour limit reached ∙ resets 3am

The agent stops. The session sits there. If you go to bed, nothing happens at 3am — the limit resets, but nobody is around to type “continue”. You lose the hours between the reset and whenever you come back.

We got tired of this, so we built claude-auto-retry: a small open-source tool that watches your Claude Code session, parses the reset time out of the limit message, waits for it, and resumes the session automatically. You run claude exactly like before. That’s the whole pitch.

The problem: limits don’t care where your task is

Claude subscriptions meter usage in rolling windows. Heavy Claude Code use — long agent runs, big contexts, lots of tool calls — burns through a window fast, and when it’s gone you get a variant of:

5-hour limit reached ∙ resets 3pm
You've hit your limit · resets 3pm
Claude usage limit reached. Resets at 2pm

The limit itself is fine — it’s how a fixed-price subscription stays fixed-price. The annoying part is what happens after: Claude Code doesn’t queue your work and resume at the reset. It just stops, mid-task, holding all the context of whatever it was doing. Resuming costs one word — “continue” — but only if a human is there to type it.

For interactive use that’s a coffee break. For the way people increasingly use Claude Code — overnight runs, long autonomous tasks, claude -p in scripts — it’s a silent halt that wastes every hour between the reset and your return.

Workarounds people try (and why they’re fragile)

Sitting there. Works, wastes an evening.

A cron job that types “continue” at a fixed time. Blind. The reset time moves with your usage, so you either fire too early (message goes nowhere, session still limited) or too late (hours lost anyway). And if Claude exited, you’re injecting keystrokes into a bare shell.

Wrapper scripts around the CLI. Most break something: interactive mode, claude -p piping, or they lose the session when your terminal disconnects.

The failure modes all reduce to the same two hard parts: knowing when the limit actually resets, and injecting “continue” safely into a live session — including after you’ve closed your laptop.

How claude-auto-retry solves it

The tool leans on tmux for session persistence, but hides it completely:

Transparent wrapping. The installer adds a shell function so claude transparently runs inside a tmux session. If you’re already in tmux, it uses your pane. You won’t notice the difference — interactive use, flags, and claude -p "prompt" | jq all work as before.
Background monitoring. A separate process polls the pane every 5 seconds looking for the known limit-message patterns (customizable if Anthropic changes the wording).
Real reset-time parsing. When a limit hits, it parses the actual reset time from the message — timezone-aware and DST-safe — computes the wait, and sleeps until reset plus a small safety margin. No guessed schedules.
Safe injection. Before sending “continue”, it verifies Claude is still the foreground process in the pane. If Claude exited, nothing gets typed into your shell.
Survives disconnects. Because the session lives in tmux, you can close the terminal, drop SSH, or shut the laptop lid. The monitor keeps waiting server-side; reattach whenever and the run has continued without you.

Separately from usage limits, it also catches transient API errors (429, 5xx, 529 overloaded) and retries those with exponential backoff — a different failure with a different fix, handled by the same monitor.

It’s pure Node.js with zero npm dependencies, MIT-licensed, and logs everything it does to ~/.claude-auto-retry/logs/.

Install

npm i -g claude-auto-retry
claude-auto-retry install

The installer adds the shell function to ~/.bashrc or ~/.zshrc and installs tmux if it’s missing (apt, dnf, brew, pacman, apk). Then use Claude Code exactly as always:

claude

When a limit hits, you’ll see the monitor take over. Check on it with:

claude-auto-retry status   # what it's waiting for, and until when
claude-auto-retry logs     # what it has done

Retry counts, the wait margin, and custom detection patterns live in ~/.claude-auto-retry.json if you want to tune them. Needs Node.js ≥ 18; tested on Ubuntu, CentOS, macOS, Arch, and Alpine.

We first shared the tool on r/ClaudeAI — the discussion there is a decent picture of how many people hit this exact wall.

When waiting isn’t an option

claude-auto-retry makes limits painless when the work can wait a few hours. Some workloads can’t — always-on agents, batch jobs, anything that has to keep moving at 4am.

Claude Code speaks the Anthropic Messages API, which means it can point at any compatible endpoint — including ours. CheapestInference serves open-weight models across two pools (Kimi K2.7, Kimi K2.6, GLM 5.2, and MiniMax M3 in the Frontier pool; DeepSeek V4 Flash and MiMo v2.5 in the budget Core pool) through an Anthropic-compatible endpoint, with a different limits model: you reserve daily 8-hour time blocks, and during your reserved hours inference is unlimited — there is no usage-limit message to parse, because there is no rolling cap to hit.

export ANTHROPIC_BASE_URL="https://api.cheapestinference.com/anthropic"
export ANTHROPIC_AUTH_TOKEN="your-api-key"
export ANTHROPIC_MODEL="kimi-k2.7"
claude

The two approaches compose. Plenty of people keep their Claude subscription for interactive work and run the long unattended stuff — the runs that would otherwise die at 11pm — on an open-weight model during a reserved block. Open-weight coding models are closer to frontier than most people assume; here’s the current evidence.

Either way: stop typing “continue”.

Self-hosted vs. API inference: the real cost comparison

Apr 15, 2026

“Why pay for an API when I can run the model myself?”

It’s a reasonable question. Open-source models are free. GPUs are available on every cloud. vLLM and Ollama make serving straightforward. The math should be simple: GPU cost per hour × hours = total cost. Done.

Except it’s not. The GPU is the minority of the cost. Here’s the full picture.

The visible costs

GPU hardware

Running DeepSeek V3.2 (671B MoE, ~130B active parameters) requires at least 4× A100 80GB or 2× H100 80GB in FP8. Qwen 3.5 397B has similar requirements.

Setup	Hourly	Monthly (24/7)	Monthly (8h/day)
4× A100 80GB (cloud)	$12.80	$9,216	$2,816
2× H100 80GB (cloud)	$8.40	$6,048	$1,848
1× A100 80GB (Llama 70B)	$3.20	$2,304	$704
1× L40S (Llama 8B)	$1.10	$792	$242

These are cloud GPU rental prices (AWS, GCP, Lambda Labs — varies by provider and availability). If you buy hardware, the upfront cost is $15K–$40K per GPU, amortized over 3–4 years, plus electricity, cooling, and data center costs.

Smaller models are cheaper — but limited

Running Llama 3.1 8B on a single L40S costs $242/month (8h/day). That’s competitive with API pricing. But 8B models can’t handle complex coding, multi-step reasoning, or nuanced analysis — the tasks where AI provides the most value.

The models worth self-hosting (70B+, MoE) require multi-GPU setups where the economics change dramatically.

The invisible costs

GPU rental is just the beginning.

1. Operations and maintenance

Someone has to:

Set up vLLM/TGI with optimal batch sizes, quantization, and memory allocation
Monitor GPU utilization and restart crashed processes
Update model weights when new versions release
Handle OOM errors, NCCL failures, and driver issues
Manage the serving infrastructure (load balancer, health checks, auto-scaling)

If this is a full-time DevOps engineer at $150K/year, that’s $12,500/month in labor. If it’s 20% of a senior engineer’s time, it’s $2,500/month. Either way, it’s more than the GPU.

2. Idle capacity

GPUs cost money whether they’re inferring or not. If your usage pattern is 8 hours of heavy use (work hours) and 16 hours of near-zero traffic, you’re paying for 24 hours and using 8.

Cloud spot instances help but introduce availability risk. Auto-scaling GPU clusters is possible but complex — model loading takes minutes, not seconds.

API pricing is purely usage-based. Zero requests = zero cost.

3. Multi-model overhead

Self-hosting one model is manageable. Self-hosting five models for different tasks — a coding model, a reasoning model, a fast classification model, an embedding model, and a vision model — requires either:

5 separate GPU instances (expensive)
Shared GPU with model swapping (slow — loading a 70B model takes 2–5 minutes)
A serving framework that handles multi-model routing (complex)

An API gives you access to many models through the same endpoint. No model loading, no GPU allocation, no routing logic.

4. Opportunity cost

Every hour your team spends on inference infrastructure is an hour not spent on your actual product. For startups, this is the most expensive cost of all — it doesn’t show up on any invoice.

Total cost of ownership

For a team of 5 developers running AI-assisted coding with a mix of DeepSeek V3.2 and smaller models:

Cost	Self-hosted	API (per-token)	API (time-block sub)
Compute/inference	$2,800	$265	$250
Ops/maintenance	$2,500	$0	$0
Idle waste (~60%)	$1,680	$0	$0
Total monthly	$6,980	$265	$250

Self-hosting costs 26x more for the same workload. The GPU is only 40% of the self-hosted cost — ops and idle waste are the majority.

When self-hosting makes sense

Self-hosting wins in specific scenarios:

Data sovereignty: If your data cannot leave your network — regulated industries, government, healthcare with strict compliance — self-hosting is the only option. No API provider can guarantee the data isolation you need.

Extreme scale: If you’re processing millions of requests per day and your GPUs are consistently at 80%+ utilization, the per-token math eventually favors owned hardware. This threshold is higher than most teams expect — typically $20K+/month in API spend before self-hosting breaks even.

Custom models: If you’ve fine-tuned a model and need to serve it, self-hosting or a dedicated inference provider (Fireworks, Together) is required. Most unified APIs don’t serve custom model weights.

Latency control: If you need guaranteed sub-100ms TTFT and your data center is co-located with your GPUs, self-hosting eliminates network hops.

For everyone else — startups, small teams, companies with variable usage patterns — the API is cheaper, faster to set up, and easier to maintain.

The migration path

Most teams don’t need to choose one forever. A practical approach:

Start with an API: Get your product working, validate demand, understand your usage patterns.
Optimize model selection: Use cheaper models for simple tasks, frontier models for hard tasks. Full guide: Multi-model architecture.
Evaluate self-hosting when: Your monthly API spend exceeds $10K, your GPU utilization would be >70%, and you have DevOps capacity to maintain it.
Hybrid: Self-host your high-volume models, use an API for long-tail models and overflow capacity.

The worst outcome is spending 3 months setting up GPU infrastructure before you’ve validated that anyone wants your product.

CheapestInference serves six open-weight models across two pools — Kimi K2.7, Kimi K2.6, GLM 5.2, and MiniMax M3 (Frontier); DeepSeek V4 Flash and MiMo v2.5 (Core) — through a single API. No GPUs to manage, no idle costs, no ops burden. Reserve a daily 8-hour time block for unlimited usage from $5.94/mo (Core) or $44.20/mo (Frontier), billed annually — reserve all three blocks for full 24/7. Get started or see the pools.

Building a multi-model architecture: route requests to the right LLM

Mar 19, 2026

Using one model for everything is the simplest architecture. It’s also the most wasteful. A 685B-parameter reasoning model answering “what’s the weather?” is like hiring a PhD to sort mail.

This guide covers how to use a small, fast model to classify incoming requests and route them to the right specialist. The result: lower latency, lower cost, and often better quality — because each model handles what it’s actually good at.

The problem with single-model architectures

Most applications start with one model:

User request --> Large Model --> Response

This works, but every request — simple or complex — pays the same latency and cost penalty. When 60% of your traffic is simple classification, FAQ, or extraction, you’re burning expensive compute on tasks a small model handles equally well.

Llama 3.1 8B

~200 t/s

DeepSeek V3.2

~60 t/s

DeepSeek R1

~30 t/s

The gap between Llama 8B and R1 is nearly 7x in throughput. Routing simple requests to the small model saves that difference on every request.

The multi-model architecture

User request --> Router (GLM 5.2) --> classify intent
                                          |
                  +-----------+-----------+-----------+
                  |           |           |           |
           simple/general  reasoning    code        agent
                  |           |           |           |
               GLM 5.2    MiniMax M3  MiniMax M3  Kimi K2.7
                  |           |           |           |
                  +-----+-----+-----+-----+
                        |
                     Response

Two stages:

Classify — The router model reads the user’s message and outputs a category. A fast model returns this in a fraction of a second.
Route — Based on the category, forward the request to the appropriate specialist model.

The router adds minimal overhead (~200ms) but saves significant compute by keeping simple requests away from expensive models.

Step 1: Classify with a fast model

A fast, lightweight model makes a good router. With low TTFT and a short, single-word output, the classification step costs almost nothing and completes before the user notices. (On CheapestInference, DeepSeek V4 Flash or MiMo v2.5 in the Core pool are natural router models; the example below uses GLM 5.2 so everything runs on one Frontier subscription.)

The classification prompt is simple — you want a single-word category, not a conversation:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.cheapestinference.com/v1",
    api_key="your-api-key"
)

def classify_request(user_message: str) -> str:
    """Classify a user message into a routing category."""
    response = client.chat.completions.create(
        model="glm-5.2",
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the user's message into exactly one category. "
                    "Respond with only the category name, nothing else.\n\n"
                    "Categories:\n"
                    "- simple: greetings, FAQ, simple factual questions\n"
                    "- general: complex questions, analysis, writing, summarization\n"
                    "- reasoning: math, logic, multi-step problems, science\n"
                    "- code: code generation, debugging, refactoring, technical implementation\n"
                    "- agent: tasks requiring tool use, web search, or multi-step execution"
                )
            },
            {"role": "user", "content": user_message}
        ],
        max_tokens=10,
        temperature=0
    )
    category = response.choices[0].message.content.strip().lower()
    # Default to general if classification is unclear
    valid = {"simple", "general", "reasoning", "code", "agent"}
    return category if category in valid else "general"

The key details: max_tokens=10 because we only need one word. temperature=0 for deterministic routing. The system prompt is explicit about format — no preamble, just the category.

Step 2: Route to the specialist

Each category maps to a model optimized for that task:

# Model routing table
ROUTE_TABLE = {
    "simple":    "glm-5.2",
    "general":   "glm-5.2",
    "reasoning": "MiniMax-M3",
    "code":      "MiniMax-M3",
    "agent":     "kimi-k2.7",
}

def route_request(user_message: str, conversation_history: list) -> str:
    """Classify and route a request to the appropriate model."""
    category = classify_request(user_message)
    model = ROUTE_TABLE[category]

    response = client.chat.completions.create(
        model=model,
        messages=conversation_history + [
            {"role": "user", "content": user_message}
        ],
        stream=True
    )

    # Stream the response back
    full_response = ""
    for chunk in response:
        if chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            full_response += content
            print(content, end="", flush=True)

    return full_response

Notice that simple requests route back to GLM 5.2 — the same model that did the classification. For simple queries, the router overhead is effectively zero because the specialist is the same model and can reuse the warm connection.

Step 3: Handle edge cases

The basic router works for most traffic, but production systems need a few refinements:

def route_request_production(
    user_message: str,
    conversation_history: list,
    force_model: str = None
) -> tuple[str, str]:
    """Production router with overrides and fallback."""

    # Allow explicit model override (for power users or testing)
    if force_model:
        model = force_model
        category = "override"
    else:
        category = classify_request(user_message)
        model = ROUTE_TABLE[category]

    try:
        response = client.chat.completions.create(
            model=model,
            messages=conversation_history + [
                {"role": "user", "content": user_message}
            ]
        )
        return response.choices[0].message.content, category

    except Exception:
        # Fallback to GLM 5.2 if the specialist is unavailable
        fallback = "glm-5.2"
        response = client.chat.completions.create(
            model=fallback,
            messages=conversation_history + [
                {"role": "user", "content": user_message}
            ]
        )
        return response.choices[0].message.content, f"{category}->fallback"

Three patterns worth noting:

Force model — Let callers bypass routing when they know what they need.
Fallback — If a specialist model is down, fall back to GLM 5.2. It handles everything reasonably well.
Return the category — Log which route each request takes. You’ll need this data to tune the system.

Cost and latency comparison

Consider a workload of 1,000 requests with this distribution: 600 simple, 300 general, 70 reasoning, 30 code. Average 500 input tokens, 200 output tokens per request.

Single-model approach (everything on V3.2)

Avg latency

~4.5s

All 1000 reqs

V3.2 only

Every request waits for V3.2’s ~1.2s TTFT plus generation time at ~60 t/s. Simple questions get the same treatment as complex analysis.

Multi-model approach (routed)

Simple (600)

~1.2s (8B)

General (300)

~4.7s (V3.2)

Reasoning (70)

~9.0s (R1)

Code (30)

~3.5s (Coder)

The weighted average latency drops to approximately 2.7s — a 40% reduction. The 600 simple requests finish in ~1.2s instead of ~4.5s. That’s a 3.7x improvement for the majority of your traffic.

The 70 reasoning requests are slower individually (~9s vs ~4.5s) because R1 generates chain-of-thought tokens. But the quality on those specific requests is significantly better — R1 scores 50.2% on HLE versus V3.2’s 39.3%.

You get faster averages and better quality on the hard tail.

Real example: a support chatbot

A customer support chatbot receives three types of requests:

FAQ (60%) — “What are your business hours?” / “How do I reset my password?”
Complex support (30%) — “I was charged twice for order #12345, can you investigate?”
Technical issues (10%) — “Your API returns 500 when I send multipart form data with UTF-8 filenames”

Without routing

All requests go to DeepSeek V3.2. FAQs get correct answers but with unnecessary latency. Technical issues get decent answers but miss edge cases that a code-specialized model would catch.

With routing

SUPPORT_ROUTES = {
    "simple":    "glm-5.2",       # FAQ, greetings
    "general":   "glm-5.2",       # Complex support
    "reasoning": "glm-5.2",       # Investigations
    "code":      "glm-5.2",       # Technical issues
    "agent":     "kimi-k2.7",     # Multi-step resolution
}

FAQs resolve quickly via GLM 5.2. Complex support issues get GLM 5.2’s full analytical capability. Technical problems also route to GLM 5.2, which understands the code context well. If a support issue requires looking up order data via API, it routes to Kimi K2.7 for tool-assisted resolution.

The classification step adds ~200ms. For the 60% of requests that drop from ~4.5s to ~1.2s, that’s an invisible cost.

When NOT to use multi-model routing

Routing adds complexity. Skip it when:

All your requests are the same type. If you’re building a code editor, just use a single coding model like GLM 5.2. No routing needed.
You have fewer than 100 requests/day. The cost savings don’t justify the engineering overhead at low volume.
Latency doesn’t matter. For batch processing or async workloads, a single capable model is simpler.
Your classification accuracy is low. If the router misclassifies frequently, you get worse results than a single good model. Test the classifier on real traffic before deploying.

The sweet spot is high-volume applications with diverse request types — chatbots, API gateways, developer tools, and customer-facing products where response time directly affects user experience.

Implementation checklist

Log your traffic. Before building a router, understand your request distribution. What percentage is simple? Complex? Code?
Start with two tiers. A fast, lighter model for simple requests, and a stronger model like MiniMax M3 for everything that needs deep reasoning, code, or long context. Add specialists only when you have data showing they help.
Measure classification accuracy. Sample 100 requests, manually label them, compare against the router’s output. Target >90% accuracy.
Add fallback. Every specialist route should fall back to GLM 5.2 if the specialist is unavailable.
Monitor per-route metrics. Track latency, cost, and quality per category. This tells you where to optimize next.

The routing pattern works with any OpenAI-compatible API. The code examples in this guide use the model ids we actually serve — GLM 5.2, MiniMax M3, and Kimi K2.7 from our Frontier pool (DeepSeek V4 Flash and MiMo v2.5 in the Core pool make great router models too); the throughput and latency comparisons cite ecosystem reference models like Llama and DeepSeek V3.2/R1 for context. If you’re building a platform that needs LLM access for your users, see how per-key plans work.

Sources: Artificial Analysis Leaderboard · DeepSeek V3.2 · HLE Leaderboard

What it takes to build your own LLM inference platform

Mar 19, 2026

If you’re building a SaaS that needs to give users access to LLMs, you have two options: build the infrastructure yourself, or use a platform that does it for you. Here’s what “build it yourself” actually looks like.

This isn’t theoretical. We built this. Here’s every component, what it does, and what alternatives exist.

0. Model access — the first problem

Before you write a single line of code, you need access to models.

Self-host on your own hardware: Buy GPUs, rent datacenter space, run the models yourself. Full control, best unit economics at scale — but massive upfront cost and you’re limited to the models you can afford to deploy. Running DeepSeek V3.2 requires multiple high-end GPUs. Running dozens of models? You’d need a data center.

Rent infrastructure: Use GPU clouds like Vast.ai, AWS, Hetzner, CoreWeave, or Lambda. No hardware to buy, but you still manage deployments, scaling, and failover. Costs add up fast — a single H100 runs $2-4/hr.

Use an inference provider: Sign agreements with DeepInfra, Together.ai, Fireworks, etc. who already have the models deployed. Pay per token, no GPU management. But you depend on their availability, pricing, and terms. If they change prices or drop a model, you need a plan B.

Mix: Most serious platforms end up here. Own hardware for high-volume models where the unit economics justify it, rented GPUs for burst capacity, and provider agreements for the long tail of models nobody runs enough to self-host.

Self-hosting dozens of models on your own is economically unrealistic. The real question is where to draw the line between own infra, rented compute, and providers.

1. Serving engine

If you self-host or rent GPUs, you need software to serve the models:

vLLM — most popular, good throughput, active community
TGI (Text Generation Inference) — Hugging Face’s solution, solid for single-model deployments
TensorRT-LLM — NVIDIA’s optimized engine, best raw performance but harder to set up
SGLang — newer, fast, good for structured generation

You’ll also need to handle model weights, quantization, scaling across GPUs, and failover when a node goes down. This is a full-time ops job.

2. API proxy layer

Your users shouldn’t hit the inference backend directly. You need a proxy that:

Translates between API formats (OpenAI, Anthropic)
Routes requests to the right model/provider
Injects authentication
Handles retries and failover
Strips provider headers so users don’t know your backend

Options:

Build from scratch with Express/Fastify + http-proxy-middleware
Use an open-source gateway: LiteLLM, Portkey, Kong AI Gateway, MLflow Gateway
Use a managed gateway: Helicone, Braintrust, Promptlayer

Each has trade-offs. Open-source gateways give you control but you manage the deployment. Managed gateways are easier but add latency and cost.

3. Authentication

Two layers:

User auth (dashboard login)

Firebase Auth, Auth0, Clerk, Supabase Auth, or roll your own
Supports email, Google, GitHub, wallet signatures

API key auth (inference requests)

Generate API keys per user
Validate on every request before proxying
Store key metadata (plan, rate limits, owner)

This is where it gets interesting for platforms. You need per-key plans — each key with its own rate limits and usage tracking. Most auth solutions don’t do this out of the box. You’ll need a custom key management layer.

4. Rate limiting

Per-key rate limiting with at least:

RPM (requests per minute)
TPM (tokens per minute)
Budget caps (dollar amount per time window)

This needs to be enforced at the proxy layer, before the request hits the inference backend. Otherwise a single user can exhaust your GPU allocation.

Options:

Redis-based counters (most common)
Token bucket algorithms
Proxy-level enforcement (some gateways include this)

If you’re using per-key plans, each key needs its own set of limits. Not one global limit — individual limits per key.

5. Usage tracking and billing

You need to know:

How many tokens each key consumed (input + output)
What model was used
Cost per request
Aggregate usage per user, per day, per billing period

For subscription billing:

Stripe for card payments
Budget windows (e.g., $X per 8-hour period)
Automatic key revocation when subscription expires

For pay-as-you-go:

Credit balance per user
Deduct per request based on token count × model price
Top-up flow (Stripe, crypto, etc.)

For crypto payments:

USDC on a supported chain
On-chain transaction verification
Wallet connector in the dashboard (wagmi, viem, etc.)

This is a significant amount of code. Usage tracking alone requires intercepting every response to count tokens, calculating cost based on the model’s pricing, and storing it per key.

6. Dashboard

Your users need a web UI to:

Create and manage API keys
View usage per key (tokens, requests, cost)
Subscribe to plans or top up credits
See available models and pricing

Tech stack typically:

React/Next.js/Vue frontend
REST API backend
Real-time usage updates

For platforms (your users creating keys for their users), you also need a management API — programmatic key creation, plan assignment, usage queries.

7. Model catalog management

Models change. New ones come out weekly. You need:

A catalog of which models you serve
Pricing per model (input/output cost per token)
Sync mechanism to update prices when providers change them
Display names, categories, tags for the dashboard
Cache pricing metadata (some models support prompt caching discounts)

This is an ongoing operational burden, not a one-time setup.

8. Documentation

Your users need:

API reference (endpoints, request/response formats)
SDK examples (Python, Node.js, at minimum)
Authentication guide
Billing/usage documentation
Quick start guide

This is easily 20-30 pages of documentation that needs to stay current.

9. Monitoring and reliability

Health checks on the inference backend
Status page for users
Alerting when latency spikes or errors increase
Logging (but not logging prompt content — privacy)
Graceful degradation when a model or provider is down

10. Compliance and privacy

Privacy policy
Data handling documentation
GDPR compliance if you serve EU users
Decision: do you store prompts? (You shouldn’t)
SOC 2 / ISO 27001 if targeting enterprise

The full stack

Component	Ongoing maintenance
Inference backend	High — scaling, failover, model updates
API proxy	Medium — format changes, new providers
Auth + key management	Low
Per-key rate limiting	Low
Usage tracking + billing	Medium — edge cases, reconciliation
Dashboard	Medium — new features, UX
Model catalog	High — weekly model updates
Documentation	Medium — keep current
Monitoring	Low
Privacy/compliance	Low

What breaks in production

Building is the easy part. The hard part is what breaks with real users:

A provider changes their API format without warning. Your proxy returns 500s for 2 hours until you notice.
A model gets deprecated. Your users’ hardcoded model IDs stop working overnight.
Token counting has an off-by-one bug. You’ve been undercharging for 3 weeks. Your margin is gone.
A user finds a way to exceed rate limits through concurrent requests. Your inference bill spikes 10x in one afternoon.
Stripe webhook fails silently. A user’s subscription expired but their API key still works. Free inference for a month.
You push a billing update and break the usage tracking. Three days of missing data. Users open tickets.

Each of these has happened to us. We fixed them. The question is whether you want to fix them yourself, with your users waiting, or use a platform that already has.

Or

You use an inference platform that already has all of this, create API keys for your users, and ship your product this week.

We built all of the above so you don’t have to. See how per-key plans work.