AI inference cost control: token budgets, caching, and eval harnesses

TL;DRMost production LLM projects fail on the token bill before they fail on quality. Three disciplines fix it: (1) hybrid model routing — cheap model for easy cases, expensive model for hard ones; (2) prompt + response caching at every layer that tolerates it; (3) eval harnesses that measure quality per dollar, not per model. Moditra runs on ~$120/month serving 40,000+ daily translations.

Every AI project we've inherited from another team hit the same problem before quality: the bill was unpredictable. First month $80. Second month $340. Third month $1,900 with a bug that no one caught in time.

This post is the discipline we apply to keep production LLM systems under control — economically, not just technically.

The three disciplines

1. Hybrid model routing — right-size the model per request
2. Cache aggressively — never call the same prompt twice
3. Eval harness measures cost/quality, not just quality

Do these three and a production system stays predictable. Skip any of them and you're gambling.

Hybrid model routing

The mistake: pick GPT-4o or Claude Sonnet 4 for everything because "quality matters." Every classification, every one-line generation, every translation runs at $2.50/M input tokens.

The fix: classify the request difficulty first, then pick the model.

• Cheap tier (GPT-4o-mini, Claude Haiku 4.5): simple classification, short-form generation, translation, cached-context Q&A. Runs at ~$0.15/M input tokens — 15× cheaper.
• Expensive tier (GPT-4o, Claude Sonnet 4): complex reasoning, agent orchestration, long-context synthesis.

A router at the entry point tags each request with a difficulty class, routes to the appropriate tier, and falls back to the expensive tier if the cheap one fails an eval check. In production this typically sends 70–85% of traffic to the cheap tier.

Cache at every layer

Response cache (highest ROI): if the same prompt has been called before with the same context, return the previous response. Keyed on hash(model + prompt + context). TTL 24h–7d depending on freshness needs. Serve from Redis or Cloudflare KV.

Cache hit rate we've seen in production:

• FAQ-style bots: 40–60% (repeated questions dominate)
• Translation systems: 60–85% (phrase reuse is heavy)
• Complex agents: 5–15% (each session is unique)

Prompt cache (Claude + OpenAI native): both providers now cache the system prompt + few-shot examples on the server side. A 20,000-token system prompt only pays full cost on first call. Discount ~90% on cached tokens. Costs nothing to enable — just set the cache breakpoint markers in your prompt.

Embedding cache: RAG systems reuse embeddings for the same text. Cache the embedding vector keyed on hash(text). Saves ~$0.02/1000 embeddings — small per unit but compounds fast on high-volume systems.

Eval harness measures cost per quality point

The failure mode: teams write evals that measure quality only. They pass 95% and celebrate. Meanwhile the bill quietly triples because the model gets more verbose over time.

The fix: eval output has two numbers — pass rate and average tokens per response.

Every prompt change runs against the eval suite before deploy. If pass rate goes up 2% and token cost goes up 50%, the change is rejected. You paid more than the quality gain was worth.

We use a simple regression: if Δ(cost) / Δ(pass_rate) > threshold, alert the change author. Threshold depends on your unit economics — for high-volume systems it's aggressive ($0.10/percentage-point).

Alerting that catches runaway bills

Every production LLM system gets a daily cost alert. Not weekly — weekly means Friday night you find out Monday's deploy tripled the bill.

Setup:

1. Log per-request cost + model tier at the router.
2. Aggregate hourly into a metric.
3. Alert on hourly_cost > 3× rolling_7day_avg and daily_cost > 2× rolling_30day_median.

The rolling median matters — averages get skewed by a single outlier day and hide slow ramps.

The Moditra numbers

Moditra runs multi-language chat translation (Turkish, English, Arabic) for hundreds of active users. Message volume: ~40,000 translations per day.

• Model routing: GPT-4o-mini handles 92% of requests. GPT-4o reserved for dispute-resolution copy where nuance matters.
• Cache hit rate: 78% on translation responses (phrase repetition is heavy in industrial chat).
• Total inference cost: ~$120/month.
• Estimated cost without any of these disciplines: ~$1,800/month (based on flat GPT-4o pricing without caching).

That is a 15× multiplier. Same output quality per the eval suite. The discipline paid for itself in the first week.

What we would not skip

If we had to cut anywhere, the last thing to lose would be the eval harness. Without it, every prompt change is a coin flip. With it, cost regressions are caught before deploy — and quality regressions are caught with them.

The second-hardest to give up is the daily cost alert. It's the difference between "we have a system in production" and "we have a liability in production."

How we approach it at Praxvon

We build every AI system with all three disciplines from day one, not after the first surprise bill. The eval suite ships with the code. The cache layers are in the architecture diagram before the first prompt is written. The alerting is wired to the same Uptime Kuma instance we use for HTTP monitoring.

None of this requires an ML platform. Redis for cache, a JSON file for evals, cron for the alert. Total operational overhead: a few hours a month.

Related caseModitra — B2B Modelist Atelier SaaSModitra · SaaS Platform / UI / UX · 2026Read the case study →

If you're running a production LLM system and the cost curve is unpredictable, contact us — an audit of these three disciplines usually finds a 30–70% cost cut without touching quality.