Evals in production: how to catch LLM quality regressions before deploy

TL;DRThe single most useful piece of AI infrastructure is an eval suite that runs before every deploy. It doesn't need to be a platform — a JSON file of test cases plus a small Node script is enough. This post shows the exact shape we use: golden set + LLM-as-judge + cost/latency thresholds + regression alerts.

Every AI team we've inherited from another vendor has the same story: "We shipped a prompt change last month and something broke. We don't know when it broke or what change caused it."

That story ends the day you have an eval harness. This post shows the exact shape we build at Praxvon — a day of work that protects a year of debugging.

What an eval harness actually is

Not a platform. Not a tool you buy. Just:

1. A JSON file of test cases (inputs + expected outputs or expected properties).
2. A script that runs each case against your current prompt + model configuration.
3. A comparison step that decides pass/fail.
4. A CI hook that blocks deploys when the pass rate drops.

That's it. The whole system fits in ~200 lines of code and one JSON file. What matters is the discipline of running it, not the tooling.

The golden set

The core artifact is a JSON array of test cases. Each case looks like:

{
  "id": "translation_tr_to_en_technical",
  "input": {
    "sourceText": "Türkçe teknik terimlerle dolu bir cümle örneği.",
    "sourceLang": "tr",
    "targetLang": "en"
  },
  "expected": {
    "type": "llm_judge",
    "criteria": [
      "Preserves technical meaning",
      "No untranslated Turkish words remain",
      "Reads naturally in English"
    ]
  },
  "priority": "high"
}

Two things to notice: the expected is structured, and cases are prioritized. Not every case is life-or-death. When a change breaks 3 low-priority cases but fixes 5 high-priority ones, that's usually acceptable.

Case counts we use in practice: 30-50 for a single-purpose system, 100+ for multi-purpose agents. Growing organically beats trying to write 500 cases day one.

Three comparison strategies

Exact match: for structured outputs where you know the answer. Classification labels, extracted fields, boolean decisions. Cheap, fast, deterministic. Use whenever possible.

LLM-as-judge: for open-ended text where an exact match doesn't work. A second LLM (usually a cheaper model like GPT-4o-mini) evaluates whether the output meets the criteria. Non-deterministic — expect ~5% variance run-to-run.

Multi-signal: score along multiple dimensions and require all to pass. For a translation task: preserves-meaning + no-untranslated-words

• natural-target-language. Each dimension is its own LLM judge or exact match.

Rule of thumb: 60-70% of your cases will be exact-match, 30-40% LLM-as-judge. Start with exact-match and add judges as needed.

Cost + latency budgets in the eval

The eval doesn't just measure quality. It measures cost per pass:

Pass rate: 94%
Avg tokens per response: 720
Avg latency: 1400ms
Cost per 1000 runs: $0.42

Every deploy compares these numbers to the last release. If pass rate holds but token count doubles, the change is rejected. If latency doubles but the model was intentionally switched to a bigger one, the change requires human sign-off.

This is the discipline that catches "quality is fine but the bill just tripled" bugs before they hit production.

CI wiring

The eval runs on:

1. Pull request open: partial suite (~10 highest-priority cases) on every push. Fast feedback for the developer.
2. Pre-merge to main: full suite. Blocks merge on regression.
3. Post-deploy smoke: same 10 cases against production. Detects config drift between staging and prod.
4. Nightly full run: catches slow drifts (model provider silently updates their model, embeddings drift, etc.).

Total CI cost: ~$5-30/month for a mid-scale system. Negligible.

The Moditra example

For Moditra's multi-language chat translation:

• Golden set: 47 cases across TR/EN/AR pairs
• Comparison: 30 exact-match language-detection + 17 LLM-judge quality
• Cost budget: under $0.05 per 1000 translations
• Latency budget: under 400 ms 95th percentile
• CI: runs on every PR, blocks merge on regression

Real regression caught by this: a prompt change that improved quality by 2% but bumped average token count from 180 to 340. Rejected. The developer went back and shortened the system prompt while preserving the quality gain. Net cost impact: zero.

Without the eval, that change would have shipped, and a month later the bill would have quietly doubled with nobody knowing why.

What to skip

Don't buy an eval platform yet. LangSmith, Braintrust, Weights & Biases all sell eval infrastructure. They're fine, but the value they add over a JSON file + Node script is small until you're running 500+ cases across multiple teams. Start with the file.

Don't try to measure everything. A test case that requires 20 subjective criteria is unmaintainable. Pick 3-5 dimensions that matter and enforce those.

Don't rely only on LLM-as-judge. Judges disagree with themselves between runs. Any dimension that can be exact-match, make exact-match.

The one-day setup

Day one, if we're building this from scratch:

1. Morning: write 15-30 test cases that cover the top 3 use cases of the system. Ship JSON file to the repo.
2. Afternoon: write the runner script + LLM-judge prompts + CI hook.
3. Test: run against the current prompt. Fix any low-hanging failures.
4. Wire: PR gates + nightly cron.

That's the whole thing. One day of work protects a year of debugging.

Related caseModitra — B2B Modelist Atelier SaaSModitra · SaaS Platform / UI / UX · 2026Read the case study →

If you're running a production LLM system without an eval harness, contact us — we've done this ~10 times and can bootstrap one for your system in 2-3 days.