What happened
A cheap LLM in production can cut inference costs by 70x without wrecking output quality — but only if you measure the tradeoff instead of guessing. That's the real story behind JobPath, a job aggregator that restructures raw job postings scraped from open sources and Telegram channels into clean, searchable listing cards every night via a scheduled Celery task. The team started with Claude Sonnet 4.6 doing all the heavy lifting: turning messy HTML dumps, emoji-laden Telegram posts, and three-line fragments into structured data with summaries, requirement lists, normalized skill names, a 1-5 difficulty score, and a normalized salary range.
Sonnet did the job well from day one. The problem was cost. At $0.0485 per listing, the projected monthly bill was around $2,000 — and climbing every time the catalog grew. In July, the team migrated to a dramatically cheaper model. The new cost: roughly $0.0007 per listing, or about $30 a month. That's a 70x reduction. Quality did drop, from 4.48 to 4.06 on a custom 5-point evaluation scale, but the team argues that's a deliberate tradeoff, not a failure.
### The evaluation process
To choose a replacement, they built a 50-listing eval set pulled from real production data — a deliberate mix of tech and non-tech postings, listings with and without salary info, long walls of text, and three-line scraps. Each candidate model processed the same 50 listings through the same JSON schema via tool calls. A blind judge model (Claude Opus 4.8) then scored each pair of outputs — original text plus two anonymized listing cards — on completeness, accuracy, language quality, and salary-estimate sanity, without knowing which model produced which card. About a third of the cards were also spot-checked manually.
Why it matters
This case matters because it quantifies something most teams only feel anecdotally: the gap between "works" and "works profitably at scale." When a feature launches, the only question is whether it functions — cost optimization comes later. But "later" tends to arrive suddenly, once a catalog, user base, or document volume grows past the point where a few cents per call turns into a real budget line.
The numbers make the incentive concrete. Here's how the candidates stacked up:
| Model | Cost per listing | Monthly estimate | Quality (1-5) |
|---|---|---|---|
| Claude Sonnet 4.6 | $0.0485 | ~$2,000 | 4.48 |
| Qwen3-235B | ~$0.0014 | ~$59 | 4.24 |
| Qwen3.5-Flash | $0.0007 | ~$30 | 4.06 |
| Qwen3.5-9B | lower | not tracked | 2.76 |
| DeepSeek-V3.1 | comparable to Flash | not tracked | failed eval |
Qwen3-235B produced the strongest non-premium output but returned salary fields as null in 52% of cases and frequently confused annual with monthly pay. Qwen3.5-9B was cheaper still but hallucinated facts often enough to be disqualified outright. DeepSeek-V3.1 never made it past evaluation: in 95 of 100 test cases it returned list fields as a single string instead of a proper array, breaking the schema contract entirely. Qwen3.5-Flash won on the balance of price and reliability, landing the team at roughly $30/month for a workload that previously cost $2,000.
How to use it today
If you're running any pipeline that calls an LLM per-record — job postings, product descriptions, support tickets, invoices — the same playbook applies directly:
### Build a real eval set before switching
Don't guess which cheaper model will hold up. Pull 30-50 real production examples, covering your actual edge cases (missing fields, inconsistent formatting, different languages), and run every candidate model through the identical schema and prompt.
### Use a blind LLM judge, carefully
Have a strong model score outputs from multiple candidates without knowing which model produced which output. This isn't scientific-grade evaluation, but it's more than enough to make a go/no-go decision on model swaps — just validate a sample manually to sanity-check the judge's calibration.
### Watch for structural failures, not just quality drops
Cheaper models don't fail gracefully. Expect empty 200-status responses, wrong field types, misspelled schema keys, and confused units (annual vs. monthly salary, for instance). Build validation on top of the tool-call schema so malformed outputs get caught and retried rather than silently corrupting your database.
If you want to prototype this kind of evaluation workflow without building custom tooling from scratch, a few no-cost utilities at https://mykreatool.com can help you quickly test prompts and compare outputs across models before committing to a production migration.
Who benefits
Any team running LLM calls at volume is a candidate for this kind of audit — not just job boards. E-commerce catalogs that generate product descriptions, support platforms classifying tickets, content pipelines summarizing articles, and marketing tools generating ad copy at scale all hit the same wall: a per-call cost that looked negligible in a demo becomes a five-figure annual line item once volume scales.
Startups and solo builders benefit the most immediately, since a 70x cost reduction can be the difference between a feature being sustainable or getting cut. But even larger teams with healthy margins should treat this as a recurring exercise — model pricing and quality shift constantly, and a model that was the best cheap option six months ago may no longer be competitive today.
Risks
Cheaper models are cheaper for a reason, and the failure modes are real, not theoretical. In this case, one unhandled billing error on the new provider caused 16,000 wasted API calls in a single night — a reminder that migrating models isn't just a prompt swap, it's a full review of error handling, retry logic, and billing edge cases on the new vendor's side.
Other risks worth planning for:
- Schema drift: cheaper models are more likely to misname fields, return wrong types, or flatten arrays into strings, which can silently corrupt downstream data if you don't validate every response.
- Domain-specific blind spots: the model that lost badly here (DeepSeek-V3.1) might perform fine on a different schema or language — always test on your actual data, not published benchmarks.
- Blind-judge bias: using one LLM to judge another's output introduces its own calibration risk, so manual spot-checks remain necessary, especially early in a migration.
- Quality-cost tradeoffs compound: a 0.4-point drop on a 5-point scale sounds small until it means slightly more inaccurate salary ranges reaching real job seekers, which can quietly erode user trust.
Conclusion
Switching from a premium model like Claude Sonnet to a lightweight one like Qwen3.5-Flash cut this team's projected LLM bill from $2,000 to $30 a month — a 70x savings — at the cost of a modest, measured quality drop from 4.48 to 4.06. The lesson isn't "always pick the cheapest model." It's that structured evaluation, blind judging, and careful error handling let you find the cheapest model that still meets your bar, and catch the structural failures — null fields, mismatched types, wasted billing calls — before they become production incidents.
Comments 0