How I Cut LLM Token Costs by 100× on a Real Project

When I joined the Quality Shop project, the goal sounded simple: take 2.5M+ scraped Shopify stores and find the ~13K that actually belonged on a marketplace for healthy food & drink brands. The catch — the obvious way to do it, sending every store through a capable LLM, would have cost a small fortune.

Here's how I got the bill down by roughly 100× without hurting quality. None of it is clever research; it's just engineering discipline applied to a naive first version.

1. Don't send everything to the big model

The first version called a top-tier model on every store. That's the trap: you pay premium per-token rates on millions of records, most of which are obviously irrelevant.

The fix was a cheap funnel:

• A plain rules/keyword pre-filter (free) threw out the clearly-irrelevant majority before any API call.
• A small, cheap model did the first-pass classification on what remained.
• Only the genuine edge cases reached the expensive model.

Most of the savings came from this single shift — the expensive model went from "every row" to "a tiny fraction of rows."

2. Cache aggressively — never pay twice

Scraped data is messy and repetitive: the same store, near-duplicate descriptions, re-runs after a crash. The naive pipeline happily re-classified all of it.

I added a content-hash cache: hash the normalized input, and if we'd seen it, reuse the result. Re-runs became nearly free, and duplicates across the dataset stopped costing anything after the first hit.

3. Shrink the prompt

Token cost is input + output. The early prompt shipped a bloated instruction block and the full raw store payload on every call.

• Trimmed the system prompt to the essentials.
• Sent only the fields that mattered for the decision, not the whole record.
• Constrained the output to a tiny structured response instead of prose.

The takeaway

The cheapest token is the one you never send.

A "100× cost reduction" sounds dramatic, but it was really four boring wins stacked together: pre-filter, right-size the model, cache, and shrink the prompt. The accuracy held because the expensive model still made the hard calls — it just stopped wasting money on the easy ones.

If you're putting an LLM in production over a large dataset, profile where the tokens actually go before you optimize. Mine were going almost entirely to rows that never needed a model at all.