Every LLM project we have been asked to rescue arrived in the same condition. The demo worked. The demo had worked for months. And nobody could say whether the thing was getting better or worse, because there was no way to tell.
That is the whole problem. Not model quality — measurement.
A demo is a single sample
When you show a stakeholder a working prompt, you are showing them one sample from a distribution whose shape nobody has characterised. It went well. You do not know how often it goes well, on which inputs it goes badly, or what happens to either number when someone edits the system prompt next Tuesday.
Production-grade starts at the point you can answer those questions on demand.
The scaffolding that actually matters
An evaluation harness, built before the feature. A graded set of cases, scored automatically, run on every change. It does not need to be large to be useful — a hundred well-chosen cases with real disagreement in them will teach you more than ten thousand generated ones.
Retrieval you can inspect. If the model can only speak from retrieved passages, then the quality of what it says is bounded by the quality of what it retrieved. Log the retrieved set. Make it visible in the interface. Most “the model hallucinated” incidents are retrieval failures wearing a costume.
Versioning that covers the prompt. The prompt is code. It belongs in the repository, in a pull request, with a diff and a review and a test result attached. A prompt change that nobody reviewed is a production deploy that nobody reviewed.
A cost envelope per feature. Token spend is a product decision. Attribute it to the feature that causes it, watch it per release, and set a budget the same way you would for latency.
What changes when you have it
The interesting shift is not technical. It is that arguments about the system stop being arguments about taste. Somebody proposes a change, the harness scores it, and the number either moved or it did not. Teams that reach this point improve their systems steadily and unremarkably. Teams that do not, ship a demo and then defend it.
None of this is exotic. It is the same discipline that was applied to every other class of software over the last twenty years, arriving late to a field that briefly believed it was exempt.
