You shipped an AI feature. Nobody can tell you whether the last prompt change made it worse.
Three weeks building the thing your test suite cannot be: an evaluation set drawn from your own traffic, a harness that scores every change against it, synthetic scenarios for the cases you have not hit yet, historical replay for the ones you already have, and a gate in your pipeline that fails a pull request when quality drops. All of it in your repository, running on your infrastructure.
Fixed price, no hourly surprises. Three weeks, one AI feature or pipeline. Paid in four milestones, you release each one only when you approve it. Yours afterwards, with no licence and nothing to renew.
How it works
Build the set. Score every change against it. Fail the build when it drops.
Three weeks on one AI feature or pipeline: a support bot, a summariser, an extraction job, a classifier, an agent, whatever is already in production and currently untested. What I need from you is access to the repository, a sample of real traffic, and a few hours total from someone who can tell me what a good answer looks like in your domain.
The set comes out of your real traffic
Not twenty examples somebody invented in a meeting. Real inputs sampled from what your users actually send, deliberately weighted toward the awkward ones: the ambiguous, the out of scope, the adversarial, and the long tail that is where this stuff actually breaks. Each one gets the expected behaviour recorded next to it.
The harness scores it, more than one way
Different cases need different instruments, and reaching for the wrong one is how a suite ends up measuring nothing while reporting a healthy number. The hard checks carry the weight, and a model’s judgement gets used only where nothing else can decide. Every run produces a number per case and a number overall, so two runs are comparable.
It becomes a gate, not a report
Wired into your pipeline so a prompt edit, a model swap, a retrieval change or a dependency bump runs the whole set and fails the pull request when the score drops past the threshold you set. The diff shows which cases changed, so a regression comes with the examples that prove it.
What you get
A set, a harness, and a gate that stays after I go.
One fixed price. All of it ships as plain code in your repository through your own review, running on your infrastructure, with no service of mine in the loop.
Cases that look like reality
- Sampled from your real traffic, covering the common path and then deliberately overweighting the awkward tail where failures actually live
- The adversarial and out-of-scope cases: prompt injection attempts, questions you should refuse, inputs in the wrong language, empty and malformed requests
- Expected behaviour recorded per case, as an exact answer where one exists and as a written rubric where the answer is a judgement call
- Historical replay, where past production traffic is re-run against a candidate change so you can see what it would have done differently
- Synthetic scenarios for the situations you have not hit yet but cannot afford to get wrong, generated and then reviewed by a human rather than trusted blind
Scoring that survives disagreement
- Deterministic checks
- Reference similarity
- Rubric-based model grading
- Must-never assertions
- Cost and latency per run
Four ways to score, and the one everybody reaches for first, a model judging the output, is the weakest of them. Which layer scores which case is the decision that makes the number mean anything, and it gets made against your actual output rather than from a template. Get it wrong and you ship a suite that passes happily while the product gets worse.
Wired in, not bolted on
- A pipeline job that runs the full set on any change that could move quality: prompts, model, retrieval, chunking, dependencies, config
- A threshold you own, so the build fails on a real drop and does not cry wolf on noise, tuned against repeated runs rather than guessed
- A readable diff naming the cases that changed and how, so the conversation is about four specific examples instead of a feeling
- Model version pinning and drift detection, so the day your provider quietly moves the model under you, the set notices before your customers do
- Documentation for adding cases, because the set only stays useful if your team keeps feeding it the failures they find
What this will not do.
One price for all of it: the evaluation set, the harness and its scoring, the synthetic scenarios, the historical replay, the pipeline gate and the handover. Everything is plain code in your repository. A second feature is quoted up front as its own engagement, never as surprise scope, and it is cheaper once the harness exists.
- 1
Scope and the evaluation set $2,000
What good looks like, agreed in writing with your people, then the set built from sampled real traffic with expected behaviour recorded per case.
- 2
The harness and scoring $2,500
Deterministic checks, reference similarity, rubric-based model grading, must-never assertions, and cost and latency captured per run. Shipped through your pipeline.
- 3
Synthetic scenarios and replay $1,500
Generated cases for situations you have not hit yet, reviewed by a human, plus replay of past production traffic against a candidate change.
- 4
The gate and handover $1,000
The pipeline job, a tuned threshold, the readable diff, drift detection on the model version, and the documentation your team needs to keep adding cases.
Each milestone is agreed before it starts and paid only when you approve its deliverable, so you never pay up front for work you have not seen. The set and the harness are useful on their own, so stopping after milestone two leaves you with something that works.
Why hand-checking misses it
Three ways this breaks without anyone noticing.
The prompt fix that broke something else
A line gets added to the system prompt to handle one complaint. It works for that case and quietly changes the tone, the length or the refusal behaviour everywhere else. Hand-checking five examples will not find it, because the five examples are the ones you were already thinking about.
The model that moved under you
Providers update models, deprecate versions and change defaults. Your code did not change, your output did. Without a set that runs on a schedule, the first report of this is a customer, and by then nobody connects it to a date.
The improvement nobody can prove
Someone spends a week on better retrieval and says it is much better. Someone else says it feels the same. There is no way to settle it, so the decision gets made by whoever is more senior. A number on the same set turns that into a two-minute conversation.
Everything stays in your repository.
Plain code, yours to change
The set, the harness, the rubrics, the assertions and the pipeline job are files in your repository, shipped through your own review. No platform account, no evaluation service in the middle, no dashboard I control. Delete my name from the commits and it still runs. My open-source work is on my GitHub if you want to read how I write before you hire me.
Runs on your infrastructure
The harness runs in your pipeline against your models. Your traffic and your evaluation data never leave your environment, and the cases are drawn and stored inside it. Where grading needs a model, it is whichever one you configure. NDA signed before specifics if that is your policy.
Paid in four phases, never up front
Each milestone is agreed before it starts and paid only when you approve its deliverable. Direct B2B, on your own contract and invoicing, with no platform in the middle.
No subscription, no per-run fee
Pay once to build it, then run it as often as you like. There is no licence, no seat count and nothing that bills per evaluation, which matters because the whole point is to run the set on every change rather than ration it.
Straight answers
What does the price cover?
Three weeks in four milestones: the agreed definition of what good looks like; an evaluation set built from sampled real traffic with expected behaviour per case; the harness with deterministic checks, reference similarity, rubric-based model grading and must-never assertions, recording cost and latency per run; synthetic scenarios and historical replay; and a pipeline gate with a tuned threshold, a readable diff, model drift detection and documentation for adding cases. $7,000 total, one feature or pipeline.
How does payment work?
In four phases: scope and the evaluation set ($2,000), the harness and scoring ($2,500), synthetic scenarios and replay ($1,500), the gate and handover ($1,000). A phase is agreed before it starts and paid when you approve that phase’s deliverable. The set and harness are useful without the gate, so stopping after phase two still leaves you with a working thing.
Our output is different every time. How can you test it?
By testing behaviour rather than text. Some things are checkable exactly: the JSON has the right fields, the citation is present and points at a real document, the refusal happened, the length is within bounds. Some are checkable by similarity against a reference. The rest get a written rubric and a model grading against it, which is noisier and gets used where it belongs rather than everywhere. Run the same set twice and the variance is measurable, which is how the threshold gets set.
Is a model grading another model not nonsense?
It is the weakest instrument in the harness and it is treated that way. It is used where judgement genuinely is the question, graded against a rubric that lives in your repository where your team can read and change it, with the grader pinned and the variance measured so you know how much movement is noise. The deterministic checks and the must-never assertions carry the weight, because those do not have an opinion.
How many cases do we need?
Fewer than people expect to start, more than they expect in the end. A set in the low hundreds, weighted toward awkward cases, catches most regressions and runs cheaply enough to sit in a pull request. Thousands of near-identical easy cases cost money on every run and catch nothing. The real answer is that the set grows: every production surprise becomes a case, which is why the handover covers adding them.
What does it cost to run after you leave?
Model spend on each full run, which falls out of your own prices and the set size, and I give you that figure during the build so there is no surprise. Deterministic checks cost nothing. There is a cheap subset for every pull request and the full set on merges or nightly, which is the arrangement that keeps it affordable enough to actually stay switched on.
What do you need from my team?
Access to the repository with the AI feature in it, a sample of real traffic, and whoever normally reviews and merges there. Most importantly, a few hours total from someone who can tell me what a good answer looks like in your domain, because the entire set is built on that judgement and I cannot supply it. One scoping call first to agree which feature and what good means.
Does this work with our stack?
The harness sits around your feature rather than inside it, so it works with any provider you call over an API, anything self-hosted behind an OpenAI-compatible endpoint, and whatever framework you did or did not use. It runs in whichever pipeline you already have. If your feature spans several services in different languages, that comes up during scoping and shapes where the harness goes.
We do not have an AI feature in production yet. Should we do this?
Probably not yet, and not in this form. Without real traffic the set is built from synthetic cases alone, which is better than nothing but much weaker, and you would be paying for the harder half of the value. Ship it, collect a few weeks of real traffic, then do this. If you are still in the deciding stage, the AI Build Plan is the one that fits.
Can you fix the quality problems the set finds?
Not inside this engagement, which builds the instrument rather than tuning the feature. Once the set exists, improving the thing is a much better-defined job because every change gets a number, and that is quoted separately. Plenty of teams take the harness and do the tuning themselves, which is the outcome it is designed for.
How do we actually engage?
Direct B2B, on your own contract and invoicing, through whatever procurement process you already have. There is no platform in the middle and no marketplace fee. Reach me on LinkedIn, tell me what the feature does and roughly what traffic it sees, and we sort the terms directly, including an NDA first if that is your policy.
Tell me what you shipped and how you check it.
This is for teams with an AI feature already in production and no way to tell whether the next change makes it worse. Tell me what it does, roughly what traffic it sees, and how quality gets checked today. If the honest answer is that someone tries a few examples by hand, you are the normal case. We get on a call, I scope it, and three weeks later a prompt edit cannot reach production without passing the set.
Direct B2B, on your own contract and invoicing. Reach me on LinkedIn and we sort the terms. Fixed price, four milestones, no per-run billing.