You have done it so many times. Changed the prompt, added more context, read the first answer back, thought it was better… and it was a disaster.
You didn’t measure it. You judged it on vibes.
You read the first answer back and thought it was better. You didn’t measure it. You judged it on vibes.
Most AI agencies operate on vibes. You are not as far ahead of them as you like to think.
Gell-Mann amnesia inside your business
Michael Crichton had a name for this trap. Gell-Mann amnesia.
You open a newspaper to an article on the one area you’re deeply specialist in, and within a paragraph you realise this author doesn’t know anything. They’re not talking any sense. Then you turn the page to an area you know nothing about, and suddenly it’s amazing. Wow, look at the insights they’ve got. What an interesting read. We’re only a few weeks away from curing death.
Same newspaper. Same kind of writer. You only caught the mistakes where you were the expert.
AI output works exactly the same way.
It does look great. That’s why it’s misleading. A report, a recommendation, a confident answer to a customer… it reads beautifully to anyone who doesn’t know the subject. Only the investigation by the specialist reveals that it’s not great.
Take the specialist out of the room and you get a massive dose of Gell-Mann amnesia right across the business, and nobody notices.
From 60% to 87%
We built an agent for a consultancy client that read spreadsheet data from their customers and sorted every line item into a category. They had loads of previous ones they’d done by hand, so they gave us the files. When we got fed this line item, said in this way, it went in this category. And we ran our agent against that again and again and again and again.
60% accurate. 63% accurate. 70% accurate. Go back and change it. 75.
And I think we got it up to about 87% accurate.
But the interesting thing wasn’t the number. It was what happened when their experts looked at the ones it was getting wrong. They realised their own team would have needed to go back to the customer and confirm those. There wasn’t enough information in the data to get them right.
A developer sees a failing agent. The expert sees missing data.
60%, 63%, 70%, 75%, then about 87%. When the experts read the ones it got wrong, they didn’t see a failing agent. They saw missing data.
That is the whole game. And it isn’t played by the people writing the code.
Why a non-technical founder has the advantage
If you are a non-technical founder, entrepreneur or business owner right now, you have a unique advantage in the world of AI. And it’s one that not many people are talking about.
We think so often that being more technical is what matters. That you can prompt the AI to build you more robust systems, things that are more secure, more GDPR compliant. But that misses the true performance gap in most of the AI systems we have built over the last five years.
Building the technology to have an AI produce a report, do some consulting, produce value for the customer is not that difficult. Often we can build the proof of concept very, very quickly.
That is not the long tail of this work.
The real work, where you need to roll your sleeves up and get stuck in, comes in the subject matter expert layer. That looks like reading through hundreds of chats between your customers and your AI agents, and reading what was actually said.
Was the output any good? Did the AI understand the customer’s intention? With your domain knowledge, your taste and your years of experience, can you spot the weakness in what it said?
The answer for most developers is no.
They have a surface level understanding of what’s good. They miss the nuance. They miss the gritty little details that make you world class at what you do.
You don’t. You are the reader who knows the article is wrong.
Evals are the dark arts of AI
Everyone wants to build the flashy thing, put the prompt in and see the AI respond. That is the easy work. That’s the 20% of the work.
The 80% of the work, the long yards, the late nights, comes from building evals.
Evals means building consistent quality metrics for your AI’s output. And this doesn’t need to be a technical job. You can do it with a spreadsheet. Read through the chat transcripts. Look at the failure modes. Note them down, classify them, and give yourself focus.
Then benchmark every change you make. Every prompt tweak gets a version and gets rerun. Every time you upgrade to the latest model, run it against the benchmark again, because the core brain of your AI is now different.
Did that actually improve things for your customers? Or did it just explode your costs and degrade the output into an embarrassing series of mistakes that are silently killing your relationships with every customer you have?
Without a benchmark, you’ll never know about it until it’s far too late.
Most AI projects fail to get a return on investment, and not because the system can’t do what it said it could. It’s because it does it at an inconsistent quality level. Sometimes it’s great, sometimes it’s poor, and most of the time it’s middling, because no one bothered to read the chat transcripts.
Without evals, quality is all over the board: sometimes great, sometimes poor, mostly middling. Read the transcripts, benchmark every change, and the spread tightens.
So here’s a test for whoever is building your AI. If they aren’t talking to you about benchmarks and evals, they have not deployed AI at scale. Because as soon as you do, you live and you die on the quality of your evals.
Don’t ignore this. It’s the long work, the hard work, and it’s the work that gets you the return on your AI spend.
Open the transcripts. Read them like the expert you are. Score every single one.
Nobody else in your business is going to notice the article is wrong.
Written by