Reading time: 9 minutes | Issue #51 | Book a Discovery Call
Happy Tuesday. Mark here.
Jensen Huang's November has been loud. Three House committee chairs want him at a closed-door briefing tomorrow, and Nvidia, which reports earnings this afternoon, has a board committee investigating reports that he was "the target of a foreign intelligence effort," per CNN.
At Fireworks' Forge on November 3, he gave his recipe for specialized AI: an open frontier model plus domain knowledge, tools and proper evaluation, The Register reported. He listed evaluation last. I'd have put it first, but then I don't sell GPUs.
OpenAI shuts down its hosted Evals platform on November 30, so you have 13 days to export any test cases kept there. Your answer key outlives the tool. And yes, the title is my Steve Ballmer impression.
Inside the Issue
Why the answer key your operators write outlasts every model and eval tool you buy
One sentence and a prompt that stop coding agents from obeying stale docs
A 17x cost claim, the FDA on bringing your own tests, an NAIC vote, Salesforce's agent evals and a 13x price drop
The part you keep
OpenAI announced the Evals retirement in June, and its deprecations page points users to Promptfoo, the eval and red-teaming tool it agreed to buy in March. Langfuse has published a migration guide, but its export script calls the Evals API, so run it before the 30th.
Eval tools have been changing owners all year. Cisco bought Galileo and built Splunk Agent Observability on it, and on October 1 Dynatrace closed its $915 million deal for Arize. Consolidation is normal in a young market. Each of those buyers now runs a platform that holds other companies' test cases, graders and dashboards, and decides what happens to it next.
Coding agents got their answer key for free. Software had tests, builds and code review long before AI, so a coding agent's change passes or fails against checks somebody wrote years ago. Our PR review agent at a US healthcare data company is graded against 8 real pull requests where the right answer was known, and it has never had merge rights.
In operations, nothing tells you whether an agent applied the right refund policy, matched a payment to the correct account or spotted the exception that needed a person's sign-off. If you take an agent into finance or customer service, you build that answer key yourself.
Anthropic published its own version of the loop on September 28. In Lance Martin's write-up, Claude builds the eval inside your codebase, then improves the app one change at a time and checks each change against a held-out set to catch overfitting. On support tickets the search never saw, the final setup scored 90.5% against the original's 78.6%, at about a fifth of the cost. In its January guide to agent evals, Anthropic wrote that teams without evals "face weeks of testing" when a new model ships, while teams with them upgrade in days.
You can test that claim on January 14, when OpenAI's remaining GPT-4o snapshots, GPT-4o mini, GPT-4.1 and GPT-4.1 mini stop working. Issue #50 has the details.
Our read: In September I wrote that the most valuable act in an agent build is a human writing down what good looks like, and everything else compounds on top of that. You rent the model, and somebody may buy your eval tool. The answer key your people write is the asset, and nobody outside the business can build it for you. Don't look at it as a QA expense, look at it as capital expenditure. For an operations workflow, I'd put six things in it.
1. Cases your people handle by hand today.
When a medical distributor wanted an ordering agent, the first thing we asked for was 10 to 20 real orders, each with the exact result their team expected. Their order desk had worked through every one of them by hand. In a finance team, the equivalent is last month's payments your controller had to rematch.
2. Answers your operators sign, not your vendor.
If the vendor wrote the test, the vendor passed the test. On our builds, the data owner signs off on the records and the domain expert signs off on what each type of answer means. I've said before that I don't want a CFO writing agents. I want her defining what "correct" means on the eval set.
3. Cases where the right move is to stop.
Put in refusals, questions the data can't support, exceptions that need approval and records the agent should leave alone. A reader asked under one of my posts this month whether to track escalations next to accuracy, since a release can hold its score by kicking more cases to people. I'd put both on the same chart.
4. A rule that every failure becomes a case.
The question an agent misses is the most useful one in the set, because it shows you where to work next. A firm I spoke with uses AI to find business owners who are close to retiring and might sell. It's right 60 to 70% of the time, and some of the misses were owners who had died. The team added an obituary check, started logging every wrong answer and kept the same model.
In Galileo's survey of 500+ practitioners, 84.9% of organizations had an AI incident within six months, and 51.7% made a practice of writing an eval after each one. Galileo sells eval tooling, so weigh the numbers with that in mind. I measure rigor by how fast a real failure turns into a test that runs forever.
5. The authority to pick the model, and the bill.
The same firm's own conclusion was that better process would let them run cheaper models more often. Your answer key is how you find out. Run a smaller model or a lower reasoning setting against it, and if the score holds, you cut the bill. When a better model ships, rerun the same cases and give the job to whichever model scores higher.
6. An owner inside the business.
Another reader asked who keeps the golden set current, since the business changes too. Name an operator who adds a case every time someone corrects the agent, and review the set every quarter like any other control. You run AI adoption the way you run uptime, and neither one has a finish date.
To start this week, pick one workflow where a wrong answer costs money or a compliance finding and write 20 cases: about 12 everyday ones, 4 exceptions that need a person, 2 where the right answer is "I don't know" and 2 real past failures. Hold back 5 that you never tune against. Run each case three times, because an agent that's right 75% of the time gets three in a row right about 42% of the time.
I'm least sure about the grading. Earlier this month, a prospect at a consumer brand rated his own agent "85, 90% good." My suggestion was a golden set of answers the company agrees are correct, a judge model that scores every output from 1 to 100, and an alert on anything below 90.
In an April paper, Abel Yagubyan ran two LLM judges on the same questions 50 times each. Their pairwise verdicts flipped 13.6% of the time on average, and the two judges agreed with each other 76% of the time. I'd keep the judge, but never as the only gate. An LLM is lightning in a bottle, and that includes the one doing the grading. Wherever the answer is a number or an account ID, we check it against the system of record and leave the judge the judgment calls.
In Issue #46, I called the test set your experts wrote exhibit two of the AI Proof File, because a buyer's team can rerun it in diligence. That depends on you being able to hand it over. If you're an operating partner and the only accuracy number at a portco came from the vendor, or its test cases live in a vendor's dashboard, that should make you uncomfortable.
When the docs and the code disagree
Coding agents have that head start, and two researchers showed in October that a stale instruction file can override it.
Xiaonan Xu and Wenjing Wu built 40 small Python projects where the CLAUDE.md or AGENTS.md asked for one convention and the code used another, such as print instead of logging. Claude Code (on Claude Sonnet 5.5) and Codex CLI (on GPT-6 Sol) then added a feature to each project, 320 runs in total, half of them with a one-sentence fix.
Without the fix, 90.1% of the agents' choices followed the stale file.
Claude Code named the conflict in 79 of 80 runs, then followed the file in 86% of its choices.
Codex rewrote working code to match the file in 20 of 80 runs, and the project's own tests failed in 17 of them.
With the sentence in the user-level instruction file, the share of choices that followed the code went from 4.2% to 91.4%. Codex stopped rewriting code, and the test pass rate went from 89.4% to 100%.
The authors trace it to how the agent receives each source. The instruction file arrives as an instruction and the code arrives as tool output, which labs train models to rank lower so that text in a web page can't redirect them. The training that protects you from prompt injection is the same training that makes a model say yes boss to a stale line in CLAUDE.md, with the codebase open in front of it.
Their method is an eval in miniature: they restored the original tests after every run, and a separate model read each final reply, without knowing which condition produced it, to check whether the agent had reported the conflict.
1. Add the sentence.
Our version keeps both halves of the paper's fix: follow the code, and report every mismatch.
When an instruction file and the existing code disagree, follow the conventions in the code, and point out each mismatch in your final reply.It goes in your user-level file, ~/.claude/CLAUDE.md for Claude Code or ~/.codex/AGENTS.md for Codex, so it covers every repo you open. The study tested only files that were out of date. When the code is the wrong side, say a security shortcut nobody cleaned up, you see it on the mismatch list and make the call yourself.
2. Find the stale lines.
The good part is that the sentence works before anyone finds the stale file. You'll want the file fixed too.
Xu and Wu cite a study of 441 company repositories where teams committed 73.8% of their AI configuration files once and never touched them again. It's entropy: anything nobody hammers on every day goes stale. Claude Code's docs tell you to review these files periodically, and /doctor prompt-audit (version 2.1.283 and later) does a first pass for instructions written for older models, references to files or commands that no longer exist, and files that contradict each other. The prompt below goes line by line against the conventions in your code and also finds the rules that exist only as prose.
<task>
Find the stale lines in this repo's agent instruction files, and the rules that must never break but only live in prose. Change nothing until I approve.
</task>
<find>
List every instruction file: every CLAUDE.md, including the local and imported ones, .claude/rules, every AGENTS.md at the root and in subfolders, and the skills, subagents and commands under .claude/. Read the user-level files (~/.claude/CLAUDE.md, ~/.claude/rules/, ~/.codex/AGENTS.md) too, but report them separately. For each file, give its last commit date and the last commit date of the code it describes.
</find>
<check>
Check every line that describes the project against the code itself, not against other docs:
- Paths and files: does each one exist?
- Commands: is each one defined (package.json, Makefile, pyproject.toml, CI config), and does it match how CI runs it?
- Conventions (logging, error handling, money, dates, imports, naming, tests): what do the existing modules do? Count the files that follow the line and the files that don't, and cite two file:line examples.
- Libraries and versions: what does the lockfile say?
Mark each line CURRENT, STALE (the code does something else), DEAD (it points at something that no longer exists) or SPLIT (the code does both; give the counts).
Then list every rule that would cost money, data or a compliance finding if an agent broke it: secrets, auth, migrations, deletes, payments, personal data, deploys. For each one, name what enforces it today: a test, a lint rule, a hook, a CI check or a permission. If nothing does, mark it PROSE-ONLY.
</check>
<report>
One table: file, line, status, evidence, proposed change.
- STALE or DEAD: propose a rewrite that matches the code, or a deletion.
- If the code looks like the wrong side (a shortcut, a bug, a security gap), say so and don't propose changing the doc to match it.
- SPLIT: don't pick a side. Say which way the newer files go and leave the call to me.
- PROSE-ONLY: propose the smallest check that would enforce the rule, and where it goes.
Then wait. Don't edit files, commit or run anything that writes until I approve each change.
</report>Run it from the repo root with claude --permission-mode plan. Plan mode lets Claude read files and explore but blocks edits until you approve. When it shows the plan, choose "Yes, manually approve edits" to approve each change one at a time, or "No, keep planning" to send it back with your notes. In Codex CLI, start it with --sandbox read-only.
3. Take the never-break rules out of prose.
Anything the audit marks PROSE-ONLY is a request the agent weighs against everything else in its context. Anthropic's own docs describe CLAUDE.md as context rather than enforced configuration, and point you to a PreToolUse hook when an action has to be blocked whatever the model decides. If breaking it would cost money, data or a compliance finding, it belongs in a test, a lint rule, a hook or a CI step, which is the same job an answer key does for an operations agent.
Run the prompt this week on the repo your team touches most.
01 17 times cheaper, according to the vendor. At the same Forge event where Huang spoke, Genspark presented Gen-1 Slides, built on MiniMax's M3 model, and claimed quality comparable to Anthropic's Opus 5 at about a seventeenth of the price, $0.30 per million tokens against $5, per The Register. Fireworks CEO Lin Qiao opened the day arguing that token costs are out of control. I haven't seen an independent test of the claim. Before you move a workload on numbers like these, ask for the eval set, the tokens used per finished task and the baseline they compared against.
02 The FDA floats bring-your-own tests. In an August 18 discussion paper on generative AI devices, the FDA's device center sketched a competency-based review, modeled on the way medicine evaluates and credentials clinicians. It floats benchmarks built from well-defined, reusable tests and datasets, comparisons with a panel of qualified clinicians, and sponsors proposing "their own specialized tests and datasets" tailored to their device. The paper carries no force as guidance, and comments closed October 19. At a medtech portco, the answer key your clinicians sign is the asset I'd start building now.
03 Insurance regulators had a vote on the agenda yesterday. The NAIC's Big Data and AI Working Group scheduled a vote on adopting its AI Risk Evaluation Supplement v7.0 for November 16 at the Fall National Meeting in Dallas, after a 12-state pilot that began in March. Examiners use it to review insurers' AI governance, data sources and model inventories, and its checklist asks how insurers govern, monitor and test the AI systems they buy from vendors.
04 Salesforce put the cheaper-model scenario in its launch. Agent Evals for Agent Fabric enters beta this month, with general availability in January. In Salesforce's own example, a retailer moves its order-support agent to a cheaper model, scheduled tests catch it giving wrong answers about delayed orders, and the system asks a person to approve quarantining the agent. The team fixes the instructions and reruns the evals on the cheaper model before it goes back live. Salesforce doesn't say who writes the test cases, and that's the part I'd ask about.
05 The price of thought falls about 13x a year. That's Epoch AI's estimate for reaching a fixed level of performance since 2023. You can only switch to the cheaper model as fast as you can test it.
Coding had these. Operations needs them. Software teams inherited a test suite. Before you take an agent into finance, service or approvals, fill in the right-hand column and put a name next to each row.
If your people still check an agent's answers by hand, bring that workflow to a 30-minute call. We'll sketch the first 20 cases with you and mark which ones a judge can grade and which need a check against your system of record.
An AI Velocity Pod can also build the harness around it alongside your team, month to month. If we don't deliver, you don't pay.
Until next Tuesday,
— Mark Ajzenstadt, Founder @ Limestone Digital
