Who Checks the Checker When AI Agents Are Tested
By Bryan Clayton12 min read
- ai
- agents
- validation
- gxp
- evaluation

TL;DR A new study found an AI agent benchmark's grader was writing to the data it checked, and fixing it cut measured success from 94.3% to 65.5%; IRT data control teams have long run the kind of independent check these evaluations need.
When I first entered the IRT and RTSM industry over 15 years ago, the company I worked for had a data control team. Its job was to run SAS checks against the live study database on a schedule, for the life of the study. The application that randomized patients and dispensed kits was built from the requirements. The SAS checks were written from the same requirements and along with the protocol, but separately from the system development. They also read the data at rest rather than inline during the application transactions. A typical rule: a patient randomized to treatment group A receives drug kit type 1 and only drug kit type 1, so the data at rest for group A should contain nothing else.
Before the checks ran against a live study, the data control team tested them in a separate environment where incorrect data had been seeded into the database on purpose, to confirm the checks caught the inconsistencies they were built to catch.
I recall a study where the checks flagged a patient holding a kit of a different treatment type. We opened a CAPA, a corrective and preventive action, to find out why. The mismatched arm-to-kit pair came from an approved, direct SQL database update. The site had pulled a kit off the shelf and given it to the patient instead of dispensing it through the IRT. This was a protocol deviation: the site had given the patient the wrong treatment. The database still had to record the kit the patient actually received to support data integrity, so the record was accurate. The biostatistician and study team would have to explain the deviation in the clinical study report, and a case like that generally means the patient is excluded from the analysis population. The application never wrote that record, so no test of the application would have found it. A data control check that was independent of the application, reading the data against the requirements, did.
AI agents are now tested by software that plays the part data control played: it reads the end state and decides whether the agent got it right. A paper posted to arXiv on October 12 by Junjie Chen asks what happens when that software is not independent of what it checks. The answer, for one real agent benchmark, is that the agent looked nearly thirty points better than it was. I want to walk through how that happened, because the mechanism will be familiar to anyone who has validated a regulated system, and because it changes what you should ask the next time a vendor shows you a success rate.
What an evaluation harness is, and what an oracle does
When an AI lab or a vendor says an agent completes 94% of tasks, that number comes from an evaluation harness. Chen describes the harness as the software that runs the test suite, generates the inputs, and decides whether each attempt passed or failed. A single attempt is called an episode: the agent is given a task, such as changing a booking in a simulated airline database, and it works through the task by calling tools until it stops.
The part of the harness that decides pass or fail is called the oracle, a term borrowed from software testing. In the benchmark Chen studied, the oracle looks at the state of the database when the agent finishes and compares it with the state it should be in. If the records match, the episode passes. This is a sensible design. It grades the agent on what it did to the data rather than on what it said it did, which is the same instinct behind checking the audit trail instead of the agent's summary.
The oracle is still code, though, and code can be wrong. Chen's paper builds an audit for the oracle itself, a way of checking the checker. It has two parts. The first is five checks that the recorded data hangs together, the way a bank statement should add up; Chen calls them state-integrity invariants, meaning things that must always be true about the data. The second is three known-answer controls, which are test cases where the correct grade is known before the oracle sees them, like a teacher grading a quiz when she already has the answer key. To prove the audit could catch problems, Chen deliberately planted seven classes of oracle defect, which are seven different kinds of mistakes, across two task domains and a synthetic one (a made-up test setting). The audit caught all seven, and once the oracle was repaired, it raised no false alarms, meaning it never flagged a problem that was not there.
The grader that touched the data before grading it
Then Chen pointed the audit at a real agent benchmark, and it found a defect nobody had planted. The benchmark's reward function, the code that computes the score, changed the state of the environment before the post-run scoring took place. In plain terms, the grading step was writing to the database it was about to inspect.
Data control's checks could be trusted because nothing they did changed the data they were reading. The benchmark's scoring code had no such separation. It was supposed to measure what the agent did to the database, but it had already written to that database, so the score partly measured the scoring code's own changes.
Chen corrected for the defect by replaying the recorded episodes against an untouched environment. Across 4,320 episodes, the number marked as passing on the database check fell from 4,194 to 2,883, and the measured success rate fell from 94.3% to 65.5%. The agents had not changed. Only the grading had.
The same paper has two more findings. In a separate run of 3,240 episodes, Chen varied the text surrounding the agent: the policy documents it was given, the descriptions of its tools, and the wording of the error messages those tools returned. All three changed how reliably the agent behaved. Tool descriptions had the largest effect in that exploratory analysis, about 9.2 percentage points, and error strings the smallest, about 5.3. Chen is careful to note that in a second task domain the direction of the language effect reversed, so the result holds only for the configuration tested.
The last finding concerns the record itself. Chen tried to apply the same audit to a second, independently built benchmark and could not, because that benchmark's correct end state could not be reconstructed from its recorded log of API calls. The record was not complete enough to re-grade.
This is a validation problem clinical teams already know
None of this is exotic to anyone who has worked under GxP. A grader that alters the thing it measures is a test step with uncontrolled preconditions. A configuration change that moves reliability by several points is a change that should have gone through change control and regression testing. A log that cannot rebuild the final state is an audit trail that fails the "complete" in ALCOA+, the attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring and available standard for trial data.
The regulators are already pointing in this direction, if not in this detail. FDA's draft guidance on using AI to support regulatory decision-making for drugs and biologics, issued in January 2025, lays out a seven-step credibility framework that runs from "Define the question of interest" through "Develop a credibility assessment plan" to "Determine adequacy for the context of use." As Amie Harpe of Sakara Digital noted in September, it is still marked as a draft and FDA has not committed to a date for a final version. In January 2026, FDA and EMA published ten shared principles for AI in drug development; Andy Studna's summary in Applied Clinical Trials lists proportional validation, a clear context of use, and lifecycle performance monitoring among them, while noting the principles are "not prescriptive requirements." Neither document tells you to audit the grader. Both assume that the evidence of performance can be trusted, and Chen's paper is a reminder that the evidence is itself a system that needs validating.
The clinical field also has an advantage here that most software teams lack: we have built this function before. The data control team I described at the start did three things Chen's paper argues AI graders need. It wrote its checks from the requirements and the protocol, independently of the application. It tested those checks against data seeded with known errors before trusting them, which is what Chen's known-answer controls do. And it kept running them against the live database for the life of the study, the same idea as the lifecycle performance monitoring in the FDA and EMA principles. Chen's paper amounts to an argument that AI agent evaluations need a data control function.
Agents are already moving into trial operations. At the Rethinking Clinical Trials Grand Rounds on October 9, Christoph Hornik of the Duke Clinical Research Institute and Moulik Shah of Maxis AI presented Project Loom, a proof-of-concept that ran a trial's operational work through agents with human oversight at each step. They reported that the agents succeeded in four of five operational domains, with failures in areas including eligibility screening and chatbot handoffs, and that document drafting, case report form generation, and backend data extraction looked ready for live use under human review. I have no reason to doubt their grading. My point is narrower: "four of five domains succeeded" is exactly the kind of statement whose meaning depends on how success was decided, and the clinical sponsors who adopt tools like these will be the ones asked to show that.
How I would test an agent before it touches a GxP record
These are the rules I would apply, whether the agent is something you are building or something a vendor is selling you.
-
Ask for the grader, not the score. When a vendor quotes a success rate, ask how pass and fail were determined, who wrote that logic, and whether anyone audited it. If the answer is that the model judged its own output, or that another model did, you have learned something important about the number.
-
Seed known answers into every evaluation. Include cases where the correct outcome is known and cases built to fail. A grader that passes a case you built to fail is broken, and you want to find that out in UAT rather than in an inspection. This is what data control did with its seeded test environment, applied to the checker.
-
Grade from a state nothing else can touch. Capture the end state from the system of record, the IRT or EDC audit trail, before any cleanup, reset, or scoring code runs. If the harness and the system share a database, assume the harness can write to it until you have shown it cannot.
-
Keep a log complete enough to re-grade. If you cannot rebuild the final state of an episode from what you recorded, you cannot re-score it when a defect in the grader turns up later, and Chen's experience suggests one eventually will. This is the same test you would apply to any audit trail: could an inspector reconstruct what happened from it alone?
-
Put the agent's instructions under change control. Tool descriptions, error messages, and policy text moved measured reliability by several points in Chen's study. Treat a rewritten tool description the way you would treat a changed configuration parameter in an IRT build: document it, assess it, and rerun the regression set.
The fair objection
Before anyone accuses me of building a governance program on one preprint: this is a single-author paper on arXiv, not yet peer reviewed, studying a specific benchmark under specific conditions. Chen says plainly that the language effect flipped direction in a second domain and that the method did not carry over to a second benchmark. It does not show that every published agent score is inflated by thirty points, and I would not repeat it that way.
What it does show is that the measurement layer for AI agents has the same failure modes as every other measurement layer, and that a careful audit with known answers can find them. That is a modest claim, and modest claims are the ones that survive an inspection. A related paper by Srimonti Dutta and Akshata Kishore Moharir makes a parallel point from another direction: on a text-to-SQL benchmark, they found that between 45.8% and 59.1% of answers that matched the reference, depending on the method, were not backed by a valid computation trace. A correct-looking result and a correctly produced result are different things to grade.
I use AI agents every day in my own work, and I expect them to keep getting better. The question this paper raises for me is not whether agents are good enough. It is whether the instrument we use to answer that question has been validated.
Something to try this week
Take one AI-assisted step in a workflow you own, whether a vendor tool or an internal pilot, and write down how its output is currently judged correct. Then build two cases: one where you know the right answer and one deliberately built to be wrong. Run both through the same check. If the wrong case passes, you have found your first oracle defect, and you found it on paper rather than in a submission.
Credit where it is due
Junjie Chen's paper did the careful work here, and it ships with an offline replication package that runs without API keys or model calls, which is a courtesy more authors should extend. Christoph Hornik and Moulik Shah's Project Loom presentation is worth the time of anyone planning agentic work in trial operations, and the slides are public.
Working on something like this?
If your team is deciding how to test an AI agent before it goes anywhere near a GxP record, I am happy to talk it through on a short call at https://bcconsulting.io/book. BC Consulting's courses on building and validating AI workflows for regulated teams are at https://lms.bcconsulting.io/catalogue.
Sources
Junjie Chen, "When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark," arXiv:2610.16067, October 12, 2026, https://arxiv.org/abs/2610.16067 Srimonti Dutta and Akshata Kishore Moharir, "Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems," arXiv:2608.26036, 2026, https://tldr.takara.ai/p/2608.26036 Rethinking Clinical Trials, "Grand Rounds October 9, 2026: An End-to-End Agentic AI Platform to Automate Clinical Trials: A Proof-of-Concept and Evaluation Study (Christoph Hornik, MD, PhD, MPH; Moulik Shah)," NIH Pragmatic Trials Collaboratory, October 2026, https://rethinkingclinicaltrials.org/news/grand-rounds-october-9-2026-an-end-to-end-agentic-ai-platform-to-automate-clinical-trials-a-proof-of-concept-and-evaluation-study-christoph-hornik-md-phd-mph-moulik-shah/ Amie Harpe, "FDA AI Credibility Guidance Still Draft: What to Do Now," Sakara Digital, September 5, 2026, https://sakaradigital.com/blog/fda-ai-credibility-guidance-still-draft-what-to-do/ Andy Studna, "FDA and EMA Align on Ten Principles to Guide Artificial Intelligence Use in Drug Development," Applied Clinical Trials, January 29, 2026, https://www.appliedclinicaltrialsonline.com/view/fda-ema-align-ten-principles-artificial-intelligence-use-drug-development