Skip to content
BC Consulting

BlogClinical Research posts

Where the Safety Limit Lives When an AI Agent Runs Lab Hardware

By Bryan Clayton15 min read

Originally published on LinkedIn.

  • ai
  • agents
  • gxp
  • validation
  • clinical-supply

How software has always been built

For decades, most business software has been organized as three layers stacked on top of each other. At the bottom sits the data layer, the database where the facts live: account balances, inventory counts, patient visit records. In the middle sits the business logic layer, the code that decides what is allowed to happen to those facts. On top sits the presentation layer, the screen the person actually looks at and types into.

A bank transfer shows how the layers divide the work. The screen collects your request to move five hundred dollars and passes it down. The logic layer checks your balance and the rules on your account, and either approves the transfer or rejects it. The database records the result. Each layer has one job, and none of them does another layer's job for it.

The screen is never trusted to enforce the rule. A well-built screen will gray out the transfer button when your balance is too low, and that is good manners, but it is not the control. The balance check runs again in the logic layer, below the screen and closer to the data, and that check is the one the bank actually relies on. If the screen misbehaves, or someone finds a way around it, the layer underneath still refuses the overdraft.

The systems you run every day are built on exactly this pattern. When a site coordinator works an IRT screen, the interactive response technology system that manages randomization and drug inventory for a trial, the screen collects the dispensing request. The logic layer checks whether the site actually holds that kit in inventory. The database records what was dispensed and to whom. If the coordinator enters a kit the site does not have, the refusal does not come from the screen; it comes from the inventory check underneath it, and no amount of clicking changes the answer.

Software engineers put the rule in the layer below the person for a plain reason. The layer closest to the data is the hardest to reach, the easiest to test, and the least dependent on anyone upstream behaving well.

Two old rules GxP already lives by

Software engineers keep those layers apart because of two design principles. The first is separation of concerns: each layer does one job and does not do the others'. The screen collects the order but does not decide whether the order is allowed. The rules layer decides but does not store the record. When each piece has one job, you can test it, fix it, and replace it without breaking everything around it.

The second is the principle of least privilege: each layer gets only the access it needs to do its job. The presentation layer can ask the business logic layer for something, and it can be told no. It cannot reach past the rules and rewrite the database directly. A layer that can only ask, never overrule, cannot cause the damage it was never trusted to prevent.

We have been building regulated systems on both rules for years, and not only in GMP manufacturing. They run through GxP broadly, and they are the working assumption of computer system validation, or CSV, the discipline of proving that regulated software does what its requirements say. Your stability chamber alarms when temperature goes out of range because its own control software enforces the range; the SOP only tells the operator what to do when the alarm sounds. The enforcement lives in the device, and the document above it describes what the device already refuses to allow. Your IRT refuses to dispense a kit the site does not hold in inventory, no matter what the coordinator types. The coordinator's screen can ask. The inventory check decides. And CSV is the reason anyone trusts that refusal: somewhere in that IRT's validation package is a script from user acceptance testing, or UAT, the pre-launch testing where people work the system against written acceptance criteria, in which somebody deliberately requested a kit the site did not hold, watched the system refuse it, and filed the refusal as evidence.

None of this was designed with AI in mind. It was designed so that the part of the system a person can talk to is never the part that holds the limit. The same two principles now decide what an AI agent should be allowed to touch inside a lab or a depot, and where the limit that stops it has to live.

The same pattern arrives for AI agents

On August 27, Anthropic opened a research preview of the Model Hardware Standard, or MHS, a shared specification that lets AI agents operate physical devices. An agent, here, means an AI system that takes actions rather than just answering questions. Strip away the novelty and this is the layer diagram from the top of this piece, applied to lab equipment: the safety limit lives in the layer closest to the device, and the model sits above it with the least privilege it needs.

The layer closest to the device is the driver, the small piece of software that translates general commands into the dialect one specific machine speaks. Your laptop has one for its printer: the application says "print," and the driver knows which printer is attached and how to talk to it. Lab instruments work the same way, and today those drivers are built one at a time. Anthropic describes integration work in automated labs that currently takes "weeks, if not months" of bespoke work by specialists, and says the standard reduces it to hours or minutes. It does this by giving every device a common interface built on a small set of primitives, meaning the basic operations every device supports. An agent can read a value, such as the current temperature inside a storage chamber, and write a value, such as a new setpoint, the target temperature the chamber is told to hold. Devices announce themselves in a standard format, so an agent can discover what is on the network without a custom integration for each one. Access runs through the Model Context Protocol, an existing standard for connecting AI systems to outside tools, or through a command line or ordinary code files, and the whole thing is model-agnostic, so it is not limited to Claude.

Alongside the code, each driver carries a set of tags describing the device in plain language: what it measures, what can be adjusted, and what safety limits are enforced. The tags compile into a reference file the agent reads before it touches anything. For the storage chamber above, the tags would record that it measures temperature, that its setpoint can be adjusted, and that any command outside two to eight degrees Celsius is refused. An operator can write the tags, or the agent can interview the operator about the setup and write them.

MarkTechPost, summarizing the specification, compressed the design into one line: safety limits live in the driver, not the prompt, the written instructions the AI is given. Map that onto the layers and the prompt is the presentation layer, the surface where requests are made, while the driver holds the rule the way the business logic layer always has. Picture a chamber holding investigational product at two to eight degrees Celsius. If an agent asks for a setpoint of twenty-five, the driver refuses the out-of-range command one layer below the model, and software at that layer cannot be talked out of it.

Side-by-side comparison of two software stacks. On the left, classic business software: presentation layer (the screen), business logic layer (the rules), and data layer (the record). On the right, an AI agent running lab hardware: prompt (the request), model (the operator), and driver (the enforced limit), with the instrument beneath. A band across both stacks marks the second layer from the bottom with the label "the limit lives here."
The bank's balance check and the driver's temperature check sit at the same level: one layer below whoever is asking.

Anthropic names Genentech, the University of Washington, Carnegie Mellon, HHMI Janelia, QuEra Computing and Tetsuwan Scientific among the early participants, alongside hardware vendors including Danaher, QIAGEN, Tecan, Automata, Doosan and Universal Robots. Several of those vendors already sit inside your GxP boundary. If your QC lab runs a Tecan liquid handler or a QIAGEN extraction platform, the question of how an agent talks to that instrument is not a 2029 question for you.

Two layers, and only one of them is a control

An MHS driver carries two different things, and they map cleanly onto the layers from the top of this piece. The enforced limit behaves like business logic. The driver will not write a value outside the permitted range; you can ask it to, and it refuses, and the refusal does not depend on the AI model driving the agent having understood the instruction, having been in a good mood, or having remembered everything it was told earlier in the session. The natural-language tag behaves like presentation. It describes the device to the agent, what the machine is for and what its constraints are, and that description is enormously useful. Nothing enforces it. If the tag on a refrigerated chamber says it holds between two and eight degrees Celsius, and the driver permits a write of twenty-five, the tag has been overruled by reality.

Inside one MHS driver, two panels react to the same event. The stated rule reads "the chamber holds between 2 and 8 degrees Celsius." The left panel, labeled documentation, shows the tag: plain-language text stating the range; when the agent writes 25 degrees, nothing intercepts it. The right panel, labeled control, shows the enforced limit: driver code that rejects any setpoint outside 2 to 8, and the same write is refused, with a UAT script recording the refusal as evidence.
Both carry the same rule. The write of 25 degrees passes the tag and dies at the enforced limit, which is the difference an auditor can test.

Anyone who has run a validation program has seen this distinction before, because a specification is not a tested control. Under CSV, you can only take credit for a control you can challenge and see fail safely. A statement in a document is a requirement. The requirement becomes a control when something in the system stops the out-of-range action and you have a test script proving it stopped. Clinical supply lives this constantly, and the IRT dispensing check from earlier is the cleanest example. The specification says a site cannot dispense a kit it does not have in inventory. The control is the inventory check in the code that returns an error. The evidence is the UAT script showing that somebody deliberately tried the forbidden dispense and got the error. The specification alone never satisfied anyone, and a tag file read by an agent is a specification.

So the useful question to ask of any MHS deployment, on day one and in every audit after, is which of your limits are in the driver and which are only in the tags. You can answer that question today, on paper, before an agent is anywhere near your equipment.

Draft Annex 22 draws the same line from the other direction

While Anthropic was publishing MHS, European regulators had already put their position in writing. The draft Annex 22 on Artificial Intelligence, released for consultation on 7 July 2025 alongside the revised Annex 11 on computerised systems, states that it does not apply to generative AI and large language models, the technology behind tools like Claude and ChatGPT, and that such models should not be used in critical GMP applications. Dynamic models that keep learning during use are excluded on the same terms, and so are models that can return different outputs from identical inputs.

Writing in Pharmaceutical Technology on August 13, 2026, Brian Drapeau argued that the exclusion may not survive industry pressure, that the EMA convened an expert workshop in mid-2026 to look at risk-based guardrails, and that final text is targeted for Q4 2026. He says regulators should qualify the whole human-AI workflow rather than trying to qualify the model in isolation, and that pharmaceutical manufacturing has decades of practice qualifying processes with variable components, analysts and analytical assays among them. Clinical supply already works this way. When an IRT goes live, nobody certifies the dispensing algorithm as a standalone component; the study team tests the whole workflow in UAT against written acceptance criteria, and a pharmacist verifies the kit before it reaches the patient.

I think that is right. Annex 22 says a generative model should not be the thing a critical GMP decision rests on. MHS puts the safety limit in the driver, one layer below the model, and gives the model, the layer above, only the privileges it needs. Those are compatible positions, because both keep the safety limit out of the model's layer. The model is in the operator's chair, and GMP has never validated an operator. It qualifies the equipment, trains the person, defines the procedure, and audits the record. Nobody has ever run an OQ, the operational qualification test that proves a machine performs as specified, on a human being.

The compatibility breaks the moment a team implements MHS with permissive drivers, drivers that accept whatever value they are given, and puts the real constraints in the tag text. That team has moved the control back into the model's reading comprehension and quietly recreated exactly the arrangement Annex 22 is trying to prohibit. The specification permits both implementations. Only one of them is qualifiable.

The cautious part is the part that is working

Anthropic reports that Claude "often stopped to wait for human confirmation before performing an action it deemed even slightly risky." In a research lab that reads as friction to engineer away. In a GxP environment it reads as the approval gate doing its job, the same job as the sign-off that holds a depot shipment at the dock until a named person releases it. The human release is one more layer above the model, doing one job: nothing proceeds past it until a person has looked.

I expect that pause to be the first thing under pressure in any regulated deployment, because a checkpoint that costs a shift's worth of output is exactly what a continuous improvement project targets in year two. Before that conversation happens, decide which actions require a human release, meaning a named person must review and approve before the system proceeds, and write down why, so the reason is on record when someone proposes removing the gate to save time. The reason is rarely that the agent is unreliable. Most gates exist because the action cannot be undone, or because its consequences reach a patient instead of a batch record. A mislabeled carton caught at the depot gets relabeled. A kit already dispensed to a patient cannot be pulled back from their medicine cabinet.

Anthropic is candid about the limits too. The same post says that because Claude learns about the physical world through text and images, "its spatial and physical reasoning have limitations," and it describes Genentech researchers having to guide Claude to recognize errors caused by foaming in protein samples. Access is a gated research preview, so a team cannot simply sign up and start using it. No clinical depot will have an agent driving a labeling line next quarter, this is not yet a product, and I am not arguing that anyone should accelerate. How you qualify a system whose vendor has built an AI agent into it is a question that arrives before the technology does. If your QC lab runs a Tecan liquid handler or a QIAGEN extraction platform from the partner list in that announcement, the next version of that instrument's software is where these answers will surface, and your validation plan for it is where the qualification question lands first.

What I would do in the next two weeks

Take one qualified instrument. A stability chamber, a liquid handler, a labeler, a temperature-monitored storage unit; whichever one you know best. Walk its parameters and sort every boundary into three buckets: enforced by the instrument's own built-in software (its firmware), enforced by the separate software that controls the instrument from a computer (the driver), or written in a document and enforced by nobody. Do it on paper, with no agent involved. What comes out is a layer map for that device, showing where each limit actually lives, and bucket three is an audit finding today, whether or not an agent ever touches the instrument.

Then hold four positions when a vendor arrives with an agent story:

  • Every limit that protects product, patient or data integrity is enforced below the model, in the driver or the instrument, and there is a test script that shows the out-of-range attempt being refused.
  • The natural-language tags are treated as controlled documentation with a change history, not as configuration nobody owns. A tag that drifts from what the driver actually enforces is a deviation, a documented departure from the approved process, and somebody should be able to detect it.
  • Human approval is required for irreversible actions, destroying returned kits and unblinding a subject are the classic cases, and for anything whose failure reaches a patient, and the reason for each gate is recorded next to the gate, so the removal conversation has to argue against something.
  • Qualification covers the workflow rather than the model. Named role, specific system version, defined use case, predetermined acceptance criteria. That is Drapeau's formulation, and it survives an inspection because it describes something you can actually test.
Four-step exercise for assessing one lab instrument before any agent touches it: pick one qualified instrument; sort every limit into three buckets (enforced by the instrument's own firmware, enforced by the controlling software, or written in a document and enforced by nobody); read the resulting layer map, where a document-only limit is an audit finding today; and hold four positions for the vendor conversation.
The whole exercise runs on paper in an afternoon, and bucket three pays for it: a limit that lives only in a document is a finding before any agent arrives.

Clinical operations has an advantage here that most software teams do not. Our work is full of questions with only two answers: a kit either exists in the depot or it does not, a chamber either held temperature or it excursioned, a randomization either matched the schedule or it did not. Those yes-or-no facts are what let a machine check its own work, because after every action something can compare what the agent did against a record it cannot argue with. When an agent gets access to physical equipment, the teams that will do it safely are the ones who already know how to write an acceptance criterion that a machine can fail.

The idea that safety limits belong in the driver and the firmware rather than in the model's instructions comes from the Model Hardware Standard, developed by Alek Kemeny at Anthropic and Arco Bast at HHMI Janelia, who built the approach behind it, a single shared record that instruments from different vendors read and write, while coordinating microscopy rigs. Drapeau's piece is the sharpest thing I have read on qualifying AI workflows under GMP, and it is worth reading in full rather than through my summary of it.

If your team is working through what agent access to validated systems should look like, this is the kind of question we work on in the BC Consulting AI bootcamp and in custom builds for eClinical teams. Both live at bcconsulting.io.

Sources

Working on something like this?

We run AI Bootcamps and build custom tools for sponsors and service providers in clinical research. A short call is the fastest way to find out whether your problem fits.

Talk About Your TeamBrowse the Catalogue