Delivering a study is a long and complex process. There are thousands of little steps, each of which can introduce errors into the final scientific result. Even the best scientists sometimes trip up. Yet sponsors and regulators set a very high bar for execution - anything but perfection is a miss! This creates a lot of risk for service providers. Genuine mistakes can lead to lost custom or - at worst - trouble with regulators.
We regularly get asked about how we ensure the quality of AI’s output. Everyone has heard about AI hallucinations at this point and it is a frequent concern. It is a fair question - and we talk about that a lot more here. However, we believe that this is also the wrong way to think about it. AI, if anything, will help you improve quality, while also increasing throughput. You just need to use the AI to check your (or its) work.
What do we mean by QC?
In most CROs, Quality Control (QC) is the process of having either a scientist colleague and/or your study director double check your work. For example, if you ran a non-GLP bioanalytical assay, you would send your final report, quantification results, processing method, and any other assay context to the reviewer. The reviewer would then double check that you filled out the dilution series correctly, processed the data in the right way, and did not make any typos or copy+pasting mistakes. If there are any issues, the reviewer will send it back to you for amendment.
It is an important part of the process, and things get caught here all the time. But it also eats up a lot of time and headspace for scientists that could be doing more valuable things instead (like prepping the next batch!). Larger companies (esp. GLP labs) often even have entire teams dedicated to QC and QA. Suffice to say, it is costing service providers a lot of money to do fairly rote work. And ironically, many Pharma and biotech companies we talk to still complain about the data quality from CROs. A few even said that this was one of the main reasons they in-sourced a lot of their work recently.
Where AI fits in QC
The same loop as today, with the cross-checking done by machine
- 1ScientistDrafts the report and its data tables
- 2AICross-checks every value, table and conclusion against the source data
- 3ReviewerJudges a short list of findings, instead of hunting through every cellSends it back to the scientist if anything needs fixing
- 4Study directorSigns off
How AI can help
The big problem with QC is that it is boring work. It mostly consists of having multiple spreadsheets and Word docs open at the same time, and painstakingly cross-checking each cell. One study director we spoke to mentioned that they spend at least 1-3h per study, with about 15 studies per week. In each study, they have to scan through every table, every chromatogram and every PDF scan of hand-written notes; checking concentrations were copied correctly, SOPs were followed, and formatting rules were applied. They found this immensely boring, of course! And if there is one thing that’s clear, it is that humans will rush through a task if they are bored by it.
This is the big benefit with AI: it never gets bored. Tasks like these are tailor-made for AI: boring, repetitive application of rigid rules on big, messy datasets. AIs can sift through thousands of documents, methodically scan through every row, remember every rule in every SOP, and never just want to call it “good enough”.
What AI QC looks like
AI QC looks a lot like human QC: the AI looks at the final report and cross-checks it with source documents. Where it finds issues, it leaves comments explaining the issue, with suggested remediations. Whether you are working on a handful of batches in a discovery project or across thousands of documents in a GLP study, the AI processes all of it and surfaces to the scientist every issue it can find, ordered by severity.
But can you trust the AI to catch what your reviewers would? To test this, we ran a small controlled experiment. We took real Method Validation Reports (MVR) built from thousands of raw data files and PDF scans, injected them with realistic errors, and then asked both the AI and experienced bioanalytical scientists to find them. Across all the reports we tested, the AI found more of the issues and found them more reliably than the scientists did. And it did this at a fraction of the time (the maximum amount of time allowed for the scientist was 10 hours per report).
We planted 18 errors across three reports, in six categories. These are the kinds of errors we planted in each:
| Error type | Example |
|---|---|
| Copying error | Report says 762.4 ng/mL; the instrument export says 726.4 |
| Calculation error | Matrix factor CV reported as 3.2%; it is actually 7.9% |
| Wrong pass/fail call | “Passed at all levels”, but the LLOQ CV is 21.4% (limit 20%) |
| Wrong acceptance criterion | ±20% quoted where the validation plan sets ±15% |
| Overstated conclusion | Claims 180 days of stability; the data cover 92 |
| Inconsistent units | ng/mL in the summary, µg/mL in Table 1 |
AI caught twice as many planted errors
Share of the 18 planted errors found in a review
AI reviewed a report in under an hour
Average time to review one report, in minutes
Every error a reviewer misses is one that goes into the submission. Best case, the regulator spots it and asks questions, and approval slips. If an inspector finds it, you get a Form 483. If it’s part of a pattern, the study data itself gets questioned. Either way, it’s your sponsor’s timeline, and your reputation with that sponsor.
This also applies to non-GLP work. Recently, a bioanalytical CRO in the US used our platform to find a transcription issue that would have misreported a concentration to a client by 10x. Thankfully, the AI spotted this and raised it immediately with the scientist, and they were able to remediate before sending the data to the sponsor.
This looks great, but how do I do this?
Here, we will only talk about how to do this in a non-GLP context. GLP is its own topic for another day, as the regulated context requires a bunch more infrastructure to make this fit for purpose.
To be able to run modern agentic systems you need 1) access to LLMs via one or more model providers, and 2) an agent harness. Which of each is best for a particular task and what is the best combination of the two is changing constantly, but generally we recommend:
- The frontier models from OpenAI and Anthropic, and
- A generic open-source harness like Pi
To access the models, we recommend using a platform like Amazon Bedrock or Google Cloud Vertex. This is because they provide separation between the AI labs and your data. We generally discourage companies from using the AI labs’ APIs directly. We will go into more details on this in another post.
Then, you need to properly orchestrate your agents. One thing that is important in QC is that you run many agents in parallel. This is because different agents can find different things. Running replicates helps find more issues in total, and also helps weed out potential false positives. We’ve also found that using different models in those agents (particularly from different providers) can help with the diversity of findings (eg one type of model might put more weight on a different class of error).
In our experience, one of the trickier issues is making sure the agent has access to all the right context and the right tooling to handle that context. For example, a lot of the data for MVRs can be locked up in scanned PDFs. Processing those accurately and at the scale needed takes specialised tooling. There are providers out there, but we recommend reading the data privacy terms closely.
Of course, if you want none of this headache (and use this for your GLP studies) we built ScienceMachine for this purpose. We are already helping bioanalytical labs improve the quality of their studies and are constantly improving the system to provide the best QC in the world.
Conclusion
QC exists because everyone makes mistakes. But today it relies on the very thing that causes them: busy people doing repetitive work against the clock. In our experiment, experienced scientists spent around six hours per report and still missed most of the planted errors. The AI found twice as many, in under an hour.
That doesn’t make your reviewers redundant. It changes their job. Instead of hunting through every cell, they get a short list of findings, each with its evidence, and decide what matters before they sign off. The judgement stays with the scientist; the drudgery goes to the machine.
If you want to try it, start small. Run AI QC alongside your normal review on your next few reports and compare what each one catches. And if you’d rather not build it yourself, get in touch, as this is exactly what we built ScienceMachine for.