How to keep your sponsor data safe in the AI era

We talk to a lot of CROs that want to deploy AI into their day-to-day operations. Whether it be to process data, write reports, or QC regulatory filings, everyone is thinking about how to become more efficient with AI. The one big concern for everyone at the same time, though, is data security.

Service providers are rightfully nervous about letting a bunch of AI agents crawl all over their sponsor’s data. Where is it going? Who has access? And is it being trained on? The sponsor has entrusted you with one of their most valuable assets and you just cannot risk this ending up in the wrong hands. And yet this creates a tension, as you have to become more efficient because competition from China is intensifying.

In this post, we go into how you can keep your sponsor data safe, while enjoying all the benefits of AI. First, we talk about the real security risks of AI in the CRO space. Then, we cover the different ways to address each.

The risks

For all its benefits, there are very real risks with AI. Broadly, they come in two forms: intrinsic risks and extrinsic risks. Intrinsic risks are the risks arising from using the system itself, which can be either intentional or unintentional. Extrinsic risks are risks posed by outside AI systems, and you will be exposed to those whether you personally adopt AI or not. So even if you choose not to use AI for your operations today, this blogpost should still help you think about your security posture in the age of AI.

1. Are my data being trained on?

One of the most common concerns of CRO leaders is whether the data are being trained on. It is a fair concern, as information learned during training could be regurgitated by an AI when someone is prompting it. For example, the AI could memorise the name and compound of the sponsor, the associated tox or efficacy data, and then spit them back out to one of their competitors. Not great!

This is an intrinsic risk. It only exists for companies using AI. And because of this - and companies’ justified concern about this - most commercial agreements with AI model builders explicitly forbid them from training on your data. However, there are persistent rumours that labs train on summaries or synthetic rewrites of user data, which most no-training clauses don’t explicitly cover. How similar or different those data truly are from your real data, is unclear. This means that this risk is also an intentional part of the AI labs’ systems.

2. Are my data leaking anywhere?

This is largely the same concern people had in the cloud era: where are my data going? How can I make sure they’re not strewn all over the internet? It is a fair concern now, as it was then, but I would argue that this has actually become a bigger risk now with AI.

Most AI agents you can get off the shelf today have extensive access to the internet and your computer’s file system. There have been recent cases where AI agents went rogue, uploading data to external systems without the user even knowing about it. Generally, the more autonomous the agents the more useful they are, but also the harder it becomes for the scientist to understand what is happening and where the data is going. What does not help is that most generic AI agents make interrogating this actively harder by hiding agent actions or model reasoning (for UX or anti-competitive reasons).

This is an intrinsic, unintentional risk. This risk only applies if you are using AI, but it is effectively a bug in the AI that it goes off the rails. Fixing this bug in the AI itself is challenging and an active area of research.

There is also a much more mundane version of this risk. The most common leak today is not an agent going rogue. It is a scientist pasting a results table into their personal ChatGPT account because it saves them half an hour. Most companies have more of this “shadow AI” than they think, and it is the worst of all worlds: consumer accounts may train on what is pasted in, and there is no record of what left the building. Ironically, this risk is highest at companies that have not adopted AI officially. Your staff already know how useful these tools are. If you don’t give them a safe one, they will use an unsafe one.

3. Am I safe from cyber threats?

We mentioned the intrinsic risks of AI, now it is worth mentioning the extrinsic risk of AI. AI is at this point better than most humans at code, and as a consequence, extremely good at cyber security. This cuts both ways, unfortunately. The same models that can help us protect ourselves are also the ones that make attackers significantly more capable. Just in the last few weeks, a huge spike in Common Vulnerabilities and Exposures (CVEs) was reported and many systems had to be patched frantically to avoid massive data leaks. The attacks also seem to be accelerating, with cyber security providers reporting millions in attempted hacks.

It cannot be overstated how serious of a risk the AI cyber threat is. You should expect a 10-100 fold increase in attacks. As we wrote elsewhere, AI never gets bored. Which means it will be extremely persistent and keep going until it has found a way into your system. And because it knows code better than any human, it will know how to exploit it better than any human also.

The remediations

If you have come this far, you might feel that the picture is pretty gloomy. But thankfully, many smart people have thought about these problems already and have found solid solutions.

1. Not training the AI

To avoid having (versions of) your data train the next generation of model, you have three options:

1. Negotiate a Zero Data Retention (ZDR) contract with the AI model builder. ZDR forces the AI lab not to store any of your data on their systems; their API becomes effectively stateless. That is a real step up from standard commercial terms. But it is still a promise from the company that builds the models, and if the rumours above are true, data derived from yours may fall outside it. ZDR contracts are also hard to come by unless you are a larger customer.
2. Use a third-party model provider. Big cloud providers like Amazon Web Services and Google Cloud host versions of the frontier labs’ (and others’) models on their platforms. This gives a degree of separation between the model builder and your data. No data of yours goes back to the model builder, and so they cannot train on it. The added benefit of these platforms is also that you get access to models from all builders, including open source ones.
3. Use open-source, on-prem. The final option is to run open-source AI models on your own hardware. This means that you control whether any training happens. The upside is that you have more control. The downside is that you have to purchase and maintain the hardware needed to run the models (which is expensive and hard). The other, non-obvious downside is that open-source models often have much weaker guardrails than the closed ones. This means that they make risk #2 (AI doing something it should not) more likely, without strong sandboxing.

Generally, we recommend option #2. It is the only option that removes the risk by design, rather than limiting it by contract: your data never reaches the model builder, so there is nothing for them to train on, derived or otherwise. It is also easy to set up, best-in-class in terms of security, and gives you access to all the world’s best models without having to deal with the maintenance.

2. Not leaking data

While better models from US frontier providers (OpenAI, Anthropic) reduce the risk of data leakage, they don’t fully eliminate it. This is because these are probabilistic systems and completely getting rid of these “unaligned” behaviours is an open research question. As it stands today, the only solution is solid agent sandboxing. Thankfully, larger and more specialised cloud providers are now offering easy-to-use, pay-as-you-go products for effective agent sandboxing. Modal is our favourite, but others exist also.

Sandboxing is basically a box that you put your agent in, and you can control what the agent can and cannot see or do in the outside world. For example, we recommend that you turn off internet access to the sandbox by default. This eliminates the risk of data exfiltration, as the AI can no longer upload data to any external sites.

The strength of the sandbox (the fine-grained controls) are also often its weakness. It takes time to fine-tune the permissions for the task at hand, such that the agent can access the right context, safely, while also stopping it from doing bad things. If you are trying to automate reporting, for example, you will still need to give the agent access to your files and SOPs somehow. But you probably don’t want it to browse Reddit.

Sandboxing only protects you from the agents you deploy, though. The fix for shadow AI is less technical: give your staff a sanctioned AI tool, set up as described above, and make it the easiest option. Then block or monitor consumer AI sites on work devices. Banning AI outright does not work. It just moves it onto personal phones.

3. Being safe from cyber threats

Even if you choose not to adopt AI for your operations today, we hope that you will adopt some of the suggestions here to secure your systems. As we mentioned before, the AI cyber threat is to be taken extremely seriously and you should assume that your systems are currently insecure. Here we recommend a few policies and tools you should adopt to harden your systems:

1. AI security scans. Most AI coding products now ship with security scans that check your codebase for any obvious vulnerabilities. Give your IT team premium seats to Claude Code, Codex, Devin or Factory to thoroughly scan your internal infrastructure for holes.
2. AI penetration testing. Adversaries will be using abliterated models to get into your systems. So why not beat them to it and try to get into your systems yourself with the same weapon? Tools like Strix, coupled with abliterated versions of open-source, can simulate real attackers on your system. This will help you find even more holes - hopefully before your attackers do!
3. An AI agent on vulnerability watch. AI has collapsed the time between a vulnerability being published and it being exploited, so patching once a quarter no longer cuts it. Set up a coding agent on a schedule (e.g. a nightly cronjob) that checks your systems and dependencies against newly published CVEs and opens a suggested fix for each one it finds. Your IT team then only has to review and approve, and the window attackers have to exploit a known hole shrinks from months to days.
4. Round-the-clock detection. AI attackers do not keep office hours. Endpoint detection with a managed 24/7 monitoring service is now affordable for mid-sized labs, and it catches what slips past everything above.

How this interacts with GLP: AI means that cyber security has become more dynamic than ever and patches need to happen more frequently and faster than ever. This fundamentally creates tension with GxP requirements for computer system validation (CSV). Even with today’s risk-based approach, every change to a validated system still needs its own impact assessment, risk assessment, testing and documentation, reviewed and signed off. This quickly becomes untenable, as updates need to happen on a rolling basis to avoid security vulnerabilities.

As you may anticipate, we suggest using AI to help with this also. In our own work, we’ve found AI agents to be quite good at this type of testing and documenting. Unfortunately there are currently no good out of the box solutions for this that we are aware of (except our own). How to do this in practice, we will cover in a separate article in the near future.

Conclusion

AI does bring new risks for CROs, but none of them are unsolvable. The training risk is solved by how you set things up: use models hosted by a cloud provider, so your data never reaches the model builder. Leaks are solved by putting your agents in a sandbox, and by giving your staff a safe AI tool so they stop reaching for unsafe ones. And the cyber threat is best met with the same weapons the attackers are using: AI that scans, tests and patches your systems faster than any human team could.

The one thing that doesn’t work is opting out. Attackers are already using AI whether you are or not, and your staff probably are too. Not adopting AI doesn’t make your sponsor data safer. It just means the AI touching it is the kind you don’t control.

Sponsors will increasingly ask how you use AI, and how you keep their data safe while doing it. The CROs that can answer both questions well will be the ones that win the work.

If you’d rather not build all of this yourself, this is how we built ScienceMachine: models hosted by major cloud providers, agents running in locked-down sandboxes, and every agent action logged so you can see exactly what happened to your data. Get in touch and we’ll walk you through it.