Protecting Patients from AI That Was Never Built

There is a clinic in Ohio where a nurse drew fourteen vials of blood from my arm and, three weeks later, an AI told me what all of it meant. 

The AI knew my results, my family history, my medications. It did not hallucinate. It did not pretend to know things it didn’t. When something was outside its knowledge, it said so and pointed me to the physician. It had no connection to outside services - no internet access, no external databases. A closed, careful, bounded system. 

I did not undergo the needle-in-the-arm procedure purely for the cholesterol data. At that time, AI startup I am co-founder of was working with a Croatian private health institution that had approached us to build something similar. 

The $365 blood panel was, in retrospect, the most cost-effective market research I have ever done. 

What I learned, once we got back to the technical design, is that the EU AI Act had a rather strong opinion about our project.

The Compliance Paradox

The system we were designing analyses blood biomarkers and generates personalised health recommendations. Under the EU AI Act, that lands it in Annex III: high-risk AI. High-risk AI requires a conformity assessment.

A conformity assessment, for those who have not had the pleasure, involves: technical documentation running to hundreds of pages, a risk management system, a post-market monitoring plan, EU database registration, and ongoing compliance obligations for the lifetime of the system. 

For Microsoft, this is paperwork. For a four-person startup, it is a second (and third) job - one that requires external legal consultants whose rates are identical regardless of whether their client is a four-person team or a multinational. 

The AI Act does include provisions for SMEs - Article 62 promises simplified documentation, reduced conformity assessment fees, and priority sandbox access. These are welcome in principle. In practice, the core conformity assessment obligation remains the same regardless of operator size. A simplified form is still a form that requires the same underlying risk analysis, the same technical documentation substance, the same ongoing monitoring. The scaffolding is lighter; the building is the same size. 

The Clause, characteristically, was unmoved by this distinction. 

This would be merely expensive, except that there is a second layer. 

A system that analyses blood results and generates health recommendations may also qualify as medical device software under Regulation (EU) 2017/745 - the Medical Devices Regulation. 

“May also” is doing a lot of work in that sentence. 

In practice, AI Act compliance and MDR compliance are cumulative, not alternative. Two separate regulatory regimes. Two separate bureaucratic processes. Neither calibrated to the size of the operator. 

The system we conceived was a fully closed system - local Mistral model, local databases, no external connections, knowledge base limited to biomarker information and what the user disclosed in an interview. A system specifically designed to minimise risk.

And to be clear: even a closed system can generate incorrect recommendations, which is why ours was designed with a physician-in-the-loop and a scope limited strictly to what the knowledge base actually contained. 

The obligation to go through the full conformity assessment process, however, was the same as if we had built an unconstrained system with no safety architecture at all. 

There is currently no Croatian patient receiving this kind of personalised, AI-assisted blood biomarker analysis from a local provider. This is not because the AI would have harmed them. It is because we ran the numbers - and there was no AI yet to help us with those. 

The system in place was not protecting patients from bad AI. It protected them from AI that was never built.

Who Is Actually Paying for Data Sovereignty?

Here is the part that is genuinely interesting from both a technology and a policy perspective. 

Our startup’s clients are municipalities, public bodies, SMEs, agricultural cooperatives. Many of them cannot send their data to American cloud providers due to GDPR special categories, data sovereignty requirements, and public procurement rules. 

These are not edge cases. And, not of lesser concern, these are also the typical EU public sector and SME clients. 

So they come to us and we recommend what EU digital policy recommends: local infrastructure, local Mistral deployment, data stays in the client’s own environment. Privacy by design. Data sovereignty. Exactly the independence from non-EU providers that European policy has been promoting for years. 

And then... 

The AI Act applies the same compliance requirements as if we had used OpenAI. 

Meanwhile, the large cloud providers - most of which are not EU companies - have a structural advantage we cannot replicate at our scale: they spread compliance costs across millions of deployments. Their marginal compliance cost per client is essentially zero. And they solve GDPR by having clients sign standard contractual clauses on onboarding. GDPR becomes the client’s problem. The provider keeps the revenue. 

An EU startup building local AI pays twice before compliance even begins: once for the hardware, once for the local model. Then it faces the same AI Act cost as operators who paid neither. 

The regulation designed to protect EU digital sovereignty is, structurally, subsidising the cloud providers it was meant to constrain. I do not think this was the intention. But intentions and outcomes are different things - and The Clause, naturally, benefits from the gap between them. 

The Sandbox Paradox

The EU AI Act includes a regulatory sandbox. The sandbox is designed to let innovative systems operate in real conditions with regulatory guidance before full compliance applies.

This is - and I want to be clear here - a good idea. In principle, it is exactly the mechanism start-ups (like ours) should be using. 

However… 

Accessing a national sandbox requires a formal application: legal framing, technical documentation, risk assessment. The provisions promise streamlined procedures and pre-deployment support for SMEs. In practice, the application itself still demands a level of regulatory literacy and documentation capacity that most start-ups or other types of SMEs simply do not have in-house. 

There is a word for this. Several, actually. “Paradox” is the polite one. 

The Clause prefers “necessary complexity”. 

What Would Actually Help?

To be clear — I am not arguing for deregulation. I am arguing for calibration. In my opinion, three specific changes would go a long way in achieving this[1]: 

- Scale compliance to operator size: A four-person team running a closed local system is not the same risk profile as a multinational cloud deployment. Article 62’s existing SME provisions are a start, but simplified forms do not address the fundamental cost asymmetry. Tiered conformity requirements - with substantively different obligations below defined revenue or headcount thresholds - would preserve protection while removing a structural barrier.

- Harmonise health AI certification: AI Act conformity assessment and medical device certification currently run as parallel, uncoordinated processes. A single integrated pathway would reduce the burden without reducing the standard. One notified body, one set of documentation.

- Fund the sandbox entry: Regulatory sandboxes should include publicly funded legal and technical support for operators who qualify by innovation profile but not by balance sheet. Otherwise, it is a door with a lock whose key requires you to already be inside. 

* * *

There is a patient in Croatia who could be receiving AI-assisted diagnostics from a local AI system - a system designed carefully, bounded appropriately, running on local infrastructure with no external connections. A system built by people who understand both the technology and the regulation. 

That system is not going to be built. Not because the AI was deemed dangerous, but because the compliance framework was insurmountable. 

That is a policy choice. It can be revised.

[1] A note on timing: when I began writing this article, the high-risk provisions were set to apply from August 2026. On May 7, the Council and Parliament reached a provisional agreement to postpone standalone Annex III high-risk obligations to December 2027. This is welcome breathing room - but a longer runway does not change the height of the bar. The structural issues described above remain identical under the new timeline.