Would You Market a Drug You Can’t Explain?
(I probably will)
TL;DR: AI drug discovery excels at prediction but stumbles at explanation- and in 2026, regulators are making that gap legally and ethically untenable. Developers now face a simple choice: explain your black boxes or don’t deploy them. Even if you are allowed to, should you?
Here’s the question keeping drug developers awake at night: Would you market a medicine that demonstrably helps patients—but whose mechanism of action you cannot fully explain?
That’s not hypothetical anymore. It’s 2026, and that question is no longer academic.
For the first time, pharmaceutical companies are running clinical trials on drugs that were designed by AI systems so complex, so layered with machine-learning inference, that even their inventors cannot trace the complete logic chain back to first principles. We can show you: This molecule works. Here’s the data. But we cannot always show you: Here’s exactly why.
This isn’t a failure of the technology. It’s a success that’s outpaced our ability to understand it. And that success is now colliding with something more fundamental: the ethics of deploying medicines we don’t fully comprehend.
Welcome to the Black Box Reckoning of 2026. There are three crises that matter right now, and they’re all variations on the same theme: AI’s predictive power has outrun human understanding—and the system is finally pushing back.
The “Phenotypic Trap”: When Cells Lie—And Patients Pay
Let’s start with the clearest cautionary tale: REC-994, Recursion’s superoxide scavenger for cerebral cavernous malformations (CCM), a rare disease affecting roughly 25,000 Americans. Most have no cure. Many face progressive neurological decline.
In February 2025, Recursion proudly announced that REC-994 had met its primary endpoint of safety and tolerability in a Phase 2 trial called SYCAMORE—and, more importantly, showed what the company called “promising trends” in lesion volume reduction. Fifty percent of patients on the 400 mg dose showed a reduction in total lesion volume compared to 28% on placebo after 12 months. (Recursion Pharmaceuticals, February 2025)
For patients with a devastating rare disease, this was hope. The AI-enabled phenotypic screening platform had done what it was built to do: scan millions of cellular images, find patterns that distinguish diseased from healthy states, and identify molecules that “rescued” those states. The AI worked. The science checked out. The patients might have a chance.
Except they didn’t.
By May 2025, when Recursion shared results from the long-term extension phase, the hope evaporated. Patients who had crossed over from placebo to the active drug showed no improvement. Worse, the patients who stayed on 400 mg the whole time didn’t sustain their initial benefit—their results became “indistinguishable from natural history,” in the clinical euphemism that means basically, the drug stopped working, and we’re back where we started. (GEN, May 2025)
This is the Phenotypic Trap in its most honest form: Recursion’s computer vision system optimized for a specific cellular readout—visual morphology that suggested disease reversal. The AI was right about what it was measuring. But the measurement itself didn’t predict what mattered: whether patients got better.
The epistemological failure here is subtle but critical. The AI learned to predict one thing (cellular appearance) while the researchers assumed it would predict another (human disease). No one set out to deceive. But the system was so sophisticated at pattern recognition that it found correlations that looked like causation—until humans tried the drug and reality intervened.
Recursion has since halted or deprioritized several pipeline programs. For the CCM patients in SYCAMORE, the news was simply: your trial drug doesn’t work. Time to wait for something else.
This is why the question matters: We trusted the AI because it made sense on a cellular level. But the cell isn’t the patient.
But Here’s the Other Side
It’s worth noting that REC-994’s failure doesn’t invalidate the phenotypic screening approach entirely. Recursion’s other AI-discovered program, REC-4881 (a MEK inhibitor for familial adenomatous polyposis), is showing what appear to be genuine clinical signals. December 2025 data showed a 43% median reduction in polyp burden after 12 weeks. (Recursion Pharmaceuticals, December 2025) The platform works sometimes—and when it does, it works well.
Additionally, traditional drug discovery has always involved failures in translation. Compounds that show promise in cell models routinely fail in humans; this is not unique to AI-driven approaches. The difference is partly one of scale: AI can generate and test more hypotheses faster, which paradoxically means both more failures and more successes, compressed into a shorter timeframe.
There’s also the question of mechanism itself. Many drugs used clinically for decades—aspirin, for example—had unknown or incompletely understood mechanisms of action when they were first approved. The assumption that we must understand why a drug works before deploying it is historically somewhat anachronistic. What matters most is that it works, and that it’s safe.
The FDA’s Credibility Demand: Protecting Patients by Demanding Answers
Here’s where the conversation gets uncomfortable for the tech side.
On January 6, 2025, the FDA released draft guidance titled “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products.” (FDA, January 2025) This was the first comprehensive regulatory framework addressing AI in drug development, and it said something that made the “move fast and break things” crowd grip their seats:
The agency is not accepting “the AI said so” as a valid reason to approve a medicine.
Instead, the FDA’s guidance outlined a “risk-based credibility framework” that requires sponsors to:
Define their AI model’s “context of use” precisely
Demonstrate that the model outputs actually predict clinical relevance
Provide explainability artifacts showing how the AI arrived at a specific prediction
Validate the model on independent data
That phrase—“Explainability Artifacts”—is doing a lot of work. (Federal Register, January 2025) It’s essentially asking: Can you show us your reasoning? Not in pseudocode. Not in a PowerPoint. In evidence that traces from the AI’s input to its clinical prediction.
Then, just last month (January 14, 2026), the FDA and EMA jointly published “Ten Guiding Principles of Good AI Practice in Drug Development.” (PharmaSource, January 2026) While these are currently non-prescriptive and voluntary, they’re the blueprint for future mandatory guidance. The principles emphasize “interpretability, explainability, and predictive performance” and demand “clear, essential information” about AI capabilities and limitations.
The message is unmistakable: Transparency is now table stakes. And it’s not a technicality—it’s a patient safety issue.
Deep learning models that even their creators can’t fully explain—like those used by Insilico Medicine or the newly merged Recursion-Exscientia entity—are now tagged as higher-risk by regulators. (Pinsent Masons, December 2025) That means slower approval timelines and more expensive validation requirements. But that slowdown is intentional. It’s there to protect patients from another REC-994 scenario: a drug that looks good in cells but fails in humans because we didn’t understand what we were actually treating.
The industry is already reacting. According to PharmaSource (December 2025), pharmaceutical companies are now “pulling back on fully autonomous AI design and re-inserting ‘Human-in-the-Loop’ steps just to ensure they have a human scientist who can sign off on liability.”
That sounds like bureaucracy. But it’s actually a return to something older: scientific accountability. Someone has to stake their reputation on the claim that they understand why this drug works. The human-in-the-loop isn’t a bottleneck. It’s a guardrail.
But Here’s the Other Side
Explainability requirements, while well-intentioned, face a fundamental challenge: some machine learning systems may be mathematically impossible to fully explain, even in principle. Deep learning models with billions of parameters operate in ways that resist human-comprehensible interpretation. Demanding explainability artifacts could create a false sense of certainty—a detailed explanation that sounds authoritative without actually being more predictive.
Additionally, pharmaceutical companies are already investing heavily in explainability research. Major biotech firms are hiring machine learning engineers specifically to develop interpretability tools (SHAP values, attention mechanisms, saliency maps). The burden isn’t unreasonable, and the FDA’s phased guidance approach (draft in January 2025, final guidance Q2 2026) gives sponsors time to adapt.
There’s also a practical precedent: regulatory agencies have long accepted empirical efficacy without complete mechanistic understanding. A drug that works is a drug that works, whether or not the mechanism is fully transparent. The risk here may be over-regulation—imposing standards of explainability that are scientifically unrealistic for complex AI systems, thereby slowing down the very innovation the agencies claim to support.
The EU AI Act “Liability Cliff”: August 2, 2026—Who Answers for an AI Mistake?
Now for the existential question: Who is responsible if an AI-designed drug harms patients in ways no one predicted?
On August 2, 2026—less than six months from today—the European Union’s AI Act enforcement provisions for high-risk AI systems become fully applicable. (EU AI Act Service Desk) This is not a guideline. This is law with teeth.
Here’s the problem: When a drug is designed by human chemists and a patient is harmed, the liability chain is clear. The pharmaceutical company knew what they were doing; they either understood the risk or failed to check. But when a drug is designed by an AI system that no one fully understands, the accountability becomes murky:
Is the pharmaceutical company liable for deploying a black box?
Is the software vendor liable for building an unexplainable system?
Is the CRO liable for providing biased training data?
Is the data broker liable for the “unbiased” patient cohort that wasn’t actually representative?
The EU AI Act answers simply: The pharmaceutical company is liable. Period. (Gardner Law)
Under the Act, high-risk AI systems (which include those supporting healthcare decisions) must be registered in an EU database, undergo conformity assessments, and be monitored for performance drift and bias. Companies must maintain detailed documentation showing how their AI system works and why it makes the decisions it makes.
If a company fails to meet these requirements by August 2, 2026, it faces fines up to €15 million or 3% of global turnover. If the violation involves a prohibited AI practice, it’s up to €35 million or 7%. (Software Improvement Group, January 2026)
But the real cost isn’t the fine. It’s the accountability itself.
Pharma companies are already shifting strategy. According to legal analysis from Pinsent Masons (December 2025), many firms are “de-risking their AI pipelines” by reintroducing governance steps that slow down discovery but create clear audit trails and human accountability. Human scientists are being asked to sign off on AI decisions—not because anyone thinks the human is smarter, but because someone needs to be responsible if things go wrong.
This creates what insiders call “governance debt”: the speed advantage that AI promised starts to vanish. But what’s lost in speed is gained in something older and more important: answerability to patients.
But Here’s the Other Side
The EU AI Act’s August 2, 2026 enforcement date, while tight, does provide a clear deadline that allows companies to plan and prepare. Some industry observers argue that regulatory clarity—even if strict—is preferable to ambiguity. Companies know what they need to do, and they have time to implement it.
Moreover, the Act’s framework allows for proportional risk assessment. Not all AI systems are classified as high-risk; systems used purely in research or early discovery phases are exempt. Companies can also reduce the AI risk classification of a drug by running it through traditional clinical validation pathways, which many are already doing. The impact may be less severe than initially feared.
Additionally, regulatory sandboxes—at least one per EU member state, operational by August 2, 2026—are designed specifically to help companies navigate novel AI-enabled drug development in a supervised environment. These sandboxes create space for innovation while maintaining oversight. And the Act does allow for graduated compliance: pharmaceutical companies deploying AI systems already in use before August 2026 have an extended transition period (until August 2, 2027) if they can demonstrate they’re taking steps toward compliance.
Finally, liability clarity may actually benefit the industry. Right now, responsibility for AI failures is murky; the Act clarifies that the pharmaceutical company is the responsible party. This removes ambiguity and could reduce expensive legal disputes downstream.
The Tension That Defines 2026
So here’s the Black Box dilemma, stripped bare:
LayerThe TensionThe 2026 RealityScientificCan we trust a drug if we don’t know its mechanism of action? Phase 3 results will answer this. REC-994 suggests: maybe not.Regulators want transparency; AI is inherently opaque.FDA and EMA are pushing toward “White Box” models (interpretable AI). Companies are hiring explainability engineers.LegalWho pays for an “AI mistake”?EU AI Act (Aug 2, 2026) forces companies to own the risk, even if they outsourced the AI.
The tension isn’t between Silicon Valley and Big Pharma anymore. It’s between predictive power and human understanding—and we’re in a standoff.
Recursion’s merger with Exscientia (closed November 2024) (Recursion Pharmaceuticals, December 2024) was supposed to create an unstoppable “full-stack” AI drug discovery powerhouse. And maybe it will. But REC-994’s stumble showed that stacking biology engines and chemistry engines doesn’t guarantee that the output will be biologically intelligible—or clinically relevant.
The old guard—the medicinal chemists who’ve been doing this for 30 years—are quietly vindicated. They never trusted black boxes. The young biotech founders are scrambling to make their black boxes look less black. And the regulators have drawn a line: Predict what you want, but explain what you find.
By August 2, 2026, that line becomes law in Europe.
The Epistemological Reckoning: Why Understanding Matters
But here’s the deeper concern, beneath all the regulatory and legal wrangling: What do we lose when drug developers deploy medicines they don’t understand?
Medicine isn’t just about outcomes. It’s about knowledge. When a physician prescribes a medicine, they carry a responsibility to understand—or at least to attempt understanding—why it works. That understanding is what allows them to:
Adjust dosing when a patient isn’t responding
Predict side effects based on mechanism
Design better combination therapies
Build on the scientific foundation for the next breakthrough
When the AI finds a drug that works but the mechanism is opaque, we’ve compressed the discovery timeline at the cost of scientific depth. We’ve optimized for outcomes at the expense of understanding.
REC-994 is the clearest example. The molecule did something in cells. But without knowing what or why, researchers couldn’t anticipate that it would fail in living humans. They were flying blind—not because they lacked data, but because the AI’s reasoning process was too complex to audit.
This is the risk of the black box: It can lead you to the right answer for the wrong reasons. And when it leads you to the wrong answer, you have no insight into why it failed.
The regulatory push toward explainability—the FDA’s guidance, the EMA’s principles, the EU AI Act’s accountability measures—isn’t just bureaucratic caution. It’s an attempt to preserve something essential: the connection between discovery and understanding.
A medicine that works is a victory. A medicine that works and that we understand is science. There’s a difference. And in 2026, that difference is becoming a dividing line.
What Comes Next
The Black Box Wars aren’t over. They’re escalating.
By August 2, 2026, pharmaceutical companies operating in Europe will face real legal consequences for deploying high-risk AI systems they can’t explain. The FDA’s final guidance on AI credibility is coming in Q2 2026—expected to be more prescriptive than the draft. And patients are becoming aware that the medicines they trust might have been designed by systems no one fully understands.
The industry will adapt. Some companies will invest heavily in explainability research, hiring teams to make their AI systems more interpretable. Others will pull back from fully autonomous discovery, reintroducing the human chemist as a check on the algorithm. A few will double down on black boxes and argue (perhaps successfully) that results matter more than reasoning.
But the core question remains, the one posed in the title: Would you market a drug you can’t explain?
For most of 2026, the answer from regulators, patients, and increasingly from developers themselves is: Only if you’re willing to stake your company’s reputation and liability on it. And only with complete documentation of your reasoning process.
That doesn’t mean the answer is always “no.” It means the answer is conditional—contingent on transparency, accountability, and proof that you’ve done your homework. The industry isn’t being asked to abandon AI drug discovery. It’s being asked to grow up about it. To acknowledge that speed and predictive power are valuable, but not if they come at the cost of scientific accountability.
That’s not a loss. That’s maturity.