The FDA's proposed pilot program to use artificial intelligence in early-phase clinical trials is facing pushback from industry and academic stakeholders who say the agency hasn't provided enough detail about how it will actually work, who will oversee it, or whether the FDA itself will deploy untested AI systems to make regulatory decisions. Nearly 240 organizations have submitted feedback on the program since the FDA issued a request for information in April, and their message is clear: the agency needs to be far more transparent before moving forward.

What Is the FDA's AI Clinical Trial Pilot Trying to Accomplish?

The FDA wants to explore how AI can speed up early-phase drug development, which the agency describes as "a critical bottleneck in drug development, often characterized by high uncertainty, limited patient populations, and inefficient decision-making processes." The pilot would test whether AI can improve trial efficiency, enhance safety monitoring, help with dose selection, and support go/no-go decisions about whether a Phase 1 study can proceed.

The agency says the pilot will be guided by principles aligned with the National Institute of Standards and Technology (NIST) AI Risk Management Framework, a set of guidelines designed to help organizations manage AI-related risks responsibly. But that's where the clarity ends, according to stakeholders who submitted feedback.

Why Are Industry Groups Raising Red Flags?

The Pharmaceutical Research and Manufacturers of America (PhRMA), which represents major drug companies, and the Biotechnology Innovation Organization (BIO) both flagged critical gaps in the FDA's proposal. Their concerns center on three main issues: governance, transparency, and whether the FDA will hold itself to the same standards it expects from industry.

PhRMA noted that the FDA has made public statements suggesting the agency itself may use AI as part of the pilot program. If that's the case, the group argued, the FDA needs to be explicit about when and how it will deploy AI, and crucially, the agency should apply the same credibility assessments and human review requirements to its own AI tools that it demands from pharmaceutical companies. "FDA AI tools as part of the pilot should themselves be under documented credibility assessment and change control, as appropriate, and their output should be subject to human review before influencing a regulatory decision," PhRMA stated.

BIO raised a different concern: it's unclear whether the FDA's primary goal is to advance real-time clinical trial capabilities or more broadly to promote AI use in drug development. The group asked the FDA to clarify its core objectives and engage more deeply with stakeholders before launching the pilot.

What Specific Governance Issues Are Stakeholders Demanding?

Both industry groups identified several governance and operational gaps that need to be addressed before the pilot can credibly move forward:

• Algorithmic Opacity: BIO noted that there is "limited clarity regarding the underlying models, intended use cases, and decision-making frameworks that FDA expects to evaluate," and that sponsor concerns about algorithmic opacity and insufficiently defined performance metrics may limit confidence in the pilot and impede meaningful participation.
• Documentation and Disclosure Requirements: Stakeholders asked for clarity on how sponsors should document AI use in regulatory submissions, whether and when AI use should be disclosed in study protocols and informed consent forms, and what level of documentation is needed to demonstrate alignment with the NIST AI Risk Management Framework.
• Confidentiality Protections: PhRMA emphasized that the FDA must establish a solid governance framework and be clear about how it will protect confidential commercial and trade secret information to ensure participant confidence and willingness to engage with the pilot.
• Distinction Between Decision Support and Autonomous AI: BIO asked the FDA to clarify how it will distinguish between AI used as human-in-the-loop decision support and autonomous decision-making systems, and how reproducibility and auditability will be ensured.

"To maximize learnings from the pilot, we encourage FDA to apply the pilot trials across different therapeutic areas, with priority on broadly applicable examples that can generate near-term learnings," PhRMA stated.

PhRMA, Pharmaceutical Research and Manufacturers of America

How Do These Concerns Fit Into the Broader AI Governance Challenge?

The FDA pilot controversy reflects a larger global struggle to regulate AI faster than the technology itself is advancing. A preliminary report from the UN Independent International Scientific Panel on Artificial Intelligence, released on July 1, 2026, warns that "the window to establish effective global governance remains open but may not stay that way for long".

The UN panel found that while more than 40 AI governance frameworks and ethical guidelines already exist in different parts of the world, they remain fragmented, inconsistent, and are rarely tested to see whether they actually work. Many safety assessments are also conducted by the companies developing the technology themselves, creating potential conflicts of interest.

The FDA pilot is a microcosm of this larger problem. The agency is trying to establish best practices for using AI in a high-stakes regulatory context, but it's doing so without clear rules, transparent governance, or even agreement on what success looks like. If the FDA can't get this right in a controlled pilot program, the implications for AI governance more broadly are troubling.

What Are the Stakes of Getting This Wrong?

AI has already demonstrated remarkable potential in drug development. The technology has predicted the structures of more than 200 million proteins, accelerated drug discovery and vaccine development, and helped researchers investigate antibiotic resistance. Speeding up early-phase clinical trials could unlock even more benefits, potentially bringing life-saving treatments to patients faster.

But deploying AI in regulatory decision-making without robust governance could also create new risks. If an AI system makes a flawed recommendation about whether a drug trial should proceed, and that recommendation is not properly reviewed or understood by human regulators, the consequences could be serious. Patients could be exposed to unsafe treatments, or promising therapies could be delayed or rejected based on algorithmic errors that no one can fully explain.

That's why stakeholders are demanding transparency now, before the pilot launches. PhRMA and BIO are essentially saying: we want to participate in this pilot, but only if the FDA is willing to be clear about how it works, who oversees it, and whether the agency itself is subject to the same standards it imposes on industry.

How to Ensure AI Governance Keeps Pace With Innovation

Experts and industry groups have outlined several steps that regulators and companies should take to build trustworthy AI governance in clinical development and beyond:

• Establish Clear Metrics and Goals: Define specific, measurable objectives for AI pilots before they launch, including success criteria, performance benchmarks, and decision-making frameworks that all participants understand and can evaluate.
• Require Independent Evaluation: Move beyond self-assessment by companies developing AI systems; establish independent third-party evaluation and auditing mechanisms to verify that AI systems are safe, transparent, and aligned with regulatory standards.
• Harmonize Global Standards: Work toward international alignment on AI governance frameworks so that companies operating across multiple jurisdictions face consistent expectations, and so that developing countries are not left behind in the AI revolution.
• Invest in Digital Infrastructure and Expertise: Countries need investment in computing infrastructure, technical education, and institutional capacity so they can govern and deploy AI on their own terms rather than depending entirely on technologies they cannot inspect, audit, or adapt.

The FDA's clinical trial pilot is scheduled to move forward, but it's clear that the agency will need to address these governance concerns before it can claim credibility with industry partners and the public. The stakes are high: if regulators can't demonstrate that they can govern AI responsibly in a controlled setting like a clinical trial pilot, confidence in AI governance more broadly will erode just as the technology becomes more powerful and consequential.