Medicare's experimental AI system for approving medical treatments has exposed a fundamental conflict of interest: the companies operating the system earn more money when they deny more care. The Centers for Medicare and Medicaid Services (CMS) launched the WISeR (Wasteful and Inappropriate Service Reduction Model) pilot in six states, deploying artificial intelligence and machine learning tools to automate prior authorization decisions across original Medicare through the end of 2031. While the program pairs algorithmic determinations with human clinical review, critics argue that the human layer is undermined by volume, time pressure, and a structural incentive embedded in the vendor contract: participating vendors earn a share of expenditures deemed averted, meaning their revenue rises when treatments are denied.

Why Does a Vendor Payment Structure Matter in AI Healthcare Decisions?

The vendor compensation model represents a textbook conflict of interest that AI governance programs must screen for during procurement. When a company's profit depends on how many claims it denies, the incentive to approve necessary treatment diminishes. This is not a theoretical concern. A 2025 American Medical Association survey found that 61 percent of physicians believe AI will increase denials of medically necessary treatments, and early reporting cites documented instances of care delays and wrongful denials in the pilot's initial months.

The problem extends beyond Medicare. Enterprises in insurance, managed care, and benefits administration that use similarly structured vendor contracts face analogous regulatory exposure as state automated-decision-making laws such as the Colorado AI Act SB205 begin to mature. The WISeR pilot is now under federal scrutiny, and the design raises direct questions about whether the human review component meets any defensible standard of meaningful oversight.

What Does "Meaningful Human Review" Actually Mean When Reviewers Are Overwhelmed?

The pilot demonstrates that nominal human-in-the-loop design does not satisfy a meaningful oversight standard when reviewers operate under volume or time constraints that prevent genuine evaluation of AI outputs. This failure pattern has been documented in other high-stakes contexts, including the Meta lawsuit over AI-assisted layoff decisions, where 26 former employees filed suit alleging that Meta used internal AI tools, including a system called "Metamate," keystroke monitoring, and algorithmic performance ranking to select approximately 8,000 workers for layoffs in May 2026 without adequate human review.

Regulators are increasingly treating inadequate human oversight as a control deficiency rather than a design choice. The question is not whether humans are involved in the decision, but whether they have the time, information, and authority to genuinely evaluate what the AI recommends. In the Medicare pilot, the volume of prior authorization requests combined with time pressure creates a scenario where human reviewers may rubber-stamp algorithmic denials rather than conduct meaningful case-by-case evaluation.

How Can Organizations Build Trustworthy AI Decision Systems?

• Audit Vendor Contracts: Examine any AI-assisted claims, benefits, or prior authorization vendor contracts to identify compensation structures that tie vendor payment to denial rates or cost-avoidance metrics, and document findings in your conflict-of-interest register.
• Assess Human Review Standards: Evaluate whether human review layers in your high-stakes AI decision workflows meet a defensible meaningful review standard, including reviewer caseload, available information, time per review, and authority to override the model without escalation friction.
• Require Algorithmic Transparency: Demand that vendors operating AI in benefit determination or eligibility contexts provide decision-level audit logs with enough detail to reconstruct the basis for each automated outcome, and verify this capability before renewal.
• Conduct Fairness Reviews: Perform a bias and fairness review of any AI model used in denial or eligibility decisions, with attention to whether denial rates differ systematically by patient demographic or treatment category.
• Monitor Regulatory Developments: Engage legal and compliance teams to map the WISeR pilot's enforcement trajectory and any forthcoming CMS guidance on AI use in Medicare decisions, and update your regulatory monitoring calendar through December 2031.

The lack of algorithmic transparency in denial decisions creates direct auditability risk. Without explainable, logged rationales for each automated determination, organizations cannot demonstrate regulatory compliance, respond to appeals, or conduct post-incident reviews when denials are later found to be wrongful.

What Broader Lessons Does the WISeR Pilot Teach About AI Accountability?

The Medicare pilot is not an isolated incident. Deloitte Australia recently returned $290,000 in fees after an AI-generated consulting report produced using an Azure OpenAI agent contained non-existent court citations and fabricated quotes. The failure traced directly to absent two-person verification for legal references and no mandatory human review of numerical and citation claims in AI-assisted deliverables. This case has become a reference point for enterprise compliance teams building controls around AI-assisted professional deliverables.

Compliance teams should monitor CMS for formal guidance or rulemaking on AI standards in prior authorization decisions, particularly any requirements for explainability, appeal transparency, or vendor incentive restrictions that could reshape healthcare AI procurement. State legislatures are likely to respond to the WISeR pilot's documented denial patterns with automated-decision-making legislation modeled on Colorado Senate Bill 189 or California Senate Bill 420, creating a patchwork of obligations for health plans operating across jurisdictions.

Litigation stemming from wrongful denials attributable to the WISeR algorithm will be an important signal for how courts treat vendor liability and the adequacy of human review in AI-assisted benefit determinations through 2027 and beyond. The stakes are high not just for Medicare beneficiaries, but for every organization deploying AI in high-stakes decision-making contexts where financial incentives and human oversight intersect.

" }