New research from University of Pennsylvania and Northeastern University researchers directly contradicts Google CEO Sundar Pichai's public statements about the success of Google's AI search features. A randomized field experiment tracking over 1,100 real Google users found that Google's AI Mode reduces traffic to publisher websites, worsens user satisfaction, and even drives some users to switch to competing search engines like Bing and DuckDuckGo.

What Does the Research Actually Show About Google's AI Search?

The study, titled "AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence," enrolled 1,444 participants and tracked the search behavior of 1,100 users over seven days using a browser extension that recorded their clicks and interactions. The researchers tested three different search experiences: regular Google Search with AI Overviews hidden, Google's normal behavior with AI features shown when Google chose to display them, and forced AI Mode searches.

The findings paint a stark picture of how Google's AI features are affecting user behavior:

• Click-Through Rates Drop Significantly: Users assigned to AI Mode were substantially less likely to click through to external websites, while removing AI Overviews increased clicks to publisher sites without any decline in user satisfaction with Google Search itself.
• User Satisfaction Declines: AI Mode reduced trust in Google Search, decreased users' perception that search was useful or satisfying, and made results feel less personally relevant compared to regular search.
• Users Switch to Competitors: Assignment to AI Mode significantly increased the fraction of users searching on competitor engines and expressed intention to switch to Bing, suggesting the experience was poor enough to drive them away from Google entirely.
• Search Activity Decreases: People randomly assigned to use AI Mode conducted fewer search sessions overall, contradicting Google's expectation that AI features would increase engagement.

How Does This Contradict What Google's Leadership Has Said?

Google CEO Sundar Pichai and other company leaders, including Liz Reid, have consistently claimed that users are "delighted" with Google AI Mode, engage with it strongly, and prefer it over regular search. Pichai has also stated publicly that Google's AI Mode is sending more users to the open web than ever before. The university research directly contradicts both of these assertions, showing that AI Mode actually traps users within Google's search experience and results in fewer clicks to external websites.

The disconnect is particularly notable given Pichai's recent remarks at the AI Impact Summit in New Delhi, where he described AI as "the biggest technology shift of his lifetime" and stated that "no technology has me dreaming bigger than AI". While Pichai emphasized the potential for AI to solve major problems in healthcare, education, farming, and scientific research, he did not address the emerging evidence that Google's own AI search implementation may be harming the broader web ecosystem.

"No technology has me dreaming bigger than AI," Pichai said in his speech at the AI Impact Summit, describing AI as the biggest platform shift of our lifetimes.

Sundar Pichai, CEO at Google

What Are the Practical Implications for Publishers and Users?

The research has significant real-world consequences for the web ecosystem. Publishers of all sizes have reported that Google's AI search features are sending them less traffic, and this study provides rigorous experimental evidence supporting those complaints. When AI Overviews are removed, clicks to external websites increase without any detectable decline in how users perceive Google Search, suggesting that the AI features are purely extractive rather than additive to the user experience.

The study also found that the effects of AI Mode were broadly similar across most users, though heavy users of Google Search experienced a more pronounced reduction in search activity when using AI Mode. This suggests that the most engaged Google users may be the most negatively affected by the AI features.

Steps to Understanding the Study's Methodology and Reliability

• Randomized Field Experiment Design: The researchers used a randomized controlled trial with real Google users, not simulated or laboratory conditions, which makes the findings more reliable than observational studies or surveys.
• Browser Extension Tracking: Researchers deployed a browser extension that recorded actual search behavior, clicks, and whether AI Overviews appeared, capturing real user decisions rather than self-reported behavior.
• Large Sample Size: The study tracked 1,100 users over seven days, with 956 completing a post-experiment survey, providing a substantial dataset for drawing conclusions about user behavior patterns.
• Multiple Measurement Approaches: The researchers measured click-through rates, search session frequency, user satisfaction, trust in Google Search, perceived usefulness, and intention to switch to competitor search engines, providing a comprehensive view of AI Mode's impact.

The researchers did encounter some technical challenges during the experiment. Google rolled out changes to AI Overview HTML during the study period that broke the extension's ability to hide AI Overviews, meaning that only 51.1% of AI Overviews were successfully hidden in the "no AI" condition by the end of the study. Despite this limitation, the researchers were able to draw clear conclusions from the data they collected.

One notable finding was that the study did not detect any clear difference in user perceptions when comparing AI Mode with regular Google Search specifically for news-related searches, suggesting that the negative effects of AI Mode may vary depending on the type of search query.

What Does This Mean for Google's AI Strategy Going Forward?

The research raises fundamental questions about Google's approach to integrating AI into search. While Pichai has emphasized Google's commitment to responsible AI development and warned that "we cannot allow the digital divide to become an AI divide," the evidence suggests that Google's current AI search implementation may be creating a different kind of divide: one between users who benefit from direct answers within Google and publishers who lose traffic as a result.

Pichai has stated that Google has trained 100 million people in digital skills and is launching a Google AI Professional Certificate to help people use AI in their jobs, reflecting his broader vision for AI's positive impact. However, the university study suggests that the company's own AI search product may need significant refinement to align with its stated commitment to improving user experience while supporting the broader web ecosystem.

The findings add to a growing body of evidence from publishers and researchers questioning whether Google's public statements about AI search align with the actual user experience and impact on web traffic. As Google continues to expand AI features across its search product, this research suggests the company may need to reconsider its approach to balancing AI-generated answers with traffic to external websites.