The best way to get results from AI search engines and chatbots isn't to accept their answers at face value, but to challenge them. A recent experiment conducted by Wall Street Journal tech columnist Christopher Mims found that teams treating AI as a sparring partner, rather than an answer machine, achieved significantly better outcomes than any other group, including AI working alone.

Why Do Most Teams Fail With AI Search Engines?

Marketers and knowledge workers often approach AI like a vending machine: enter a prompt, get a finished answer. But large language models (LLMs), which power AI search engines and chatbots, are fundamentally limited in ways that make this approach problematic. These systems generate the next likely word based on patterns in training data, making them useful for speed and structure but weak at judgment, originality, and strategic thinking.

Mims ran a test with three groups: humans working alone, AI models like ChatGPT and Gemini working alone, and hybrid teams of humans and AI working together. All groups had one hour to make predictions about real-world events using scenarios from the prediction market platform Polymarket. The results revealed a troubling pattern: many hybrid teams simply accepted the AI's answers without questioning them, while others used AI to validate their own assumptions. This confirmation bias actually made some teams perform worse than AI working alone.

What Changed When Teams Started Arguing With AI?

However, something remarkable happened in roughly 5 to 10 percent of the hybrid teams. Instead of passively accepting AI outputs, these teams transformed their approach. The AI became a true sparring partner. Teams pushed back, demanding evidence and interrogating assumptions. When the AI expressed high confidence, the humans questioned it. When the humans felt strongly about an intuition, they asked the AI to come up with a counterargument.

Those teams did significantly better than any other group, even beating the Polymarket results in some cases. The implication is clear: the problem wasn't AI itself. The problem was how people interacted with it.

How to Get Better Results From AI Search and Answer Engines

• Critique the Output: Don't accept the first answer. Ask follow-up questions, request evidence, and push back on claims that feel uncertain or generic.
• Test Confidence Levels: When AI expresses high confidence, treat it as a hypothesis to test, not a conclusion. Ask it to explain its reasoning and identify potential weaknesses in its logic.
• Request Counterarguments: If you have a strong intuition or belief, ask the AI to argue against your position. This forces the model to explore alternative perspectives and can reveal blind spots in your thinking.
• Demand Specificity: Generic, flat outputs are a sign you're not engaging deeply enough. Ask for specific examples, data points, and reasoning that connects to your actual situation.
• Iterate and Refine: Treat each interaction as a conversation, not a transaction. Use the AI's responses to sharpen your own thinking, then feed that back into the next prompt.

This shift in approach has real implications for how organizations are adopting AI tools. Marketing teams, in particular, are beginning to recognize that AI works best when treated as a collaborative partner rather than a replacement for human judgment. The experiment suggests that the teams achieving the best results are those willing to invest time in genuine dialogue with AI systems.

The broader lesson extends beyond prediction markets. Whether you're using AI search engines to research a topic, asking chatbots to help draft marketing copy, or relying on AI to analyze data, the quality of your output depends directly on the quality of your engagement. The 5 to 10 percent of teams that cracked this code weren't smarter or better informed; they simply understood that AI is a tool for thinking, not a replacement for it.