A quirky test involving pelicans on bicycles has become a surprisingly useful way to evaluate AI models, but it's now exposing a fundamental problem with how we benchmark artificial intelligence. The test, which asks models to generate an SVG image of a pelican riding a bicycle, started as a joke about how difficult it is to compare AI systems fairly. Over time, it developed an unexpected correlation with actual model quality. Now, as newer and more powerful models emerge, that correlation is breaking down, revealing that simple creative tasks don't measure what developers actually need.

What Can a Pelican on a Bicycle Actually Tell Us About AI Models?

The pelican benchmark emerged as an informal way to test whether models could handle spatial reasoning, creative instruction-following, and valid code generation. When researcher Simon Willison ran the test on Kimi K3, a new 2.8 trillion parameter model from Chinese AI lab Moonshot AI, the results were illuminating. The model generated a detailed SVG image with proper geometry and spatial awareness, but the cost was surprisingly high: 25 cents for a single prompt.

This happened because Kimi K3 currently offers only one reasoning effort level, labeled "max," which consumed 13,241 reasoning tokens just to output 3,417 tokens of actual response. The model also appears to use an 85-token hidden system prompt, which inflates token counts even for simple requests like "hi." These details emerged only through hands-on testing, not from official documentation.

The pelican test revealed something else important: Kimi K3's vision capabilities work well. When the model was shown the rendered SVG image and asked to describe it, it generated a detailed and accurate alt text in just 0.6 cents worth of tokens. This kind of multimodal capability is increasingly important for real-world development work, but it's invisible in traditional benchmarks.

Why Traditional Benchmarks Miss What Matters Most?

The biggest limitation of the pelican benchmark, and most other creative task evaluations, is that they don't test the capability that matters most for modern AI development: agentic tool calling and the ability to operate tools reliably as conversations grow longer. A model might generate a perfect SVG image on the first try, but that doesn't tell you whether it can maintain context across dozens of API calls, handle errors gracefully, or chain complex tasks together without losing track of its goals.

This gap explains why the pelican test's correlation with actual model quality has weakened. GPT-5.6 and Claude Fable 5 now produce pelicans that are outclassed by GLM-5.2, yet most observers wouldn't rank GLM-5.2 as a Fable-class model. The test has become decoupled from what developers actually care about: building reliable AI agents that can work autonomously on real projects.

Willison noted that the pelican test was never intended to be a rigorous benchmark. It started as a joke about the absurdity of comparing models fairly, and it served its purpose for about a year. Now, as models have become more sophisticated and specialized, the test's limitations are more apparent than ever.

How to Use Simple Tests to Evaluate AI Models Effectively

• Forcing Function for Real Testing: Running a simple prompt through a model forces you to actually try it rather than relying on marketing claims or benchmark tables. If you can generate a pelican, you've confirmed the model works and can follow creative instructions.
• Token Efficiency Measurement: Simple prompts reveal how a model uses tokens, including hidden system prompts and reasoning overhead. Comparing token counts across models shows which ones are more cost-effective for similar tasks.
• Multimodal Capability Assessment: Testing vision features on a rendered output reveals whether a model can process images and generate accurate descriptions, a capability increasingly important for code review and documentation tasks.
• Comparative Analysis Within Model Families: Running the same test across different versions of a model family shows how improvements accumulate. Kimi K3's pelican was notably better than Kimi K2.5, providing concrete evidence of progress.
• Reasoning Effort Comparison: When models offer multiple reasoning levels, running the same prompt at different effort settings reveals the cost-benefit tradeoff. GPT-5.6 shows significant variation in token usage and output quality across effort levels.

Willison continues to find value in the pelican test despite its limitations. The test serves as a "hello world" exercise for prompting a model, provides rough cost and reasoning estimates for a simple task, and confirms that a model can output valid SVG with basic geometric understanding. For smaller models that run on personal machines, this last point is particularly valuable.

What's Next for AI Model Evaluation?

The real challenge ahead is developing benchmarks that actually measure agentic capabilities: tool calling reliability, context maintenance over long conversations, error handling, and task chaining. These are the skills that determine whether an AI system can be trusted to work autonomously on real projects. A model that generates a perfect pelican but fails to maintain state across multiple function calls is not actually useful for modern development workflows.

The pelican benchmark's decline as a predictive tool doesn't mean simple tests are worthless. Rather, it highlights the need for more targeted evaluation methods. Developers should combine multiple approaches: simple creative tasks to verify basic capabilities, specialized benchmarks for tool use and reasoning, and real-world testing on actual development tasks. No single test can capture the full picture of what makes an AI model useful in practice.