Perplexity's crawler is one of at least 62 AI bots actively fetching web pages right now, yet it doesn't appear in the major directories that most site owners rely on to control crawl access. A comprehensive audit published on August 4, 2026, reconciles data from Cloudflare's verified bot registry, actual traffic patterns, operator documentation, and robots.txt telemetry to reveal the full landscape of AI crawlers. The findings expose a critical gap: the directories everyone trusts are incomplete, and bots are crawling your site whether or not you can name them in your robots.txt file.

What's Actually Crawling Your Website Right Now?

The problem starts with trust. Cloudflare Radar maintains what appears to be the authoritative directory of AI bots, tracking 87 entries across three categories: bulk crawlers for training, assistants for answering questions, and search bots for retrieval. But this registry is a verified-bot list, not a census of what's actually on the web. Perplexity's bot, along with CCBot, Bytespider, and several others, don't appear in Cloudflare's directory at all. Yet Cloudflare's own traffic data tells a different story. Over the 28 days ending August 3, 2026, Bytespider alone accounted for 4.73% of all AI-bot requests measured across the internet.

This disconnect matters because most site owners rely on a single source to understand which bots to allow or block. If a bot isn't listed, it's invisible to the robots.txt rules that are the primary crawl-control mechanism for millions of websites. The audit was built by reconciling four independent sources instead of trusting one: Cloudflare's verified registry, Cloudflare's actual traffic measurements, each operator's own documentation, and real-world robots.txt telemetry showing what site owners are actually writing rules about.

Which AI Crawlers Dominate Web Traffic?

Traffic concentration at the top is steep. Googlebot accounts for 24.66% of all AI-bot requests over the same 28-day period, with ClaudeBot second at 15.59%. These two alone represent nearly 40% of AI crawling activity. The remaining traffic is distributed across dozens of smaller bots, each with different purposes and different levels of transparency about what they're collecting.

The three categories of AI crawlers serve fundamentally different functions, and conflating them is where most confusion starts. Bulk crawlers collect content in large volumes to train or improve AI models. Search crawlers index pages so an AI assistant can retrieve and cite them in answers. User-triggered fetchers grab individual pages because a person just asked a specific question. Same HTTP request protocol, three completely different consequences for your site's bandwidth and data exposure.

How to Manage AI Crawler Access to Your Website

• Match user-agent strings as substrings, not exact matches: Most AI operators append version numbers that change without notice, so a robots.txt rule keyed to "GPTBot/1.4" will break the day OpenAI ships version 1.5. Use substring matching to future-proof your rules.
• Identify bots by their published user-agent header: Of the 87 AI bots tracked, only 62 send an identifiable user-agent string. The remaining 25 are identified by IP range and signed metadata, making them impossible to block with robots.txt alone. Knowing which bots you can actually name is the first step to controlling access.
• Pair the user-agent directive with explicit allow or disallow rules: The user-agent line alone does nothing. Pair it with "Disallow: /" to block a bot entirely or "Allow: /" to permit it. Without the second directive, the rule has no effect.
• Monitor which crawlers are actually requesting your site: Check your server logs against the complete list of 62 named bots. You may find crawlers you didn't know existed, including Perplexity's bot and others absent from popular directories.

Why the Major Directories Are Incomplete

The audit reveals a structural problem: no single source lists AI crawlers correctly because the sources disagree with each other. Cloudflare's directory is a verified-bot registry, which means it only includes bots that have been formally registered and documented. But registration is voluntary, and many operators don't bother. Perplexity, for example, operates an active crawler that's measurable in real-world traffic but absent from the registry.

The content these bots want is heavily skewed toward text. Across all AI-bot requests measured, HTML comprises 72.29% of what they fetch, compared to 6.90% JSON and 5.21% images. These bots are after your prose, not your assets. This distinction matters for site owners deciding what to block. If you're concerned about training data extraction, you're primarily protecting text content. If you're worried about bandwidth, you can relax slightly knowing that images and structured data are lower priorities for most AI crawlers.

The audit was conducted by someone with direct experience on both sides of the crawling equation. The author spent five years on Google's Search team working on large-scale crawling and indexing, and now runs detection systems that scan more than 50 million domains per month. That perspective reveals why lists assembled from third-party directories go stale within a quarter: a crawler's user-agent header is trivial to change, and operators update them constantly without public announcement.

What This Means for Perplexity and Other Answer Engines

Perplexity's absence from major registries is not unique, but it is significant. As an AI answer engine, Perplexity operates a search crawler that indexes pages so the system can retrieve and cite them when answering user questions. This is different from a bulk training crawler, which collects content to improve a model. The distinction matters because search crawlers typically respect robots.txt rules more consistently, since they're designed to work within the existing web ecosystem. Training crawlers, by contrast, sometimes operate more aggressively because their goal is to build a comprehensive dataset.

The broader implication is that site owners cannot rely on a single directory to understand their crawl landscape. WordPress powers 64.06% of all websites with a detectable CMS as of July 26, 2026, and for most of those sites, a robots.txt file is the only crawl control mechanism available. If the bots you're trying to block don't appear in the directories you're consulting, your rules won't work.

The audit provides all 62 identifiable AI bots, their user-agent strings, their operators, what each one collects, and the robots.txt directive that controls it. It also documents the bots and control tokens that major directories leave out entirely. For site owners, the takeaway is clear: verify your crawl rules against actual traffic, not against a single registry. For researchers and technologists, the finding underscores a larger truth about AI infrastructure: the systems that power modern AI are often invisible until you look for them directly.