The AI Crawler Access Checklist

If your brand is missing from AI answers, the first thing to check is not your content. It is whether the software that fetches pages for those answers is allowed to read your site at all. This is free to check, takes about ten minutes, and is the only item in this field where the mechanism is documented by the vendors themselves rather than inferred from a study.

It is also the item people get wrong in a specific and expensive way. A blanket rule intended to keep a site out of model training usually also removes it from the live retrieval that decides what an engine says about you today.

The three jobs, and why they are separate

A single company can point several different agents at your site, and they are not doing the same thing. Sorting them by job rather than by brand is what makes the decision tractable.

Bulk collection for training. A crawler walks the web and stores pages to be used in building a model. This is the one most publishers are thinking about when they say “block the AI bots”. Whatever it takes is baked into a model months later; nothing about it affects an answer given this afternoon.

Indexing for a search product. A crawler builds an index that the assistant queries when a user asks something. This is much closer to how a classic search crawler behaves, and being absent from it means being absent from the candidate pool an answer gets assembled from.

A live fetch triggered by one user. Someone pastes your URL, or asks a question that causes the assistant to go and read a specific page right now. This is not a crawl in any meaningful sense — it is one request, on behalf of one person who is already interested in you.

The three sit at completely different points on the trade between protecting your content and being found. A publisher fighting over training data and a B2B vendor trying to get shortlisted have almost opposite interests, and they are being asked to express those interests in the same file.

What each vendor documents

Every one of these lists is published by the vendor, and every link below is to their own page rather than to a third party summarising it. We checked all of them on 13 September 2026. Read the source rather than a listicle: names change, listicles do not get updated, and a stale user-agent string in your robots.txt is a rule that silently applies to nothing.

Vendor	Agents it documents	Where
OpenAI	GPTBot, OAI-SearchBot, ChatGPT-User	OpenAI bots documentation
Anthropic	ClaudeBot, Claude-SearchBot, Claude-User	Anthropic crawler support article
Perplexity	PerplexityBot, Perplexity-User	Perplexity bots guide
Google	Googlebot, Google-Extended, GoogleOther, Google-CloudVertexBot	Google crawler list
Apple	Applebot, Applebot-Extended	Apple crawler documentation
Common Crawl	CCBot	Common Crawl CCBot page

Two things in that table catch people out.

The first is Common Crawl. CCBot is not an AI company’s crawler at all — it belongs to a non-profit that publishes an open web archive, and a great many models have been trained on that archive. Blocking it does not remove you from any assistant’s live answers, because no assistant queries it at runtime. If your objection is to training, it is one of the most consequential lines in the file. If your objection is to being invisible, it is irrelevant.

The second is Google. Google’s own position, stated in its guidance on AI features, is that its generative features are rooted in its core Search systems, so optimising for them “is optimizing for the search experience, and thus still SEO”. The practical consequence is that your visibility in those features is governed by the same Googlebot access that governs your ordinary rankings, and the separate Google-Extended control is about a different question. Treating Google’s AI surfaces as a separate crawler problem is a category error that costs people a lot of unnecessary configuration.

Check yours in ten minutes

No tooling required. Work through it in this order, because the failures get rarer and harder to fix as you go down.

1. Open https://yourdomain.com/robots.txt in a browser. Read what is actually served, not what is in your repository. A CDN, a security product or a hosting platform can serve a different file from the one you committed, and this is a genuinely common cause of a policy nobody remembers choosing.
2. Find every Disallow under a wildcard agent. A User-agent: * block with a broad Disallow applies to every agent that does not have its own more specific block. Most accidental AI blocks are inherited from this rather than aimed at anybody.
3. Look for a blanket AI rule. Several security vendors and hosting platforms ship a one-click “block AI crawlers” toggle. If someone enabled it, the evidence is usually a long list of user-agent names you did not write.
4. Check the three jobs separately. For each vendor above, decide training, indexing and user-triggered fetch independently. The common answer for a business that wants to be found is: allow the indexing and the user fetch, and make your own call about training.
5. Check for a stale name. An agent that no longer exists is a rule that does nothing. This site’s own file still names two retired Anthropic strings alongside the current ones, which is harmless here only because the underlying policy is permissive.
6. Check your firewall, not just the file. robots.txt is a request, and a WAF rule or aggressive bot-management setting will refuse the connection before anything reads it. If your file says allow and the crawler still never appears in your logs, this is where it is being stopped.
7. Confirm from server logs. The only proof is a fetch. Look for the agent strings above in your access logs over the last thirty days. If a name never appears, the rule you wrote is not the rule you are enforcing.

If you want the file shape rather than the reasoning, this is the whole of it for a site that wants to be read:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Sitemap: https://example.com/sitemap-index.xml

What robots.txt cannot do for you

It cannot make an engine cite you. Access is necessary and nowhere near sufficient: it puts your page in the candidate pool, and everything after that is decided by whether your page is the best thing available on the question. We grade that whole chain on our evidence register, including the parts that cut against selling the work.

It cannot be enforced. robots.txt is a convention that well-behaved agents follow voluntarily. If your requirement is legal or contractual rather than advisory, this file is the wrong instrument.

And it cannot substitute for being findable in the first place. There is no special AI file that fixes visibility — Google states explicitly that it does not use llms.txt and that creating one will neither help nor harm you in Search. We ship one anyway and claim nothing for it, which is a different position from the one most of this industry takes.

Where this stops mattering

Crawler access is a threshold, not a lever. Once the agents can read you, opening them further does nothing, and the next constraint is whichever third-party pages the engines are actually reading in your category. That is usually a small set — a review site, a couple of forum threads, one or two listicles — and it is where the argument about your product is genuinely being had.

Finding that set is a measurement problem rather than a configuration one. The method is written up in full in how to track your brand’s visibility in ChatGPT, it costs nothing, and it takes an afternoon. If your access check came back clean and you are still absent, that is the next thing to do.

Questions we get asked

Does blocking training crawlers hurt my rankings? Not in Google Search: Google’s documentation separates the control for its generative training and grounding from the Googlebot access that governs Search. It does affect whether your content ends up in future models, which is a different decision with a different time horizon.

Should I allow the user-triggered agents? Almost always yes, for a commercial site. That request happens because a person asked about you specifically. Blocking it means the one moment someone is definitely interested is the moment your page will not load.

Is there a downside to allowing everything? Bandwidth, mostly, and it can be real at scale. Crawl-rate directives and caching are the proportionate response. Refusing access outright to solve a bandwidth problem is using the wrong tool.

Can I check a competitor’s setup? Yes — their robots.txt is public, and comparing yours to theirs takes a minute. It is one of the few genuinely useful things you can learn about a rival for free.

The bottom line

Sort the agents by job, not by company. Allow indexing and user-triggered fetches if you want to be found, decide about training separately and deliberately, read the vendor’s own list rather than a blog post, and then verify against your server logs rather than your intentions. It is the cheapest work in this entire field and the only part of it where the mechanism is documented rather than inferred.

If you would rather have someone else check it against the rest of the picture, the Demand Diagnostic includes the access test alongside the sampling, and our methodology explains how the two fit together.