Reference
AI crawler reference
23 crawlers from 9 operators, taken from what each operator publishes about its own. 5 of them say robots.txt may not apply to them. 5 publish nothing at all.
| Crawler | Operator | What it is for | Says about robots.txt | Can you check it? |
|---|---|---|---|---|
| Amzn-User | Amazon | Fetches when a user asks | Says robots.txt may not apply | IP list |
| ChatGPT-User | OpenAI | Fetches when a user asks | Says robots.txt may not apply | IP list |
| facebookexternalhit | Meta | Security and integrity checks | Says robots.txt may not apply | No published method |
| meta-externalfetcher | Meta | Fetches when a user asks | Says robots.txt may not apply | No published method |
| Perplexity-User | Perplexity | Fetches when a user asks | Says robots.txt may not apply | IP list |
| Bytespider | ByteDance | Trains a model | Operator publishes no documentation | No published method |
| Meta-ExternalAds | Meta | Checks pages submitted as ads | Operator publishes no documentation | No published method |
| meta-externalagent | Meta | Trains a model | Operator publishes no documentation | No published method |
| Meta-WebIndexer | Meta | Builds a search index | Operator publishes no documentation | No published method |
| OAI-AdsBot | OpenAI | Checks pages submitted as ads | Operator publishes no documentation | IP list |
| CCBot | Common Crawl | Builds an open archive | Treats robots.txt as the opt-out, without saying it respects it | Reverse DNS + IP list |
| GPTBot | OpenAI | Trains a model | Treats robots.txt as the opt-out, without saying it respects it | IP list |
| OAI-SearchBot | OpenAI | Builds a search index | Treats robots.txt as the opt-out, without saying it respects it | IP list |
| PerplexityBot | Perplexity | Builds a search index | Treats robots.txt as the opt-out, without saying it respects it | IP list |
| Amazonbot | Amazon | Trains a model · Builds a search index | Says it respects robots.txt | IP list |
| Amzn-SearchBot | Amazon | Builds a search index | Says it respects robots.txt | IP list |
| Applebot | Apple | Builds a search index · Trains a model | Says it respects robots.txt | Reverse DNS + IP list |
| Applebot-Extended | Apple | Not a crawler — a robots.txt token | Says it respects robots.txt | No published method |
| Claude-SearchBot | Anthropic | Builds a search index | Says it respects robots.txt | IP list |
| Claude-User | Anthropic | Fetches when a user asks | Says it respects robots.txt | IP list |
| ClaudeBot | Anthropic | Trains a model | Says it respects robots.txt | IP list |
| Google-CloudVertexBot | Trains a model | Says it respects robots.txt | No published method | |
| Google-Extended | Not a crawler — a robots.txt token | Says it respects robots.txt | No published method |
Entries that are not crawlers
Two entries never send a request. They are robots.txt tokens that set a preference about content already crawled, so they do not appear in a log file at all — an absence that is routinely read as the setting having failed.
Verification coverage
14 of 23 publish something you can check a visit against — a reverse-DNS hostname, a list of addresses, or both. For the rest no method is published, so a visit claiming to be one of them can be neither confirmed nor contradicted.
Each crawler has its own page with the operator’s wording, the full user agent string where one is published, what blocking it affects, and the source link. Where an operator’s documentation does not address a point, that absence is recorded with the date the document was checked.