How to see which AI crawlers read your WordPress site, and block the ones you choose

Look before you block

Most “block AI” advice blocks everything, and that is the part worth slowing down on. AI crawlers do not behave the same way, and two of them that look alike in a log have opposite consequences when you turn them off.

• Some send people back. OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch a page because someone asked a question, and the answer can carry a link to you. Blocking them costs visits.
• Some take and return nothing. GPTBot, ClaudeBot, CCBot and Bytespider pull content into a training corpus or a commercial dataset. No link, no citation, no visitor.
• Some are not crawlers at all. Google-Extended and Applebot-Extended never send a request — they are robots.txt tokens that set a preference about content already crawled. You will never find them in a log, which is exactly why people conclude their setting did not work.

Which group a given crawler is in, in its operator’s own words, is on the AI crawler reference — one page per crawler, with the operator’s wording, what blocking it affects, and how to verify a visit.

Record who actually comes

Install AumCrawl from the Plugins screen and open Settings → AI Crawlers. It records every known crawler that reaches the site — AI, search engine, SEO tool, link preview — with who came, when, and which pages they read.

1. Install and activate the plugin, then leave it alone for a few days. A decision made on an empty log is a guess.
2. Open Settings → AI Crawlers and read the activity list: who came, how often, and which pages they read.
3. Check the most-crawled pages. A crawler that only ever reads one section is a different proposition from one that reads everything.
4. Only then decide which crawlers you are willing to lose.

What each control actually does

The settings screen separates three controls, and the difference between them is the whole subject:

• Rules in robots.txt — honoured by the crawlers that say they honour it, and ignored by anything that chooses to. It is a request, not a fence.
• Turning blocked crawlers away at the door — the strongest option: the page is never served. It only works against crawlers that say who they are.
• A “do not train on this” note — the X-Robots-Tag: noai convention. Weak: some tools read it, many do not. It costs nothing, which is the only reason it is there.

The screen’s What is actually in effect section does not assume a rule worked. It fetches the live robots.txt and looks. If another plugin is replacing the file, or a robots.txt exists on disk, it says so, names the plugin responsible, and gives the exact lines to paste instead.

Check that a visit was real

Anyone can send User-Agent: GPTBot. The Identity checks section verifies the claim against the hostname each operator publishes, using forward-confirmed reverse DNS, and shows which visits could not be verified.

Found: GPTBot        → verified   (resolves to an OpenAI host, and back again)
Found: GPTBot        → unverified (no published method, or the check failed)

Not every operator publishes a way to check. Where none exists, a visit claiming to be that crawler can be neither confirmed nor contradicted — which is a different statement from “this was forged”, and the reference page for each crawler says which of the two applies.

No IP addresses are stored. The address is used inside that one request and then discarded. When a check fails, only a shortened network range is kept — the last part of the address removed first — which is enough to see that a range is impersonating a crawler without recording who connected.

What it will not let you block

There is deliberately no switch for Googlebot, Bingbot, or the crawlers that build link previews on Facebook, X and LinkedIn. Turning one of those off does real damage, and that belongs in your own robots.txt rather than behind a button that can be hit by accident. They are still recorded, so you can see when Google last came by.