Free on WordPress.org
AumCrawl
AumCrawl is a free WordPress plugin that answers a question most site owners cannot answer: is ChatGPT actually reading my site? It records every known crawler that arrives — AI, search engine, SEO tool, link preview — with what it read and when, and lets you turn away the ones that take content and send nothing back.
What it does
- Logs every known crawler that reaches the site: who came, when, and which pages they read.
- Splits AI crawlers into the ones that send visitors back (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Google-Extended) and the ones that do not (GPTBot, ClaudeBot, CCBot, Bytespider, and the SEO backlink crawlers), so the switch sits next to the evidence.
- Verifies that a crawler is who it claims to be, using the reverse DNS each operator publishes — anyone can send a GPTBot user agent. No IP addresses are stored.
- Appends its rules to robots.txt instead of taking the file over, and tells you when Yoast, Rank Math or All in One SEO is the one actually writing it.
- Says plainly how strong each control is: turning a crawler away at the door works, a robots.txt rule is a request, and the noai header is weak but free.
Who it is for
Any site whose content is the product — a catalogue, a knowledge base, a blog someone spent years on. It is most useful before you decide anything: most owners are arguing about blocking AI without knowing which bots have been there at all.
Requirements
- WordPress
- 5.5+
- PHP
- 7.4+
- Version
- 1.0.1
Common questions
- Will it block Google?
- It deliberately gives you no switch for Googlebot, Bingbot, or the crawlers that build link previews on Facebook, X and LinkedIn. Turning one of those off does real damage, and it should not sit behind a button you can hit by accident. They are still recorded, so you can see when Google last came by.
- Does blocking actually work?
- Against a crawler that identifies itself, yes — the page is never served. A robots.txt rule is honoured by the operators listed in the plugin and ignored by anything that chooses to. The plugin labels which is which rather than implying they are the same.
- Does it conflict with my SEO plugin?
- No. robots.txt is crowded, so it appends rather than replaces. If another plugin already writes a rule for a crawler, it stands aside and tells you instead of writing a second, conflicting line.
- Does it generate llms.txt?
- No, that is a different job. AumViso does it, and so do other plugins.
Need the part no plugin does?
These are the pieces that were general enough to finish once and hand over. When the requirement stops being a plugin, that is the work we do to order.