WordPress plugin · Free
AumCrawl
v1.1.0AI crawler controls
AumCrawl is a free WordPress plugin for monitoring and controlling crawler access. It records visits from known crawlers, including AI crawlers, search engines, SEO tools and link-preview services, with the time of each visit and the pages requested. Site owners can block selected crawlers or add robots.txt rules for them.
Official guide
AumCrawl
Read the guide →
Features
- Logs visits from known crawlers: the crawler, the time of the visit and the pages requested.
- Groups AI crawlers into those associated with search and answer services that can refer visitors (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and those that do not refer visitors (GPTBot, ClaudeBot, CCBot, Bytespider and SEO backlink crawlers), with the access control shown next to each crawler’s visit history.
- Verifies crawler identity against the reverse DNS published by each operator, since a user agent such as GPTBot can be sent by any client. No IP addresses are stored.
- Appends its rules to robots.txt without replacing the file, and indicates when Yoast SEO, Rank Math or All in One SEO is generating it.
- Labels the strength of each control: a block at the server prevents the page from being served; a robots.txt rule is a request that compliant crawlers follow; the noai header is a weaker signal provided at no cost.
Interface
Intended use
Sites whose content is the primary asset, such as product catalogues, knowledge bases and long-running blogs, and site owners who need a record of which crawlers visit before setting access rules.
Requirements
- WordPress
- 5.5+
- PHP
- 7.4+
- Version
- 1.1.0
Common questions
- Does AumCrawl block Google?
- No. The plugin provides no block control for Googlebot, Bingbot, or the link-preview crawlers used by Facebook, X and LinkedIn, because blocking them affects search visibility and link previews. Their visits are still recorded, including the time of Google’s most recent visit.
- How effective is each blocking method?
- A server-side block applies to crawlers that identify themselves: the page is not served. A robots.txt rule is followed by the operators listed in the plugin, and is not enforced against crawlers that do not comply. The plugin labels each method accordingly.
- Is AumCrawl compatible with SEO plugins?
- Yes. AumCrawl appends to robots.txt rather than replacing it. When another plugin already writes a rule for a crawler, AumCrawl does not add a conflicting rule and shows a notice instead.
- Does AumCrawl generate llms.txt?
- No. llms.txt is provided by AumViso and by other plugins.
Custom development
These plugins cover general requirements. Requirements beyond their scope are handled as custom development.