AumCrawl

Decide which AI crawlers may read your site, and see which ones actually do.

This guide is for people who already have AumCrawl installed. Everything below is in the WordPress admin under Settings → AI Crawlers — it sits with the other settings screens rather than in a top-level menu of its own.

The AumCrawl admin is in English on every site. No translation has been published for it yet, so the labels quoted here are what you will see whatever language your WordPress is set to.

1. What this plugin is for

Dozens of crawlers now read the web on behalf of AI products. Some send you readers. Some take your text to train a model and send you nothing. They all arrive the same way, and by default your site answers all of them.

AumCrawl does two things: it writes the rules that tell those crawlers what they may read, in the file they look for, and it records who actually came, so your decision rests on your own traffic rather than on a list someone published.

It does not block anything by force. Crawler rules are a request. A crawler that ignores robots.txt is not stopped by any plugin — that is a job for your host or a firewall. What AumCrawl governs is the behaviour of the ones that do read the rules, which is most of the ones that matter.

2. The three groups

Open Settings → AI Crawlers. The crawlers are sorted into three groups, and the sorting is the whole argument.

2.1 These send traffic back

They read a page because a person asked a question, and the answer credits you. Blocking them costs visits.

Seven crawlers. Each one is attached to a product that shows sources. If someone asks an assistant a question your page answers, this is the group that can put your link in the answer.

2.2 These take and give nothing back

Training corpora and commercial datasets. Your content goes in; no link, no citation and no visitor comes out.

Fourteen crawlers. Blocking any of them costs you nothing in reach.

2.3 Recorded, never switchable

Search engines and link previews. This plugin deliberately offers no switch for them: turning one off does real damage, and that belongs in your own robots.txt rather than behind a button you can hit by accident.

Fourteen crawlers, folded away by default. Googlebot is in this group, and you cannot block it here. That is on purpose. A switch that removes a site from Google is a switch somebody eventually hits. AumCrawl still counts their visits, so the record is complete, but it writes no rule for them.

If you genuinely need to exclude a search engine, write it in your own robots.txt — a decision that deliberate should take more than one click.

A reasonable starting point: allow the first group, block the second, and leave the third alone. Look at the record after a few weeks and adjust.

3. What is actually in effect

The top section shows the rules as they are being served right now, not as you set them.

Those can differ. If a real robots.txt file exists in your web root, your web server serves that file and WordPress is never asked — so nothing AumCrawl writes has any effect. The screen says so when that is the case. Delete the file if you want the plugin to manage the rules.

Read this section after changing anything. It is the only place that shows what a crawler will actually receive.

4. The record

AumCrawl logs each visit by a crawler it recognises: which one, and when. The main screen summarises it, and each row shows its own count and last visit.

What the record is good for:

• Deciding. A crawler that has never visited is not worth thinking about.
• Noticing. One that suddenly arrives far more often than before is worth a look.
• Checking. After you block one, its count should stop rising. If it does not, that crawler is ignoring the rules, and the answer is at the server level.

A crawler that has never arrived says Never seen. One in group 2.3 shows Always allowed where the others have a switch. And a row may carry a badge like 26 could not be verified: that many visits claimed to be this crawler and the claim was actively contradicted. Anyone can put any name in a user agent.

AumCrawl checks in two ways, because operators publish two different things. Google, Bing, Apple, Baidu, Yandex and Ahrefs publish the name their addresses resolve back to. OpenAI, Anthropic, Perplexity and DuckDuckGo publish lists of their addresses instead. Both are checked, so a visit from any of them can be confirmed or contradicted.

Those lists are fetched in the background on a schedule, never while someone is being served — a page load should not wait on somebody else's server.

Some rows will still never carry that badge, and that is correct. Where an operator publishes neither, AumCrawl says nothing rather than guessing: a guess here would accuse an honest crawler of forgery, and the owner would reasonably block it. A fresh install also says nothing for the first visits, until the lists have been fetched for the first time.

Start the record over clears the counts. Use it after a deliberate change, so new numbers are not averaged with old ones.

5. Settings

The Settings section holds the switches that are not per-crawler. The important one is whether AumCrawl manages robots.txt at all. Turn it off and the plugin keeps recording without writing any rules — a reasonable way to spend the first few weeks.

6. Working with an SEO plugin

Most SEO plugins also offer to write robots.txt. Two plugins writing the same file means one wins and the other silently does nothing.

Pick one. If you keep AumCrawl for this, turn the robots feature off in the other plugin, then check section 3 to confirm what is actually being served.

7. Crawler reference

Thirty-five crawlers, as of this version. The note on each row is the same one shown in the admin.

7.1 These send traffic back

7.2 These take and give nothing back

7.3 Recorded, never switchable

8. Troubleshooting

My changes have no effect. A real robots.txt file in the web root takes priority over anything WordPress serves. Section 3 says when this is happening.

A blocked crawler keeps coming. Rules are a request. That crawler is ignoring them — block it at the server or firewall instead.

I cannot find a switch for Googlebot. There is none, on purpose — its row reads Always allowed where a switch would be. See 2.3.

A crawler I never allowed appears in the record. The record covers everything recognised, including crawlers with no rule of their own.

Counts look wrong after I changed the rules. Use Start the record over so before and after are not averaged together.

A row says "could not be verified". Someone sent that crawler's name while coming from somewhere that crawler's operator does not own. The count is kept but the claim is not trusted.

This is narrower than it sounds, and deliberately so. It does not appear when AumCrawl simply has no way to check — in that case nothing is shown at all. It appears only when the visitor's address resolves back to a name, and that name belongs to somebody else.

A crawler I expected to see verified never shows as verified. Either its operator publishes nothing to check against — AumCrawl will not invent one, so those visits are counted and left unjudged — or the published list has not been fetched yet. Lists are fetched in the background within a few hours of installing, and refreshed twice a day after that.