Meta-WebIndexer
Operated by Meta · Builds a search index
If you block it
You stop being surfaced in Meta AI search results.
What Meta says it is for
Meta says it crawls to improve the quality of Meta AI search results.
Published details
- robots.txt
- Operator publishes no documentation
- robots.txt caching
- Meta says changes take up to 24 hours to take effect, because its crawlers use a cached copy.
- User agent
- Not published as a full string. Meta publishes only the token.Match on the prefix
meta-webindexerrather than the whole string: operators version these, and an exact match stops working on the next release. - Verification
- No published method.
Not addressed in the documentation
- As of , Meta’s documentation does not address whether it respects robots.txt — the page makes no statement about this one.
Source: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers Read .