reference

robots.txt

Four lines, and only one of them carries information. A missing robots.txt already allows everything, so Allow: / is a statement rather than an instruction — the Sitemap: line is the part that tells a crawler something it did not know.

User-agent: *
Allow: /

Sitemap: https://example.org/sitemap.xml

This is also where reach gets lost quietly. The most authoritative source for a claim on the reach page was simply disallowed to the crawler in its own robots.txt: no error, no symptom, just absence. Read yours and check it says what you think, particularly if a platform or a plugin generated it.

Named AI crawlers can be allowed or blocked by user-agent here. That is a decision about reach, not a security measure — the file is a request that well-behaved crawlers honour, and nothing else is obliged to. This site's own is at /robots.txt.