AI Guide
robots.txt for AI bots
robots.txt is usually the first policy file people inspect when they try to understand whether AI crawlers are welcome, blocked, or simply left ambiguous.
What it does well
robots.txt gives you a public, auditable place to express crawler rules. That matters because teams often need a simple answer to a simple question: did we intentionally allow this bot, or did we block it by accident?
- Use explicit user-agent sections for important AI bots instead of relying only on wildcard logic.
- Keep the file readable. Complex rule stacks are harder to reason about and easier to misread during incidents.
- List your sitemap to make discovery paths clearer for both people and crawlers.
Where teams trip up
The common failure mode is not “we forgot robots.txt exists.” It is “we used one defensive rule months ago and now nobody is sure what it blocks.” Blanket root disallows, inherited wildcard sections and copied vendor snippets are the usual culprits.
A practical pattern
Start from a file that is short enough to review in one pass, then decide on purpose which bots belong in explicit sections.
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://example.com/sitemap.xml
Whether you allow all of these is a business decision. The point is clarity, not a universal allow list.
What the audit helps confirm
The AI crawlability audit checks whether robots.txt is fetchable and whether tracked bots appear allowed, blocked, or ambiguous at the root path. That is useful when you need a quick sanity check before you change policy.