AI Guide
How to allow AI bots without being careless
Letting AI crawlers access public content should be an intentional publishing decision, not an accidental side effect of a default setup.
Start with the public pages that matter
The first question is not “which bot names do we paste into robots.txt?” It is “which pages do we actually want discoverable?” Marketing pages, docs, help content and blog posts are usually the clearest candidates.
- Keep important pages server-rendered and publicly fetchable over HTTPS.
- Avoid bot challenges on content that is meant to be seen by unknown crawlers.
- Make sure canonical public pages are linked internally and easy to discover.
Then express the policy clearly
Once the target pages are clear, use robots.txt to state access policy for the user-agents you care about. If you want them allowed, say it plainly. If you want them blocked, say that plainly too.
Allowing bots is not the same as guaranteeing citations
Crawlability helps content become available for fetching. It does not guarantee ranking, answer inclusion, attribution, or how often a model refreshes its view of your site.
Checklist before you ship the change
- Homepage returns 200 and is visible without auth.
- robots.txt exists and is easy to review.
- Sitemap is listed and publicly reachable.
- Any intentional blocks are documented internally.