AI Guide
Why AI crawlers cannot access your site
When a site feels invisible to AI tools, the cause is often boring and technical rather than mysterious: fetchability, policy or discovery.
1. The homepage is not fetchable
If the homepage does not return a normal public response, later discovery steps become much less relevant. Redirect loops, challenge pages, temporary blocks and unreliable hosting can all break first contact.
2. robots.txt sends the wrong signal
A single broad disallow can be enough to make teams think “AI just is not picking us up” when the real issue is that the policy file already answers the question.
3. Discovery files are missing or weak
No sitemap, no linked sitemap in robots.txt, or a stale llms.txt all increase ambiguity. None of these guarantees inclusion, but each one reduces unnecessary friction.
4. The content exists, but the access path is confusing
Heavy client-side rendering, weak internal linking, fragmented public docs and inconsistent canonical URLs can make a site harder to interpret even when the server technically responds.
The useful debugging order
- Confirm the homepage is fetchable.
- Read robots.txt as if you did not know the intended policy.
- Check whether tracked bots appear allowed or blocked.
- Verify sitemap and llms.txt reachability.