AI Guide
llms.txt explained without the hype
llms.txt is best understood as a helpful hint file for AI-oriented consumption, not as a magic switch that controls all model behavior.
What it can help with
If you publish docs, product information or structured public guidance, llms.txt can point readers and tools toward the most useful resources. It is a way to express editorial intent, not a replacement for public crawlability basics.
What it does not replace
- It does not replace robots.txt for crawler rules.
- It does not make an inaccessible homepage accessible.
- It does not replace a sitemap for ordinary site discovery.
- It does not guarantee that any specific model will obey it.
The useful mental model
Treat llms.txt as supplementary orientation. If your base site is hard to fetch or your policies are unclear, llms.txt will not fix the foundation. If the base site is healthy, it can add context.
How the audit uses it
The AI crawlability audit checks whether llms.txt is discoverable so you can separate “nice supplementary file exists” from “core crawlability is working.” That keeps teams from over-weighting a secondary signal.