What stops an AI system retrieving your site.
Indexed is not the same as retrievable, and the failures here are silent: nothing errors, the page looks fine, and the answer simply never includes you.
What stops an AI system from finding and quoting our pages?
Three failures account for most of what we find, and all three are invisible from the browser. A person loading the page sees exactly what they expect.
1. The answer is not in the HTML
If the substance of the page arrives after JavaScript runs, a retrieval system fetching the URL may receive a shell: a header, a footer and a loading state. Search engines will often render and index it anyway, so the page can read as healthy in a coverage report while the thing worth quoting was never in the response.
We found exactly this on one of our own properties. The page was indexed, the report was green, and what a crawler actually received was a spinner. The check that matters is what comes back from a plain fetch of the URL, not what the browser paints.
2. A robots directive that does more than intended
Robots files are matched by the most specific group, not by accumulation. Adding a named group for one crawler means that crawler now follows only that group, and every rule sitting under the wildcard stops applying to it.
We did this to ourselves across several properties while trying to be explicit about AI crawlers: naming them replaced the private-path rules they had been inheriting. Nothing errored and nothing warned. It is worth re-reading the production file after any robots change, because the rule that stops applying is never the one you were thinking about.
The related distinction, which is often collapsed: a crawler that fetches pages to answer a question now is not the same as a crawler that collects text for training. Blocking the second is a legitimate business decision with no effect on being cited. Blocking the first removes the business from answers being composed today. They are separate decisions and should never be made with one line.
3. The page is reachable but not quotable
A retrieval system prefers a passage it can lift without distorting it. Pages that answer the question only across four scattered sections, or that put the answer behind an accordion, or that state it in a way that depends on the paragraph above, are harder to quote correctly. They get skipped in favor of a page that answered plainly.
This is a content property rather than a technical one, and it is the one most often missed by tooling.
Why all of this belongs in tracking rather than an audit
Every one of these is introduced by ordinary work: a template change, a platform migration, a well-meant robots edit, a redesign that moves content into a component. They are not mistakes anybody would report, and none of them produces a notification. A re-read catches them while they are one line, which is the entire economic argument for watching continuously instead of auditing occasionally.
This is one question beneath AI visibility tracking, the ongoing work Digilu does through The Observatory. The free point-in-time baseline is AIOInsights. Digilu cannot make a private AI model recommend a business. It can make the public evidence clearer, stronger and easier to verify, then track whether visibility improves.