Definition
An AI crawler is a bot operated by a model vendor to fetch web content — for training, for live retrieval, or for both. Each identifies itself with its own user agent.
Why it matters
Being fetched is the precondition for being cited. A site that blocks these agents, deliberately or through a CDN default, has ruled itself out before any content work begins.
How it is used
Test by sending a request with each crawler's user agent and checking the status code. Reading the robots.txt file is not a test.
Related terms
Concepts that only make sense together
Find out where your brand actually stands in AI answers.
An AI visibility diagnosis across the platforms your customers use, on your real questions. No charge for the first look.