Your robots.txt says GPTBot is welcome. Does your server agree?
I send live requests as twelve AI crawlers — the ones behind ChatGPT, Claude, Perplexity, Gemini and Google’s AI answers — and compare what your robots.txt declares against what your server actually returns. Then I email you what I found.
I ran this across 30 well-known sites first. About one in five did not serve the AI crawlers it officially allows — and none of them knew, because nothing reports it. No error, no warning, no line in any log. Method and raw numbers here.
What you get back
- How many of the twelve crawlers your server actually served, and how many it refused while your robots.txt allowed them.
- Whether robots.txt and llms.txt exist — llms.txt is the only direct say you get in how answer engines summarise you.
- Whether your pages carry structured data, author markup and dateModified — the three signals engines lean on when deciding whom to credit.
- How much visible text your homepage returns in raw HTML, before any JavaScript runs.
The honest part
This is a measurement, not a diagnosis. It tells you what happened on twelve requests from one network at one moment. If your site refuses my requests too, the report will say inconclusive rather than invent a number.
One email, one report. You are not added to a list and there is no follow-up sequence. If you want the specific rule that is blocking a crawler, or you want it fixed rather than described, reply to the report — that part is my paid work, and I will say so plainly rather than pretend the free check was a sales funnel.
Andrew K. — aviation engineer in Montreal who now builds and repairs multi-agent systems in Python. I came to this subject the boring way: my own pages were missing from AI answers.