Figma blocks GPTBot in robots.txt and serves it anyway. WordPress.com allows four crawlers and rate-limits all of them. Here is the data, and how to check your own site in a minute.
Everyone treats robots.txt as the answer to “can ChatGPT read my site?”
It is not an answer. It is a statement of intent. The actual decision happens somewhere else entirely: at Cloudflare, at a WAF, in a bot-fight rule, in a rate limiter keyed on User-Agent. Nobody reconciles the two. Your robots.txt can say welcome while your edge says 403, and nothing in your stack will ever tell you.
So I measured it. I sent live requests as 12 real AI crawlers to 30 major websites and compared what each site declares with what it actually serves.
The method, stated plainly
For each site: fetch robots.txt, parse the rules for each crawler, then make a real HTTP request using that crawler’s exact User-Agent and record the status code.
Want the same numbers for your own site?
I run the identical test on request — twelve AI crawlers, live requests, compared against what your robots.txt declares. You get one email with what I found. No list, no follow-up sequence.
One control matters more than everything else: before comparing bots, I fetch the site with an ordinary Chrome User-Agent. If that fails too, the site is blocking my network, not AI crawlers, and reporting “12 crawlers blocked” would be a lie. Those sites are excluded, not counted as findings.
That control disqualified 7 of 30 sites. Canva, GoDaddy, Namecheap, Etsy, Udemy, Medium and Substack all return 403 to everything from a datacenter IP, browser included. Which is its own finding. Nearly a quarter of major sites treat datacenter traffic as hostile by default, and any tool that skips the control fetch will happily report them as “blocking AI”.
That leaves 23 sites where a fair comparison is possible.
Result: 5 of 23 disagree with their own robots.txt
21%. Roughly one in five.
Two different failure modes, and the second one is the interesting half.
Mode 1: allows in writing, refuses in practice
Notion. Its robots.txt welcomes Bingbot. The edge returns 403 with a bot challenge page. Bing powers Copilot, so this is a live channel quietly closed.
Slack. Google-Extended and Bingbot both get 429. Not a block, a rate limit. To a crawler, the difference is academic: it did not get the page.
WordPress.com has the widest gap in the set. Google-Extended, Amazonbot, CCBot and meta-externalagent all receive 429, while Chrome from the same network gets 200. Four crawlers, all permitted on paper, all turned away.
I verified this one by hand outside my own tooling:
Google-Extended -> HTTP 429 Amazonbot -> HTTP 429 CCBot -> HTTP 429 Chrome -> HTTP 200
Mode 2: forbids in writing, serves anyway
This is the one nobody checks, because nobody thinks to.
Figma‘s robots.txt contains, explicitly:
User-Agent: GPTBot Disallow: /
And the server’s actual response to GPTBot:
GPTBot -> HTTP 200, 1,593,059 bytes Chrome -> HTTP 200, 1,593,059 bytes
Byte for byte identical. The same is true for Google-Extended and CCBot, which are also disallowed in the file.
To be precise about what this means: robots.txt is voluntary. A well-behaved crawler reads it and leaves. Figma has not failed at anything technically — they simply have no enforcement behind the declaration, and are trusting the honour system. Which is fine, right up until someone assumes the file is doing work it is not doing.
Coursera shows the same pattern with meta-externalagent.
Which crawlers get caught most
| Crawler | Sites where declaration ≠ reality |
|---|---|
| Google-Extended | 3 |
| Bingbot | 2 |
| CCBot | 2 |
| meta-externalagent | 2 |
| GPTBot | 1 |
| Amazonbot | 1 |
Google-Extended leading is not an accident. It governs whether your content trains and feeds Google’s AI surfaces, and it gets caught in generic rate limits more than any other, because nobody thinks of it as a “search” bot when writing the allow-list.
One more number
18 of 23 sites (78%) now serve an llms.txt. A year ago that file barely existed. Whatever you think of the format, the market has decided it is table stakes. If you do not have one, you are now in the minority.
Why this is worth your five minutes
If your content stopped showing up in ChatGPT, Perplexity or AI Overviews, the first instinct is to blame the content. The data says: check the plumbing first.
Nothing in your analytics will surface this. Crawlers do not file complaints. A 403 to GPTBot looks exactly like nobody asking.
And it usually is not a decision anyone made. It is the accidental result of turning on bot protection, tightening a rate limit, or inheriting a robots.txt written for a different internet.
Check your own site
I packaged this as a tool. It sends the same live requests as 12 crawlers, includes the control fetch so a datacenter block is never reported as an AI policy, and additionally checks extractability — whether structured data, author attribution and dates are present, because being fetched and being cited are different problems.
AI Crawler Reality Check. One URL, about 20 seconds.
Check the exact page you care about, not just the homepage. Protection rules routinely differ per path, and the article you want cited is rarely the front page.
If the tool finds a gap
The test tells you the plumbing is broken. It does not tell you what it is costing you.
That is the AI Visibility Audit: 40 real buyer prompts run against ChatGPT, Perplexity, Gemini and Google AI Overviews, the crawler-access matrix above run across your whole site, competitor citation share, and a ranked list of technical fixes — implemented, not just listed. Report and priority fixes typically ship within a week of the prompt list being confirmed. $1,500. White-label available for agencies.
No one can guarantee a model cites you. I remove the reasons it does not, and measure the change 30 days later with the same method both times.
See the audit →
or start with the free snapshot — 5 prompts, no email gate
Limits, honestly
- This runs from a datacenter IP. Residential traffic is sometimes treated differently, which is exactly what the control fetch exposes rather than hides.
- It measures whether a crawler is served, not whether a model chose to cite you. No public API exposes citation data. Anyone selling you that number is estimating.
- A 429 is a rate limit, not a permanent block. Repeated over time it has the same effect, but it is a different fix.
- Sites change. This snapshot is 27 July 2026.
I build and repair multi-agent systems in Python. This started because I wanted to know why my own articles were invisible to AI search, and found the answer was not in my content.
Andrii Klymenko. Avionics engineer turned solo builder. I write about what actually breaks in production systems, with the logs attached.
📚 Related Articles
- 30 AI Art Generators Tested: Only 4 Beat Canva
- 30 AI Art Generators Tested: Only 4 Worth Paying For
- 30 AI Prompt Generators Tested: Top 4 That Save Time
- I Built a Fully Autonomous AI Business. 111 Days, 149 Articles and 63 Cron Jobs Later, It Made $54.
Get the free AI Automation Starter Kit
Ready-to-use workflows and prompts I actually run in a live, 24/7 AI-automated business — no fluff, instant access.
🚀 Level Up Your AI Game
Get weekly AI tools, prompts & automation strategies — free, every week.
No spam. Unsubscribe anytime.