You checked robots.txt, it allows everything, yet AI-referral traffic is flat or dropping. The block is almost certainly one layer down: your server or CDN.
Two gatekeepers, not one
robots.txt is a request you make of well-behaved crawlers. Your CDN's bot management is enforcement that happens before a request reaches your app — it can return 403/429 to a crawler regardless of robots.txt. Since Cloudflare's Sept 15, 2026 default AI blocking, many sites are open in robots.txt but closed at the edge.
How to tell which layer is blocking
Compare responses: a normal browser vs each crawler's user-agent, hitting the same page. If the browser gets 200 and the crawler's UA gets 403, that's edge-level blocking — not robots.txt. Our live crawler check does exactly this comparison and labels the result. Then fix it in your CDN's bot settings, not robots.txt.
One caveat worth knowing: some CDNs verify a crawler's IP, so a real verified GPTBot may be allowed while an unverified look-alike (including our probe) is blocked — meaning the true crawler might get through even when the probe doesn't. The report notes this.