6-Node Multi-Edge Live Fetcher · RFC 9309 Compliant
Free AI Crawler & LLMs.txt Policy Checker
What is an AI Crawler Checker? An AI Crawler Checker fetches your live robots.txt, llms.txt, and HTML <meta name="robots"> tags in real time to verify whether AI search engines (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) and user fetchers (ChatGPT-User) can crawl and cite your URLs.
01 / Live Domain Scanner & Policy Editor
Meta Robots: Ready
llms.txt: Ready
Quick Test Live Domains:
Interactive RFC 9309 Parser
100%
Citation Ready
5/5
Search Bots
3/3
User Fetchers
0/6
Training Bots
How Citation Scoring Works: Your score reflects Search Citation & User Fetcher access. Blocking model training crawlers (GPTBot, Google-Extended) does not penalize your AI search visibility.
| Crawler User-Agent | Lane / Role | Status | Matched Rule | 1-Click Edit |
|---|
Auto-Generated
/llms.txt File for This Domain