Free LLMs.txt Checker
Enter a URL to check for an llms.txt file, inspect your robots.txt rules for the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot), and probe whether your CDN or firewall blocks AI bots even when robots.txt allows them. See whether AI search engines can read and cite your site.
What this llms.txt checker checks
This free llms.txt checker looks for an llms.txt file on the domain you enter and inspects your robots.txt rules for 14 well-known AI crawlers, split into the search and citation agents that answer live questions (OAI-SearchBot and ChatGPT-User for OpenAI, PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User for Anthropic, and Google-Extended for Gemini) and the training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, and Amazonbot). It tells you whether AI search engines and assistants are allowed to read your site and whether you are publishing an llms.txt to guide them.
This is about generative engine optimisation (GEO): being visible and citable in AI answers, not just the ten blue links.
Why AI crawler access matters
A growing share of search now happens inside AI assistants and answer engines that summarise the web and cite their sources. If your robots.txt blocks GPTBot, ClaudeBot, or PerplexityBot, those systems cannot read your content and will not cite you, so you lose visibility in a fast-growing channel.
An llms.txt file is an emerging convention: a plain-text file at your root that points AI crawlers at your most important, cleanest content, much as a sitemap does for search engines. It is optional today but a cheap way to signal what matters.
When robots.txt says yes but your CDN says no
robots.txt is only a request. Many CDNs and firewalls, Cloudflare among them, now block AI crawlers at the network level on their default settings, so a request identifying as GPTBot gets an HTTP 403 even though your robots.txt welcomes it. Every robots.txt-only checker misses this, because the block never touches robots.txt.
This checker probes your URL twice, once as a normal browser and once as GPTBot, and compares the two. If the browser request succeeds but the bot request is blocked, your CDN or firewall is the thing turning AI crawlers away, and no robots.txt edit will fix it. You will need to allow the verified AI crawlers in your CDN bot-management settings instead.
How to open up (or control) AI access
Decide deliberately whether you want AI systems reading your site, then make robots.txt match that intent:
- To be citable: make sure robots.txt does not Disallow GPTBot, ClaudeBot, PerplexityBot, and similar agents.
- To opt out: add explicit Disallow rules for the AI user-agents you do not want crawling you.
- Publish an llms.txt at your root linking to your key pages and documentation to help AI systems find your best content.
Honest limits
This tool checks for the presence of llms.txt and inspects robots.txt rules for the well-known AI crawlers on the domain you enter. It cannot guarantee a given AI system will or will not crawl you, since compliance is voluntary. To confirm your robots.txt is not also blocking Google itself, pair it with the robots.txt checker. For the full set of AI Visibility (GEO) checks alongside your traditional SEO, with AI fix instructions for each issue, run a full free audit.
Frequently asked questions
Is this llms.txt checker free?
Yes, free with no signup, up to 15 checks per hour.
What is llms.txt?
It is an emerging plain-text file at your site root that points AI crawlers at your most important content, similar in spirit to a sitemap for search engines.
Which AI crawlers does it check?
It inspects robots.txt for 14 agents: search and citation bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Google-Extended) and training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot).
Should I let AI crawlers read my site?
If you want to be cited in AI answers, yes. If you would rather opt out, add explicit Disallow rules for those user-agents.
Why would AI crawlers be blocked if my robots.txt allows them?
Because robots.txt is only one layer. Your CDN or firewall (Cloudflare, for example) can block AI crawlers at the network level regardless of robots.txt, returning a 403 to GPTBot while a browser loads fine. This checker probes as both a browser and GPTBot to catch that mismatch, then you fix it in your CDN bot settings, not robots.txt.
Do I need an llms.txt file?
It is optional today, but a low-effort way to guide AI systems to your best content. Run a full free audit for the complete AI Visibility check set.