Which rules each crawler follows
Search engines and AI companies send programs called crawlers to read your pages. A robots.txt is a short file of rules for them, in groups: each group starts with User-agent and a crawler's name, its user agent, and its Disallow lines say what that crawler may not read.
Googlebot follows the group under User-agent: Googlebot. When no group names it, it follows User-agent: *, the group for everyone else. So one file can let Google in and keep another crawler out.
Which rule wins when two rules match
Two rules can match the same page. Take /blog/post, the part of a page's address after your site's name: both Disallow: / and Allow: /blog/ match it.
Google follows the rule with the longer path, so Allow: /blog/ wins. When both are just as long, Allow wins.
That is how Google reads the file, and the check at the top of this page follows it line by line.
AI crawlers for answers and for training
Some AI crawlers find pages for answers. OAI-SearchBot does it for ChatGPT search, and PerplexityBot for Perplexity. Perplexity says PerplexityBot is not used for training.
Others take pages for training. GPTBot collects them for OpenAI's models, and Anthropic's crawler, ClaudeBot, collects pages that may be used to train Claude. Google-Extended is not a crawler but a setting: a User-agent name you can keep out to keep your pages out of Gemini training. Google says it does not change whether or where a site shows in Google Search.
OpenAI says each of its crawlers follows its own rules. So you can keep the training crawlers out and still be found by ChatGPT search and Perplexity.
What robots.txt cannot do
robots.txt keeps crawlers from reading a page. It does not keep the page out of search results.
When other pages link to a blocked page, Google says it can still appear in its results, without a description.
To keep a page out, let Google read it and add a noindex tag: a line in the page's code that asks search engines to leave it out of their results. The Noindex checker shows whether a page has one.
Six common mistakes
- A “Disallow: /” left over from building the site. It keeps every crawler out of everything.
- A misspelled rule, such as “Dissalow”. Google forgives a few common slips; other crawlers skip the line.
- A robots.txt address that answers with an error, such as 503, instead of the file. Google pauses crawling the site for about 12 hours, then falls back on the last copy it saved.
- Blocking the style and script files a page needs, so Google cannot see the page the way visitors do.
- Blocking a page that carries a noindex. Google never reads the page, so it never sees the noindex.
- A Sitemap line that still names an old address. A sitemap is the list of your pages for search engines; the Sitemap checker finds the one they use and checks it.
Other free checks
- Noindex checker: whether a page asks search engines to leave it out of their results.
- Sitemap checker: whether search engines can find and read your sitemap, and what it lists.
- Crawlability test: whether search engines can open a page, read it and show it in their results.
- Submit your website to search engines: a guide to putting your site into Google and Bing, step by step.
Questions about the robots.txt checker
Where is my robots.txt?
Always at the root of the site, the top level of its address: yoursite.com/robots.txt.
Each subdomain, such as shop.yoursite.com, has its own file.
What should a robots.txt file look like?
A few short lines of plain text.
A common one has three: User-agent: * for every crawler, Disallow: /admin/ to keep them out of one folder, and Sitemap: followed by your sitemap's full address.
User-agent: * with nothing after Disallow: lets every crawler read everything.
Do I need a robots.txt at all?
No.
Without one, every crawler may read every page, which is what most sites want.
The file is only needed to keep crawlers out of something.
How do I add my sitemap to robots.txt?
Add one line anywhere in the file: Sitemap: followed by the sitemap's full address, such as Sitemap: https://yoursite.com/sitemap.xml.
Every major search engine reads it, and one file can name several sitemaps.
What do "Blocked by robots.txt" and "Indexed, though blocked by robots.txt" mean?
They are messages in Google Search Console.
Blocked by robots.txt means your file keeps Googlebot from reading the page; if the page belongs in Google, remove the line that blocks it, and the check at the top shows which line that is.
Indexed, though blocked by robots.txt means Google lists the page without reading it, because other pages link to it.
To take it out of Google, remove the block and add a noindex: Google must read the page to see the noindex.
How do I edit robots.txt in WordPress?
How do I let ChatGPT and other AI crawlers in?
Does keeping GPTBot out hide my site from ChatGPT?
No.
GPTBot collects pages to train OpenAI's models.
ChatGPT search finds pages through OAI-SearchBot, which follows its own rules in the file.
Does Google read Crawl-delay?
No.
Google ignores Crawl-delay, a line that asks crawlers to wait between pages.
Bing reads it.
How big can a robots.txt be?
Google reads about the first half a megabyte (500 KiB) and ignores the rest.
A normal robots.txt is far smaller than that.