Free tool

Robots.txt checker

Test and validate your site's robots.txt: see which search engine and AI crawlers may read a page, and which line of the file decides it.

Free. We read your robots.txt the way Googlebot does, and keep nothing.

Type a page address above to see which crawlers may read it.

Search engines

AI crawlers

robots.txt

Your robots.txt shows here, line by line, with the line that decides marked and every line crawlers would skip noted.

Which rules each crawler follows

Search engines and AI companies send programs called crawlers to read your pages. A robots.txt is a short file of rules for them, in groups: each group starts with User-agent and a crawler's name, its user agent, and its Disallow lines say what that crawler may not read.

Googlebot follows the group under User-agent: Googlebot. When no group names it, it follows User-agent: *, the group for everyone else. So one file can let Google in and keep another crawler out.

Which rule wins when two rules match

Two rules can match the same page. Take /blog/post, the part of a page's address after your site's name: both Disallow: / and Allow: /blog/ match it.

Google follows the rule with the longer path, so Allow: /blog/ wins. When both are just as long, Allow wins.

That is how Google reads the file, and the check at the top of this page follows it line by line.

AI crawlers for answers and for training

Some AI crawlers find pages for answers. OAI-SearchBot does it for ChatGPT search, and PerplexityBot for Perplexity. Perplexity says PerplexityBot is not used for training.

Others take pages for training. GPTBot collects them for OpenAI's models, and Anthropic's crawler, ClaudeBot, collects pages that may be used to train Claude. Google-Extended is not a crawler but a setting: a User-agent name you can keep out to keep your pages out of Gemini training. Google says it does not change whether or where a site shows in Google Search.

OpenAI says each of its crawlers follows its own rules. So you can keep the training crawlers out and still be found by ChatGPT search and Perplexity.

What robots.txt cannot do

robots.txt keeps crawlers from reading a page. It does not keep the page out of search results.

When other pages link to a blocked page, Google says it can still appear in its results, without a description.

To keep a page out, let Google read it and add a noindex tag: a line in the page's code that asks search engines to leave it out of their results. The Noindex checker shows whether a page has one.

Six common mistakes

  1. A “Disallow: /” left over from building the site. It keeps every crawler out of everything.
  2. A misspelled rule, such as “Dissalow”. Google forgives a few common slips; other crawlers skip the line.
  3. A robots.txt address that answers with an error, such as 503, instead of the file. Google pauses crawling the site for about 12 hours, then falls back on the last copy it saved.
  4. Blocking the style and script files a page needs, so Google cannot see the page the way visitors do.
  5. Blocking a page that carries a noindex. Google never reads the page, so it never sees the noindex.
  6. A Sitemap line that still names an old address. A sitemap is the list of your pages for search engines; the Sitemap checker finds the one they use and checks it.

Other free checks

Questions about the robots.txt checker

Where is my robots.txt?

Always at the root of the site, the top level of its address: yoursite.com/robots.txt.

Each subdomain, such as shop.yoursite.com, has its own file.

What should a robots.txt file look like?

A few short lines of plain text.

A common one has three: User-agent: * for every crawler, Disallow: /admin/ to keep them out of one folder, and Sitemap: followed by your sitemap's full address.

User-agent: * with nothing after Disallow: lets every crawler read everything.

Do I need a robots.txt at all?

No.

Without one, every crawler may read every page, which is what most sites want.

The file is only needed to keep crawlers out of something.

How do I add my sitemap to robots.txt?

Add one line anywhere in the file: Sitemap: followed by the sitemap's full address, such as Sitemap: https://yoursite.com/sitemap.xml.

Every major search engine reads it, and one file can name several sitemaps.

What do "Blocked by robots.txt" and "Indexed, though blocked by robots.txt" mean?

They are messages in Google Search Console.

Blocked by robots.txt means your file keeps Googlebot from reading the page; if the page belongs in Google, remove the line that blocks it, and the check at the top shows which line that is.

Indexed, though blocked by robots.txt means Google lists the page without reading it, because other pages link to it.

To take it out of Google, remove the block and add a noindex: Google must read the page to see the noindex.

How do I edit robots.txt in WordPress?

WordPress makes a robots.txt of its own until a real file sits at the root of the site.

To change it, use the Yoast SEO plugin (Tools, then File editor) or the Rank Math plugin (General Settings, then Edit robots.txt), or upload your own robots.txt file to the site's root folder.

How do I let ChatGPT and other AI crawlers in?

A crawler that no group names follows User-agent: *, so if that group lets everyone in, they are in already.

To be found in ChatGPT search, make sure no group keeps out OAI-SearchBot, the crawler ChatGPT search uses; the check at the top shows what each AI crawler may read.

Does keeping GPTBot out hide my site from ChatGPT?

No.

GPTBot collects pages to train OpenAI's models.

ChatGPT search finds pages through OAI-SearchBot, which follows its own rules in the file.

Does Google read Crawl-delay?

No.

Google ignores Crawl-delay, a line that asks crawlers to wait between pages.

Bing reads it.

How big can a robots.txt be?

Google reads about the first half a megabyte (500 KiB) and ignores the rest.

A normal robots.txt is far smaller than that.