Free tool

Crawlability test

See whether search engines can open a page, may read it and may show it, and whether a sitemap helps them find it.

Free. We read the page, robots.txt and the sitemap, and keep nothing.

PageNot checked

Does the page open for search engines?

We open the page the way a search engine does and see how it answers.

robots.txtNot checked

Does robots.txt let Google and Bing in?

We read your robots.txt, the file that tells search engines what they may read.

Open the Robots.txt checker

NoindexNot checked

Does the page ask to stay out of search?

We look for a noindex in the page's code and in its headers.

Open the Noindex checker

SitemapNot checked

Can search engines find your sitemap?

We look where search engines look for a sitemap, then open it.

Open the Sitemap checker

What the test checks

A crawler is the program a search engine sends to read your pages, such as Googlebot for Google and Bingbot for Bing. The test reads a page the way a crawler does, in four steps: the first three decide whether the page can be shown in search results, and the fourth helps search engines find it.

  1. The page opens: its address answers with the page itself, not with an error such as 404 (not found) or 500 (server error), the status codes a server sends. A redirect loop, addresses that keep sending visitors to each other, fails too.
  2. The door lets crawlers in: robots.txt, the small file of rules for crawlers on your site, does not keep them out. The Robots.txt checker shows the line that decides.
  3. No sign asks to leave: the page has no noindex, the tag that asks search engines to leave a page out. The Noindex checker shows it.
  4. The map lists it: a sitemap, the list of your pages for search engines, helps them find the page before any other page links to it. The Sitemap checker counts what yours lists.

Google explains how it finds, reads and lists pages.

What the test cannot see

Three things only Google can tell you:

  • What a firewall lets Google see. A firewall is a guard in front of your site, such as Cloudflare. Some turn away readers they do not know, like this test, and still let Google in; the test then says it cannot tell whether search engines can read the page.
  • What JavaScript adds to the page. JavaScript is code that adds parts of a page after it arrives. The test reads the page as it arrives; Google runs the code later.
  • How often Google comes back to read the page again.

For all three, the URL Inspection tool in Google Search Console shows exactly what Google gets.

Other free checks

Questions about the crawlability test

What is crawlability?

Whether a search engine's crawler can open a page and read it.

A page it cannot read shows up in results at best as a bare address, without its text.

How do I check whether Google can crawl my page?

Use the URL Inspection tool in Google Search Console: paste the page's address and press Test live URL.

It shows whether Google could read the page and what it saw.

The test on this page checks the same basics for any site, without signing in to Search Console.

Is my site in Google?

Search Google for site: followed by your address, such as site:yoursite.com.

The Page indexing report in Google Search Console gives the full count and the reasons pages are left out.

My page passes. Why is it not in Google yet?

Passing means search engines can read the page, not that they have read it yet.

A new site that nothing links to is read last.

Search engines find it sooner when you submit the sitemap in Search Console, link to the page from your other pages (internal links) and get links from other sites.

Does the test run the page's JavaScript?

No.

It reads the page as it arrives from the server.

Google does run JavaScript, later; URL Inspection in Search Console shows the page as Google sees it after that.