Robots.txt checker and tester

Validate a robots.txt file and test any path against it, using the same matching rules Google documents (RFC 9309).

Free, no account needed. Requests are made by TheSEODetectiveBot and only reach publicly accessible addresses.

What this tool checks

  • Whether /robots.txt responds, and how crawlers treat its status code
  • Syntax: lines crawlers cannot understand
  • User-agent groups, Allow/Disallow rules and Crawl-delay
  • Declared sitemaps
  • Whether the page is blocked for Googlebot or all crawlers
  • A tester for any path and user agent

How to read the results

Within the group for a crawler, the longest matching rule wins, and Allow wins a tie. A crawler uses only the most specific group naming it, otherwise the * group. A server error on robots.txt makes Google treat the whole site as blocked.

Problems are confirmed technical faults. Opportunities are optional improvements. Each finding shows the evidence we observed and a recommended fix.

Questions

Does Disallow remove a page from Google?

No. Disallow stops crawling, but a blocked URL can still be indexed from links without its content. To keep a page out of results, allow crawling and use a noindex directive.

Is Crawl-delay supported?

Google ignores Crawl-delay; Bing and some other crawlers honour it. The checker shows it so you can see what is declared.

More SEO tools