Our crawler

TheSEODetectiveBot

TheSEODetectiveBot makes the requests for investigations that visitors start on this website. It does not crawl the web on its own.

How it identifies itself

Mozilla/5.0 (compatible; TheSEODetectiveBot/1.0; +https://theseodetective.com/bot)

What it requests

For each investigation: the submitted page, its http:// and www/non-www variants, robots.txt, the XML sitemap, the canonical URL if it differs, up to 25 internal links (status check only) and up to 6 of the page’s own CSS, JavaScript or image files (headers only). Requests are made one at a time, with a short pause between link checks.

A site crawl requests the site’s pages one at a time, at most one request every half second (or slower, if robots.txt sets a Crawl-delay for us, up to five seconds), up to the page limit chosen by the visitor (at most 1,000). Only one crawl of a site runs at a time. Links to other sites found during a crawl are checked with a single HEAD request each, and only where that site’s robots.txt allows it.

The individual SEO tools make the requests their check needs: for example the redirect checker requests the http/https and www variants of a URL, the sitemap validator reads up to 25 sitemap files and requests up to 25 listed URLs, and the broken link checker requests each link on one page (at most 150), honouring the robots.txt of every site it contacts.

It never submits forms, logs in, runs JavaScript or probes for vulnerabilities. The same website can be investigated a limited number of times per hour across all visitors.

robots.txt

The page a visitor submits is fetched once, as a browser would, because a person asked for it. Every further URL found on that page, and every page of a crawl, is requested only if robots.txt allows it for TheSEODetectiveBot. To stop crawls and link checks on your site:

User-agent: TheSEODetectiveBot
Disallow: /