Crawler policy
CheckWebsiteSEOBot
We audit how other crawlers treat your website, so it would be poor form not to be completely clear about our own. This page is the reference for server administrators.
How to identify us
Mozilla/5.0 (compatible; CheckWebsiteSEOBot/1.0; +https://checkwebsiteseo.com/crawler)
We never impersonate Googlebot, Bingbot or any other crawler. Every request we make
carries this user agent and a From header of
crawler@checkwebsiteseo.com.
What we fetch
HTML pages, robots.txt, sitemaps, and HEAD requests to check whether linked URLs resolve. We do not download large media files.
How often
Only when a customer who has added your site asks for a scan, or on their scheduled cadence — at most daily on the highest tiers.
How fast
One request at a time by default, with a delay between requests. We honour Crawl-delay in robots.txt.
What we store
Page metadata, structured data, link targets and content hashes. We do not republish your content.
How to slow us down or block us
We obey robots.txt. To limit or stop our crawler, add either of these:
# Slow us down User-agent: CheckWebsiteSEOBot Crawl-delay: 10 # Or block us entirely User-agent: CheckWebsiteSEOBot Disallow: /
Blocking us does not affect how search engines or AI systems see your site — but it does mean any customer auditing your website will see fewer results, and will be told why.
Questions or a problem with our crawling? crawler@checkwebsiteseo.com