KiwiRankBot

Which sites it visits

KiwiRankBot only visits websites that were added to a KiwiRank account. It does not crawl the web at large.

How to recognise it

Every request carries this user agent:

Mozilla/5.0 (compatible; KiwiRankBot/1.0; +https://kiwirank.com/bot/)

How often it visits

• Once when a site is set up in KiwiRank.
• Then once a week.
• If the site asks us to slow down (HTTP 429, or 503 with Retry-After, without a bot-wall marker), up to two more visits that week, about 4 hours apart.
• When the site’s owner presses Recheck: at most once every 6 hours per site, counted from when the last Recheck that finished was requested. If a Recheck could not finish, the owner can press it again 10 minutes after it ended. If the site limited our request rate or its firewall blocked us, Recheck opens again at our next scheduled visit or 6 hours after it ended, whichever is sooner (and never within 10 minutes).

How it behaves

• It reads the home page and robots.txt, then obeys robots.txt for everything else it reads, llms.txt and the site’s sitemaps included. If robots.txt answers with a server error, it stops there.
• It makes at most 2 requests at a time.
• When a server answers 429 or 503 it drops to one request at a time, waits for Retry-After up to 30 seconds, skips pages the server keeps refusing, and stops the visit when the server asks for a longer wait.
• For 14 days after a site has limited our request rate (HTTP 429, or 503 with Retry-After, without a bot-wall marker), it visits that site gently from our own servers: one request at a time, at least 2 seconds apart, waiting for Retry-After up to 60 seconds.
• If such a site keeps limiting our own servers that way, it is read through our crawling provider’s standard proxy network instead (ScrapingBee, from addresses in the United States), still as KiwiRankBot with the same user agent, and still obeying robots.txt for every address it asks for. Anything refused there too is left unread until the next check. A site that answers our own servers with a bot wall or a challenge page is never read this way, and this route stops at the first bot wall or challenge page it meets. On that route it makes at most 2 requests at a time without the 2-second spacing, waits for Retry-After up to 30 seconds, and may ask for a page up to 3 times when the answer redirects to another host. The provider follows redirects itself, so the address a page redirects to is requested before KiwiRankBot can check it against robots.txt; a page it may not read is then left out.

How to block it

Add these lines to your robots.txt:

User-agent: KiwiRankBot
Disallow: /

If the site is in your KiwiRank account, its site health page will then show that the check could not run.

Questions

Contact support@kiwirank.com.