Why Google doesn’t Support Crawl Delay?

Every website operator might know about a robot. txt which is also known as the robot exclusion protocol or standard. It is a text file that tells web robots (search engine) which page on your site to crawl.  Basically, this file is used by the website to communicate with web crawlers and other web robots. And if the robot.txt file is not present then the search engine will crawl the entire website. Robots.txt plays a significant role in SEO of the website as it tells the search engines how they can best crawl the site. Search engines check the website’s robots.txt file regularly to find instructions for crawling the site. These instructions are called directives. One such directive of robot.txt is “Crawl Delay” which helps webmasters manage aggressive crawlers from overwhelming their site. But now webmasters won’t longer use crawl delay in robot.txt file as Google has stopped supporting crawl delay.

Before I talk more about why Google doesn’t support crawl delay anymore, first let me clear you with the concept of Crawl Delay below so that you can easily grasp the whole matter.

What exactly Crawl Delay is?

Crawl-delay is an unofficial robot.txt directive, is used to prevent overloading servers with a large number of requests. Crawl-delay as the name itself says the delay of the bot timing of crawling a page.  It sets a time of 2-3 seconds so that the search bot crawls the website at a moderate rate and avoids causing traffic peaks. Search engines such as Bing, Yahoo, and Yandex at times crawl-hungry and they can be slowed down using this directive to which they respond.

Furthermore, a crawl delay setting tells the bot to wait for a specific amount of time between two requests. Crawl-delay is effective to attract bots not to consume extensive hosting resources. However, you need to be careful while using this directive in the robot.txt file.

Moving on to why webmasters use Crawl Delay?

If not every website, but many sites have a large number of pages and a lot of them are linked from the index. So when the bot starts crawling the site, it generates too many requests to the website for a short time period. Such traffic can drain the hosting resources recorded on an hourly basis. In case, you’re facing such a problem, you can set the crawl-delay setting of 2-3 seconds to let search bot to crawl your site and avoid the traffic peak. Crawl-delay can be used in social bookmarking sites like twitter, Facebook, etc. which frequently update.

When you set a crawl-delay of 10 seconds it means the search engine would wait for 10 seconds before re-accessing the website after crawling once. If you don’t know the let me tell you that a lot of pages and contents like e-commerce sites would be in trouble as crawlers can drain the resources pretty quickly.  Using robot.txt. file might keep the bots from accessing images and scripts to retain resources for site visitors.

However, after knowing its many uses Google has stopped supporting crawl delay. Let’s understand why.

Google doesn’t support Crawl-Delay. Why?

First of all, the crawl delay directive for robot.txt file was introduced by search engines like Bing, Yahoo, and Baidu. The purpose of introducing crawl delay was to let webmasters specify how many seconds a search engine crawler should wait between single requests in order to limit the load on a server.

Recently Google’s John Mueller said on Twitter that Google “never supported crawl-delay in robots.txt.” Bing’s Frédéric Dubut said on Twitter “Bingbot supports crawl-delay.” 

I tell you the reason why Google doesn’t support crawl-delay. It is because their servers are dynamic and following the time frame between requests doesn’t make sense for it. Google says the value specified in seconds for the time between requests is not so useful now as most servers are capable of handling so much traffic per second. Google automatically adjusts the crawling depending on the server reaction rather than following the crawl-delay rule. In case, the server gets slow down or shows an error then it slows the crawling. Webmasters can easily specify in the robot.txt file about the parts of their websites they don’t want to get crawled.

Final words!

I hope now you know why Google doesn’t support crawl-delay in robot.txt file. Google doesn’t support it also because it not an official directive as Google said, “In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.”

For more updates get in touch with us till then keep reading.