: How to decrease the frequency of Google/Search-engine bots crawling my site? My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my

@Cofer257

You should also specify a crawl delay in robots.txt for all the other search engines (Yandex and Baidu can be quite aggressive in their crawling). Add this:

User-agent: *
Crawl-delay: 5


The crawl delay is in seconds. Make sure not to go too high - 5-10 seconds max should lighten the server load considerably. If you have 1000 new pages per day you want search engines to be able to find them all.

However the best method for Google (and possibly Bing) is still using their webmaster tools, as Google ignores the crawl-delay directive in robots.txt.

Vote Up Vote Down

@Eichhorn148

I think this help document from Google should be solving my problem:


Change the crawl rate:


On the Webmaster Tools Home page, click the site you want.
Click the gear icon , then click Site Settings.
In the Crawl rate section, select the option you want.


The new crawl rate will be valid for 90 days.

Vote Up Vote Down

@Dunderdale272

The de facto standard for telling robots what to crawl is robots.txt.

Make the URLs of your posts fit a pattern such that you can generate a robots.txt that selects just the new postings. You can generate it automatically with a script, e.g. on the fly (CGI script) or in a batch job you run every hour, or in some other way.

Vote Up Vote Down