: Google changing crawl speed: doesn't seem to work. Why? I've changed 3 days ago the google crawling speed of mywebsite. Here it is: This means: 2 demands by second. I've got the message on

@Courtney195

As is written in google faqs:


Crawling is the process by which Googlebot discovers new and updated
pages to be added to the Google index.

We use a huge set of computers to fetch (or "crawl") billions of pages
on the web. The program that does the fetching is called Googlebot
(also known as a robot, bot, or spider). Googlebot uses an algorithmic
process: computer programs determine which sites to crawl, how often,
and how many pages to fetch from each site.

Google's crawl process begins with a list of web page URLs, generated
from previous crawl processes, and augmented with Sitemap data
provided by webmasters. As Googlebot visits each of these websites it
detects links on each page and adds them to its list of pages to
crawl. New sites, changes to existing sites, and dead links are noted
and used to update the Google index.


Said that, we can say that: The more google visits your site, the more information would be indexed. The problem with that is that the traffic to your server will be affected.
Google's resources are involved too... so, if your page doesn't need to be crawled again (google knows), they will not follow the speed you set.
That's an option, but it is not a rule.

If your site could need it, google will do it (at least always in the beginning, until they notice that you don't really need to index the information so often),

Vote Up Vote Down

@Hamm4606531

[Update]Okay, after a few days of analyzing, it seems that Google doesn't actually asks for the "crawling speed" he should use, but for the "**maximum crawling speed" the webserver can handle. This makes a huge difference to me.

So, for example, I've set it up to 10 connexions per seconds, and sometimes it's only 1 "crawl" every ten seconds, and sometimes it goes up to to 10 "crawls" per second.

Vote Up Vote Down