: Googlebot algorithm for crawling? Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?

@Shelley277

Search engines never reveal the way their algorithms work or people will just start exploiting them... Google also stopped respecting the meta description tags as well as meta keywords tags as developers used irrelevant keywords to top their websites in search engines, also this will lead to inaccurate search results ...

Which is the next website a Google bot will go for?

Well, this process is never pre defined, Google never ques up the websites to be crawled, say for example bot visited a URL, on that page it will get 10 links, 10 leads to 100 and so on. So more the websites points the links to a single website, more often is that website visited by the bots and this way it ranks up the page...

From Google Support :


For most sites, Googlebot shouldn't access your site more than once
every few seconds on average. However, due to network delays, it's
possible that the rate will appear to be slightly higher over short
periods. In general, Googlebot should download only one copy of each
page at a time. If you see that Googlebot is downloading a page
multiple times, it's probably because the crawler was stopped and
restarted.

Googlebot was designed to be distributed on several machines to
improve performance and scale as the web grows. Also, to cut down on
bandwidth usage, we run many crawlers on machines located near the
sites they're indexing in the network. Therefore, your logs may show
visits from several machines at google.com, all with the user-agent
Googlebot. Our goal is to crawl as many pages from your site as we can
on each visit without overwhelming your server's bandwidth.


Control Crawl Rate

Vote Up Vote Down

@Kevin317

No. They have their own way of determining what pages get crawled and how often and it is not public information. Pages that are updated often and rank well seem to be crawled more often than static and poor ranking pages. But there is no way to predict when a page will be crawled or how often.

Vote Up Vote Down