: Understanding Google Crawl Stats I launched my website in Jan '11 and it is indexed in Google which is great - around 300 pages of content so far. I'm trying to understand what happened the

@BetL925

I guess Google changes crawl rate with site age, popularity (links to your site), markup and header, proper site maps, etc.
They also changed their crawler a while ago so content now can appear in search results much faster than it used to (at least 2 weeks before the change).

So when I released my blog 2 years ago it took Google months to index all the content and weeks to index new posts. Now I see the any post in search results the same day I publish it.

That simple, google doesn't like new sites but respects the elders.

Vote Up Vote Down

@Si4351233

If you feel that they are over-crawling your site (perhaps even missing deeper content) you should ensure that your HTTP headers are returning good values for things like 'lastmodified' time etc. It may be that Googlebot overestimates how much your site is changing. As a bonus your site will behave better with regards to caching (be it proxy or browser based) and thus feel a little faster.

You'd do well to study what URLs are being crawled (by reviewing your server logs). If they are recrawling the same URL again and again, you definitely have a problem. A common variant on this is if you have a page that can be displayed in many different ways using request variables. Googlbot may try to crawl every possible combination of those variables.

An example I've encountered as a crawl operator was a page that had a list of twenty headings, any combination of which could be expanded. Basically, that page had 2^20 different URLs!

Make sure that Googlebot isn't stuck crawling basically the same page over and over again with trivially different parameters (I've seen it get caught up in this)

Vote Up Vote Down