: Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot

@Shanna517

Using any of the above mention 3 "solutions" (nofollow, noindex or Disallow) will seriously harm your SEO rankings.

You should control your crawl rate via GMTW (as suggested by @Anagio [+1])

If this still doesn't work, you may be dealing with Googlebot impersonator that uses Google's user-agent to scrape or spam your site.
I think this might be the case because I never (in my 8 years as SEO) experienced any problems with Googlebot default crawl rating...

To identify impersonator I suggest using Botopedia.org IP verification feature to check if this visitor's IP is indeed connected to Google.
Here is the link: Googlebot IP check

You can learn more about Googlebot impersonation phenomena here: study shows: 16% of all Googlebot visits are fake

Vote Up Vote Down

@Si4351233

You can set the rate at which Googlebot crawls your website in Google Webmaster Tools. support.google.com/webmasters/bin/answer.py?hl=en&answer=48620
This is how you should limit Googlebots access not by denying it access. Because noindex pages they will still visit causing hits on your server. Robots.txt disallow may work but be careful you correctly edit that file. nofollow is only Good for one specific link. If you have a link to that page in your sitemap they'll still crawl it.


Crawl rate refers to the speed of Googlebot's requests during the
crawl process. It doesn't have any effect on how often we crawl or how
deeply we crawl your URL structure. If you want Google to crawl new or
updated content on your site, use Fetch as Google instead.

Google has sophisticated algorithms that determine how much to crawl
each site. Our goal is to crawl as many pages from your site as we can
on each visit without overwhelming your server's bandwidth.

If Google is crawling your site too often, and slowing down your
server, you can change the crawl rate (the time used by Googlebot to
crawl the site) for sites that are at the root level—for example,
example.com and subdomain.example.com. Changing the crawl
rate can cause some problems (for example, Google will not be able to
crawl at a faster rate than the custom rate you set), so don't do this
unless you are noticing specific problems caused by Googlebot
accessing your servers too often.

You can't change the crawl rate for sites that are not at the root
level—for example, example.com/folder.

Vote Up Vote Down