: Serious Googlebot crawl spike using up a ton of bandwidth One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do

Posted in: #Bandwidth #Google #Googlebot #RobotsTxt #Security

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month.

This seems utterly mental to me, what can I do about it?

I have reduced the crawl rate on webmaster tools, and edited the robots.txt, submitted a sitemap.

A serious spike like this feels like a hack, could Googlebot be spoofed?

10.02% popularity Vote Up Vote Down

: Read this http://codex.wordpress.org/Giving_WordPress_Its_Own_Directory Follow the pre-existing subdirectory instructions.

@Shakeerah822

0 Comments

: Sitename alternate suggestion SEO I had a question about SEO. Say, I've made a restaurant website, which is running for a while now. I added some titles, meta tags, keywords,description,etc,etc

@Shakeerah822

Posted in: #GoogleSearch #Html #Seo

2 Comments

: AdSense block not displaying anything I have copied and pasted the following code into my page, but it seems to be having no effect. It should have a fall back block colour, but nothing is

@Shakeerah822

Posted in: #Ajax #GoogleAdsense #Javascript

3 Comments

: Extend depth of .htaccess to all subfolders and their children I need to be able to use .htaccess in all subfolders for full depth. E.g. I have .htaccess in public_html folder: public_html.htaccess

@Shakeerah822

Posted in: #Apache #Htaccess #Php

2 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Gloria169

It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming.

Check that the I.Ps making these request are owned by Google.

10% popularity Vote Up Vote Down

Feed

: Serious Googlebot crawl spike using up a ton of bandwidth One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do

More posts by @Shakeerah822

: Read this http://codex.wordpress.org/Giving_WordPress_Its_Own_Directory Follow the pre-existing subdirectory instructions.

: Sitename alternate suggestion SEO I had a question about SEO. Say, I've made a restaurant website, which is running for a while now. I added some titles, meta tags, keywords,description,etc,etc

: AdSense block not displaying anything I have copied and pasted the following code into my page, but it seems to be having no effect. It should have a fall back block colour, but nothing is

: Extend depth of .htaccess to all subfolders and their children I need to be able to use .htaccess in all subfolders for full depth. E.g. I have .htaccess in public_html folder: public_html.htaccess

Login to post a comment!

1 Comments

Back to top | Use Dark Theme