: Bandwidth saturation due to Googlebot search for spam pages I recently had to update the hosting plan of a WordPress site I run due to bandwidth excess, which was already quite high. I've

@Yeniel560

If the URLs they are requesting follow some patterns, I think it is a good idea to block those pages. Based on your comment above, in your robots.txt you could add:

User-agent: *
Disallow: /data/


That will work as long as you don't have any real URLs beginning with data.

The second thing you may want to do is decrease Google's crawl rate. You can do this in Google Webmaster Tools.

There is also a Crawl-delay setting in robots.txt - Google ignores this but it is useful for other search engines. I'd advise not setting it higher than 4.

Vote Up Vote Down

@Sherry384

404 errors are a part of business. They can come from poorly made links on other sites, current link on your own site, or from pages that have been removed without a current existing link.

You do not want to block the search engines from accessing pages that do not exist unless you do not want to be in the search engine at all.

If you are blocking Google, for example, using a robots.txt file, you will be delisted. You are saying to Google, I do not want what you are offering.

You do want to allow 404 errors to occur. How else can Google know these pages are gone? You can replace a 404 error with a 410 which would shorten the process. A 404 means temporarily gone while a 410 means gone. Google will try and access pages that return a 404 error for a period of time for a certain number of accesses before it determines the page is truly gone. If you return a 410 error, Google will stop trying to access the page after one access. But keep in mind that any search engine may try again if they find a link on another site just in case.

Vote Up Vote Down