: Edit robots.txt on google sites Getting many 404 errors on my website for the following URLs: /system/app/pages/admin/revisions* I was going to put Disallow in robots.txt, but it seems there is

@Welton855

There's no need to disallow crawling of 404 pages -- having URLs that return 404 is absolutely fine & normal. Having 404 errors on URLs that do not exist will not affect your site's overall crawling, indexing, or ranking (see Do 404's hurt my site?).

Also keep in mind that by disallowing crawling of URLs like this, it can result in them actually being indexed (since we can't be sure of what's behind the URL). On the other hand, if the URL returns a 404, and if we can crawl it to see that, then we won't index that URL.

Vote Up Vote Down