: Accidentally submitted broken URLs in a sitemap. How can I stop Google crawling them? We recently added a sitemap to our site but didn't exclude unpublished items (which, unless you're logged

@Caterina187

Block the links using robots.txt.

This will work for a large set of URLs which have some kind of common origin URL.


A directory and its contents by following the directory name with a
forward slash:

Disallow: /sample-directory/


From Google Search Console Help.

Vote Up Vote Down

@Heady270

Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article about Google's big memory for 404 pages.

You say that you already removed the URLs from the sitemap. That is good. If you hadn't already done so, that would be the first step.

You don't want to add 40,000 individual URLs to your robots.txt file. That could produce a file that is too big. The max robots.txt size is 500KB for Google. Other crawlers may not process even that much.

Having "404 Not Found" error pages on your site is not a problem. Google's John Mueller says:


404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html


Having that many 404 errors will make that report in Google Search Console much less usable. One way to remove them from it would be to return a more appropriate status. I would suggest: "401 Unauthorized". That indicates that there is content there, but that the user would be required to login to see it. If the user is logged in, but not an admin, then a "403 Forbidden" status would be appropriate.

Vote Up Vote Down