: What to do about crawler errors after removing non-essential parts of my website? So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google

@Jamie184

If the page is gone, then let it 404 error. If you can, then I suggest doing a 410 gone, but if there are quite a few, then it may not be worth the effort. NEVER 301 redirect a page that no longer exists unless you are redirecting to another relevant page. This should be done where ever possible for the sake of the user.

Here is what happens:

Using Google for an example, if a page request results in a 404 error, Google will keep trying for a number of times over a period of time before deciding the page is gone- essentially an equivalent to a 410 gone. Unless there is a link for that page, Google will not request it again. However, if there is a link, then Google will continue to follow that link as long as it exists and continue to request the missing page. This is not a problem. Just let Google discover again that your page is gone. This is where forcing a 410 gone error can really help. I will cut down on the number of requests anyone makes for the missing page.

What is important to realize is that not all crawl errors as Google lists them should be fixed. This is where it gets confusing to people. They think there is a problem when the process is doing exactly what it was designed to do. 404 errors are not a problem unless you get one for a page that should exist. Otherwise, you can ignore them.

Vote Up Vote Down

@Cugini213

You can perform a 301 redirect for the 404 pages to their relevant equivalent on the new site, or just to the home page. You should be more concerned with human visitors. The web crawlers will eventually stop crawling the old pages.

Vote Up Vote Down