: Confused about redirect and robots.txt I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess?

@Nimeshi995

Do not use robots.txt to remove pages unless there is a catastrophic emergency which this is not. The robots.txt file is not designed for this. It is designed to inform robots where they can and cannot go. The robots.txt file is generally not edited too often. In fact, many are not changed for years.

If a page is removed and not replaced, a 410 error is best though not always practical or easy to apply. For that reason letting the page 404 for a period is the standard. All major search engines will drop pages with a 404 error after a period of time. Until this time period expires, the page will remain indexed but may drop from the SERPs or at least drop position within the SERPs before dropping entirely. For that reason, any 404 error should be captured and directed to a 404 page giving the user the opportunity to still be satisfied.

If a page is replaced, then you would use a 301 from the old page to the new page for a period. After you are satisfied that all of the search engines have (in effect) dropped the old page, and there are no valuable links to the page, then you can remove the older page. If you have valuable links to the old page that you do not want to lose, then maintain the 301 redirect. I say in effect because as long as the 301 exists, the old page is not actually dropped from the index, but appears to be. It drops from sight in the SERPs.

Vote Up Vote Down