Mobile app version of vmapp.org
Login or Join
Gail5422790

: Google Bot spending way too much following old 301 redirects We changed our URL structure a couple of years ago but we still have a lot of backlinks pointing to old URLs. This causes Google

@Gail5422790

Posted in: #RobotsTxt

We changed our URL structure a couple of years ago but we still have a lot of backlinks pointing to old URLs.

This causes Google bot to keep crawling old paths again and again and spending a lot of crawl quota on old URLs.

If we block old URLs via Robot.txt like this:

Disallow: /olddirectory/

This would mean Google won't crawl them but would they still pass link juice?

Thanks

10.01% popularity Vote Up Vote Down


Login to follow query

More posts by @Gail5422790

1 Comments

Sorted by latest first Latest Oldest Best

 

@Nimeshi995

You would lose the effect of the link/301 redirect if you block it in robots.txt. Google needs to know that the link has something at the end of it- ie. your web page- or it cannot know that the link is valid. Therefore the search engine must be able to follow the link all the way through to your page.

I have mentioned this on several similar answers before. What is often forgotten in these cases where a 301 redirect is used to preserve a link is that using a 301 redirect should be temporary. Certainly you can keep a 301 redirect for as long as you like, however, what typically happens is there comes a day where the idea of the 301 seems less desirable. If it is just a few links then they are no real bother, but when there are a bunch of them, then it is likely that one day the link/301 redirect glimmer will fade. It always happens.

I try and preempt this often where a site moves from an old domain name to a new one. I advise that the new domain must be able to stand on it's own as if the old domain did not exist. The same advice exists here. Before you tire too much of these links, try and replace the old link profile with a new one so that the old links do not matter any more. This is good advice for anyone who uses 301 redirects to preserve a link. There comes a time where you will want to remove the 301 redirect and you want to have already replaced it's value with new links. It becomes so much easier to say good bye to old rubbish that way.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme