: Does repetition of same rules for each major crawler bot makes sense in robots.txt? SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by

@Heady270

Posted in: #RobotsTxt #Seo

SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by any advice but still persists.
Now I mean something like this:

User-agent: Googlebot
Disallow: /page/
Disallow: /ajax
Disallow: *?back*
User-agent: Yandex
Disallow: /page/
Disallow: /ajax
Disallow: *?back*
...
User-agent: *
Disallow: /page/
Disallow: /ajax
Disallow: *?back*


Does this makes some sense? Those rules are really totally same.

Vote Up Vote Down

Login to post a comment!