: Where should the robots.txt file be? If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ Should the robots.txt file be in: subdir.a.com/robots.txt or a.com/myfolder/robots.txt

@Moriarity557

You can have multiple robots.txt files if you're managing too many subdomains as it helps you manage bots access restriction on specific folders.

If you're managing too many subdomains then it is always better to have robots.txt for each subdomain. You can also write crawling specific instructions in robots.txt of your root level directory. If you don't want to do that then you can just keep the robots.txt in the directory- "a.com/myfolder/" folder. Because you're redirecting permanently to this directory. You're doing permanent redirection to this directory so "robots.txt" should be here.

Vote Up Vote Down

@Kristi941

It goes in the root web directory of the subdomain. In your example physically it will be in your a.com/myfolder/ directory but over HTTP it will be accessible at subdir.a.com/robots.txt.

Vote Up Vote Down