: I do not recommend a redirect but a rewrite. Google will follow a certain number of redirects (up to 5 per RFC specifications), but you cannot expect this of all bots. The Google Developer's

@Becky754

I do not recommend a redirect but a rewrite. Google will follow a certain number of redirects (up to 5 per RFC specifications), but you cannot expect this of all bots.

The Google Developer's site provides details on Robots.txt Specifications. And specifically about how they process the results.

A Redirect would send the bot to a different URL. Using a RewriteRule you can serve
www.domain.com/robots.txt directly with a HTTP 200 response code.

In your case, you could use something like.

RewriteCond %{REQUEST_URI} ^/robots.txt$
RewriteRule .* /robots.txt/ [L]


If you do use this, you will need to assure that robots.txt does not actually exist, your Apache will use it instead. You may want to replace /robots.txt/ with something like index.php?robots or something unambiguous.

Vote Up Vote Down

Login to post a comment!