: Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect? I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a

@Odierno851

Making the sitemap file dynamic is fine -- it's a good way to auto-update your sitemaps.

Making the robots.txt file dynamic (for the same host! Doing this for separate hosts is essentially just a normal robots.txt file for each of them.) would likely cause problems: it's not crawled every time a URL is crawled from the site, so it can happen that the "wrong" version is cached. For example, if you make your robots.txt file block crawling during business hours, it's possible that it's cached then, and followed for a day -- meaning nothing gets crawled (or alternately, cached when crawling is allowed). Google crawls the robots.txt file about once a day for most sites, for example.

Vote Up Vote Down

@Jamie184

Yes, the same way any request can be "dynamic".

However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably already doing.)

For example, in your root .htaccess file:

RewriteEngine On
RewriteRule ^robots.txt$ robots.php [L]


RewriteEngine should only occur once (although it doesn't really matter if it occurs multiple times).

You just have to make sure that it doesn't conflict with any other directives in your .htaccess file. So, this should probably be near the start of the file, certainly before your front controller.

Vote Up Vote Down

@BetL925

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.

RewriteRule ^robots.txt$ /robots.php [L]


That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce redirects.

Note that even if you power it with PHP, your robots.txt should appear to be static to each crawler for each domain. It is fine to serve different content for different domains, or even for different user agents. However, serving different content randomly, or based on time of day can really confuse search engine crawlers and mess up your SEO.



Sitemaps are fine to name however you want. You could redirect those, or use a rewrite rule to power them dynamically at the same URL. You can also name them like


site-a-sitemap.xml
site-b-sitemap.xml
site-c-sitemap.xml


Then refer to them in robots.txt:

Sitemap: www.example.com/example-sitemap.xml

or submit them to the search engines manually through their webmaster tools or search console.

Vote Up Vote Down