: Robots.txt for multiple domains with same website I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder

@Harper822

Since you're allowing search engines to only one domain, your RewriteRules can be made simpler. Just use this:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^example3.com$
RewriteRule ^robots.txt$ /robots-allow.txt [L]


In the public_html folder you need to create two files. robots.txt and robots-allow.txt

In robots.txt you need to add the following:

User-agent: searchengine
Disallow: /


replace searchengine with google or whatever engine you want to block. Repeat the above lines if you want to add more search engines. Then your robots.txt will turn out to be something like this:

User-agent: searchengine
Disallow: /
User-agent: searchengine2
Disallow: /
User-agent: searchengine3
Disallow: /


Then in robots-allow.txt you could leave it as a blank file or adjust the crawl delay for all search engines like this:

User-agent: *
Crawl-delay: 1


The number after crawl-delay represents the minimum waiting time in seconds between requests to the server from the same user agent.

Vote Up Vote Down

@Courtney195

I work on a similar website where I manage this with PHP.

In my .htaccess I have a line reading:

RewriteRule ^robots.txt$ robots.php


In robots.php I have the following:

<?php
header('Content-Type: text/plain; charset=utf-8');

if($_SERVER['HTTP_HOST']=='www.allowed-site.fr'){
echo 'User-agent: *
Disallow:
Host: www.allowed-site.fr/ Sitemap: www.allowed-site.fr/sitemap-fr.xml ';
} else if($_SERVER['HTTP_HOST']=='www.allowed-site.lu'){
echo 'User-agent: *
Disallow: /example/
Host: www.allowed-site.lu/ Sitemap: www.allowed-site.lu/sitemap-lu.xml ';
} else {
echo 'User-agent: *
Disallow: /
';
}
?>


This allows you to have a single file where you can tailor your robots.txt for each individual domain.

Vote Up Vote Down

@Alves908

You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(example1.com|example2.com)$
RewriteRule ^robots.txt$ /robots-disallow.txt [L]


This is specific to your example, where it will serve a "disallow" robots.txt (robots-disallow.txt) to #1 and #2 , otherwise the request will fall through and serve your regular robots.txt file for #3 which "allows".

Vote Up Vote Down