: Google Webmaster Tools: Robots disallow does not seem to be working for staging site I've added robots.txt on my staging site at staging.mydomain.com as: User-agent: * Disallow: / I then added

@Fox8124981

Looks like what done is perfectively fine.

A typical robots.txt for a production site might be as simple as:

User-agent: *
Disallow:


This is the least restrictive. It says that all crawlers are allowed to crawl the entire site.

For our dev or staging site, we want to use the following:

User-agent: *
Disallow: /


This requests that the entire site not be crawled.

But some time if you didn't take proper precautions prior to creating your dev or staging sites, there's a good chance that the search engines found your work-in-progress.

What now? Well, let's be careful here.

1.First, understand that search engines will cache your site for a certain length of time.

2.Second, you'll need to keep in mind that restricting crawling of your site does not mean that existing indexed pages will disappear from search engine results.

If you find your staging site pages in search results, it's a good idea to go ahead and tell search engines not to index each page. The best way to is to add a "noindex" meta tag to all your pages. The noindex tag looks like this:

<meta name="robots" content="noindex" />


OR :

Advised Approach:

1.Add Authentication (HTTP or otherwise) infront of requests.

2.Respond with appropriate response code if not permitted (e.g. 401 Unauthorized).

3.Everything else in the Basic Approach above.

By adding a robots.txt it prevents search engines from accessing and indexing the content. However, that doesn't mean they won't index the URL. If a search engine knows about a given URL, it may add it to the search result index. You'll sometimes see these in the search results. The title tends to be the URL with not description. To prevent this from happening, the search engines need to be told not to show the content or URLs. By adding Authentication infront and not responding with a 200 OK status code it is a strong signal to the engines not to add these URLs to their index. From my experience I haven't ever seen a 401 response code page listed in a search engine index.

Vote Up Vote Down