: Robots meta not blocking indexing We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site

@Alves908

As I mentioned in my comment above, if the robots meta tag has been on these pages all the time AND the pages are not blocked by robots.txt (which prevents crawling, but not indexing) then these pages should not get indexed.

Digging a little deeper, these pages do appear to be fully "indexed", with a complete description in the SERPs. So they have not been blocked by robots.txt and this is consistent with the robots.txt file in the question, which does not block /staging1. And the live pages do indeed have a noindex robots meta tag.

However, checking the Google cache of these pages in the SERPs reveals the problem: There is no robots meta tag! So, you would seem to have experienced a temporary "glitch" about a month ago (the Google cache shows dates of 15 and 21 April) that resulted in the robots meta not being output in the page as it should have been. Consequently Google indexed the pages!


I also tried adding an X-Robots-Tag header and submited the site to be re-crawled. Did that a few days ago and I still see no changes.


That's the right idea, but it seems you'll need to wait more than a few days. As I mentioned above, the cached pages in the SERPs are over a month old - so that suggests that these pages have not been recrawled yet, or Google has simply not updated it's index.

Vote Up Vote Down

@Megan663

I've seen this before and I think it's caused by a vicious circle!

If you are blocking pages from being crawled by Google in robots.txt then Google cannot access the page to see the NOINDEX tag, so the pages will not be removed from the index if they had already got indexed

Blocking pages in robots.txt will stop Google crawling them, but it won't stop them getting indexed. If Google finds them linked elsewhere, they can still get indexed.

But where did Google find the links, well that's another topic entirely!

But if you are using NOINDEX tag and blocking them in robots.txt, the pages can still appear in the SERPS, as closetnoc mentioned usually with message saying


'A description for this result is not available because of this site's
robots.txt – learn more.'


The sure fire way to guarantee Google doesn't incliude your URLs in the SERPs is password protecting the directory they are in

Vote Up Vote Down