: Why does my staging site show up in Google even though my robots.txt file shouldn't allow it? I'm having a lot of trouble with something that should be simple. I don't want my development

Posted in: #Google #GoogleSearchConsole #Noindex #RobotsTxt #Seo

I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says to block this through robots.txt. I set this up correctly months ago, but my development sites still show up. When I log into Google Webmasters and use their robots.txt tester, it shows my site and every URL I've tested should be completely blocked.

User-agent: *
Disallow: /

As an extra measure, I added a simple .htpasswd to the website too, but cached pages still showed up. I went through each and every page and removed it using the Google removal tool - most of them disappeared temporarily but week after week more and more pages pop up as being indexed. I've tried to use Wordpress settings to disallow robots, and I've tried including page-specific code in the header too. This has been going on for quite some time.

Long story short, this is starting to drive me crazy as I feel like I've tried everything I can to stop Google from indexing my site, but they won't. Can anyone think of what I could have done wrong?

10% popularity Vote Up Vote Down