: Best way to take down site for a long period while I develop it Rather foolishly, I have been developing my site on a live server (with VCS don't worry), kind of using it as a portfolio

@Odierno851

I'd do two things:


Verify your site in Google Webmaster Tools and request removal of your full site. Remember to undo this when you're ready :-). Keep in mind that this will only last 90 days, so you may have to re-request a site removal at that time.
Return 403 (use HTTP authentication) for all URLs on your development site, including the robots.txt. Returning 403 for the robots.txt will prevent the site from being crawled, so you don't need to block it in the robots.txt file (use the normal robots.txt file that you'd use when you make the site public).


Reasons for using HTTP authentication over robots.txt disallows include blocking the site to all other visitors, and making it harder to accidentally publish a robots.txt file with a full disallow :). The latter has happened too many times, even to really large sites.

Vote Up Vote Down

@Hamm4606531

Well, first of all as general advice, I suggest to put all the files in a subfolder that only you'll know. For example, if you have the domain example.com put all your file in example.com/private. If Google starts to crawl your website and starts to index them you'll pay a little penality later because of the fact that some pages will no longer exist (usually the test pages).

Besides, it is a good pratice, as you said, to use the robots.txt to exclude same (or all) pages from being indexed. Anyway, this tip is more useful now than if you used the private folder as I suggest. Keep in mind that good robots.txt is a good way to exclude pages but not the best. Infact, while "good" crawler as Google, Yahoo and so on will respect your will, the "evil" ones will use this information to grab some of your (maybe private) data.

Vote Up Vote Down