: Not being indexed is nothing difficult if you understand how crawlers work, and by that I don't mean understanding their algorithm, just basic mechanics. Also, there is no need to consider

Not being indexed is nothing difficult if you understand how crawlers work, and by that I don't mean understanding their algorithm, just basic mechanics. Also, there is no need to consider how a specific crawler works at an specific moment since they may change their behaviour at any given time.

They check ips and links, doesn't matter how they get their first resource, but once they got one, they follow links from there and try some basic combinations, which means that if is reachable, it will be indexed.

So, if you want to avoid being indexed, you want to "not be there". How can you achieve that?

No being there. Don't publish
Hide behind a closed door. Like a protected system with user and password.
Use HTTP codes like 307 or 302.

Option 1 is is obvious, so it doesn't require much details.

Option 2 is very good one while you are testing on live environment, showing to a client, doing some maintenance, etc. It requires that you implement some user/password system and so, most people won't do this unless they are using a prepackaged system that includes such mechanism. Still is advisable. You can combine this with method 3 if you use IP restrictions, for instance to allow your self or a client to look at the website but nobody else.

Option 3 is easier to implement than option 2, but may not be the right one on all situations. If you need limited access, remember to craft the rules allowing some IPs. This option is recommended a lot of times using 5xx codes. Don't do that. There is nothing wrong with your server, so don't give wrong information. If the request is made with version 1.1, or newer, of the HTTP protocol, the answer should be 307. If the request is made with version 1.0, the answer should be 302. Both options imply that you should provide an alternate URL, it may be a page explaining the reasons (ideally), or at least a page on your site.

This options don't assume that you had content and moved it, which is the case for a 301 redirect. Assume the situation that you are publishing new content and you don't want it to be indexed.