: Best way to prevent Google from indexing a directory I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've

@Angela700

Supposedly only robots.txt will be fine, but that depends on how the crawler bot is programmed. Almost all 'legal' crawlers (If not all) use robots.txt, and for disabling a directory robots.txt is the best option, as you don't have to add a meta tag to each file, and also disallows txt files, images and everything you forgot/can't add a meta tag and may be linked from elsewhere. Also, if you upload more files in the future to the directory, you don't have to worry about remembering about adding meta tags too.

Definitely is safer to use robots.txt to deny the access to an entire directory.

EDIT: Although Disallow tells robots to disallow a directory from being crawled, maybe Google indexes it, as stated here: developers.google.com/webmasters/control-crawl-index/docs/faq#h17 so using the tag instead of the robots.txt will be better.


However, robots.txt Disallow does not guarantee that a page will not
appear in results: Google may still decide, based on external
information such as incoming links, that it is relevant.

Vote Up Vote Down