: If C-Panel Indexing Manager sets a folder to "No Indexing" can it be crawled by a webcrawler? People are able to view directories / folders on my site right now. So, they could go to mysite.com/images

@Alves908

No, the previous answer is incorrect.

The "No Indexing" that you see in cPanel controls whether or not the web server shows a directory of all your files when there is no page. It is completely separate from the "no-index" used in robots.txt which stop search engines from indexing page.

e.g. If all your images are in /images/ folder, then allowing indexing means that people, and search engines, can view a page at /images/ that lists all of your images - whether they're used on your site or not. It also means they can see every file in every other folder on your server too - including .php files, etc.

Selecting "no indexing" will prevent that directory list from being shown to users, but will not effect whether the contents of the directory can be crawled by search engines. Search engines will still be able to index any pages and images you have on the site ... but they won't know about pages that are hidden or images that are on the server but aren't used on any pages.

Vote Up Vote Down

@Turnbaugh106

It will only prevent crawlers who respect the no index or robots.txt. All of the major search engines like Google, Bing do respect this, but spambots for instance likely don't.

Vote Up Vote Down