: Robots.txt and "Bad" Robots I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to
I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do not want accessed. While I am not looking to get into a debate about that I do have a question:
If I have a structure like this:
/Folder/
/Sub-Folder 1/
/Sub-Folder 2/
(Note: There are no pages within /Folder/ only other folders.)
If I Disallow: /Folder/, it will prevent "good" robots from accessing the directory and any contents within the sub-folders.
While we know that bad robots will see the /Folder/ will they be able to see and access the sub-folders and the pages within the subfolders if they are not listed in the robots.txt?
(Note: I do not fully understand how good or bad robots crawl a site beyond using a robots.txt and the links within the site.)
More posts by @Merenda212
2 Comments
Sorted by latest first Latest Oldest Best
robots.txt only gives instructions, it doesn't enforces any kind of security.
It exist in order to tell search engines what should/shouldn't be listed, nothing more.
Lets say you have and admin.php script located under sub-folder1. A bad bot could "guess" and try to load it. If isn't password protected, you are in problems.
It depends. Pull up that URL in your browser. Do you see those folders listed? If so, they can find them.
To prevent these folders from being listed just add an htaccess file in that directory with this line in it:
Options -Indexes
This will prevent the listing of that directory's contents from appearing.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.