: My first robots.txt I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each
I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that included in the search by the bots. I also want to point to my sitemap which gets auto generated by a PHP page. So here is what I have, what else should I include, and if I need to fix anything with it?
Also, it's linking to xmlsitemap.php because that generates the sitemap when called.
My goal is to allow any search bot crawl the forums to grab meta data.
User-agent: *
Disallow: /admin/
Disallow: /ali/
Disallow: /benny/
Disallow: /cgi-bin/
Disallow: /ders/
Disallow: /empire/
Disallow: /komodo_117/
Disallow: /xanxan/
Disallow: /zeroordie/
Disallow: /tmp/
Sitemap: www.vivalanation.com/forums/xmlsitemap.php
Edit, I'm not sure how to handle all the user's folders under /public_html/ since the robots.txt will be going in /public_html.
More posts by @Vandalay111
1 Comments
Sorted by latest first Latest Oldest Best
If you have an FTP area for some users why not place their ftp folders outside of your web root so they cannot be indexed? Robots.txt is not a safe way to secure those folders. What is going to stop someone from linking directly to those files? What about robots which do not follow robots.txt rules? All you're really saying to those robots is please don't look here. And anyone viewing the robots.txt file can see which folders you've tried to restrict access to.
You should re-configure your FTP server to move the FTP folders outside of the web root or public_html folder.
At the least use htpasswd on those folders so users have to login to view the content. That would prevent bots from indexing the content of the folders and also prevent anyone from linking directly to files.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.