: Hiding php includes from search spiders? Quick and simple question. I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content
Quick and simple question.
I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are in a separate folder on the server and contain the core content for the individual product pages.
I just wanted to ask, if I use robots.txt or .htaccess to prevent crawling of the directory that holds the php content files, will there be no issue crawling the html pages which include these files?
What I want to achieve is have the html files indexed with the php content included in them, but I don't want visitors landing on the php content pages, nor have these php files indexed as duplicate content. Just clarification needed as to whether it is safe to block spiders from accessing the php folder, without this affecting the html files being indexed with the included content.
Is this the best way to do things? Or should I just leave the content php files to be crawled?
More posts by @Jessie594
3 Comments
Sorted by latest first Latest Oldest Best
I'd also add that PHP is processed server side so if the files are access directly and the source is viewed its blank. When you include them into HTML files you should be using your .htaccess file to render PHP in .html files. Then when the source of the .html is viewed it'll include the PHP and not look any different. You should be using <?php include("file.php"); ?> or some variation. Not SSI's
Move your include files outside of the web directory. The web server never needs to serve up those files.
For example, if you have your website in /somedir/www and your includes in /somedir/www/include, move them to /somedir/include.
You really shouldn't leave the PHP files to be crawled. It is a security hole and those files have no special reason to be crawled.
Block spiders with robots and disable directory list in that includes folder.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.