: Preventing search engines to crawl on specific folders I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty

@Correia994

You really only need the disallow. Search engine crawlers will automatically assume they are allowed everywhere that isn't disallowed.

User-agent: *
Disallow: /templates_c


But to answer your question, according to Google:


At a group-member level, in particular for allow and disallow
directives, the most specific rule based on the length of the [path]
entry will trump the less specific (shorter) rule. The order of
precedence for rules with wildcards is undefined.


Source

Thus, your current rules should work as intended for properly behaving bots (i.e. allowed everywhere but the templates_c directory).

Note: Disallow: /templates_c will disallow all files and folders in the root that contain that within their name. If you intended to disallow only the specific templates_c directory, add a trailing slash: Disallow: /templates_c/

Vote Up Vote Down

@Pierce454

I'm sorry if this is wrong, I'm not familiar with the Smarty template engine.

Does the template_c folder have to be publicly accessible at all? I imagine that Smarty does use functions like include() to fetch the template code, so you could easily create a .htaccess file in the template_c folder that disallows access to any visitor. It would look like this:

DENY FROM ALL

Vote Up Vote Down