: Robots.txt - CSS allow or disallow Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find

@Michele947

The strategy for your robots.txt should always be answered by the question: which sections of my web must not be crawled by a robot and which sections may be crawled by a robot.

Robots implement their own logic and have multiple purposes (not only Google has a crawler...) so if you're assuming that a robot get's somehow "distracted" by your CSS and JS files you open up the robot's black box and assume what the current implementation and the current intent of the robot is. This is no useful long term strategy.

Instead of thinking in the robot's domain try thinking in your web's content domain.

I want to point out that a robots.txt file is no security mechanism.

Vote Up Vote Down

@Megan663

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would.

If you block CSS or JS files, it could harm how well your website performs in the rankings.

More info here: Updating our technical Webmaster Guidelines and here: Webmaster Guidelines

As this is a recent recommendation, many websites and CMS's (such as Joomla) will often have such files blocked in robots.txt. The reasoning behind this was usually that search engines did not need to crawl or index these files, so to stop unnecessary files and directories from getting indexed and to save 'crawl budget', these would often be blocked in robots.txt.

Vote Up Vote Down

@Ann8826881

Templates are used by the blogging or content management software (CMS) application and should not be accessible externally where as CSS is read by the browser and search engines and should be read. Having said that, I would not block either one but I also would not change any that are by the CMS. Search engines and bots do not care about your templates. Just block access to your site by URL/URI that you do not want indexed or read but do not think of robots.txt as a security tool. It is not designed for that.

Vote Up Vote Down