: Wildcard matches in robots.txt that allow crawling of all JS and CSS are not working To allow the robots to crawl all CSS and JS files we have used the following code: Allow: /*.css$ Allow:

@Heady270

Robots.txt uses the longest rule that matches. So when your rules are

Disallow: /directory
Allow: /*.js


Then /directory/file.js would be blocked. Both rules match, and the longer one (the disallow rule) takes precedence. See this article which explains it.

One work around might be to use lots of stars in your wild card rules to make them longer than your other rules:

Allow: /********************.js


Here are some screenshots from Google's robots.txt testing tool showing the technique working:

Vote Up Vote Down

@Turnbaugh106

Regex is not supported in robots.txt and this was asked awhile back on Pro Webmasters. However the code that you have provided is valid which will match certain paths and files. According to Google's bot it defines:

Using /*.js$ matches:


/jquery.js
/scripts/jquery.js


Using /*.js$ would NOT match:


/jquery.js?ver3.0
/jquery.js/
/jquery.JS.MIN
/jquery.JS


If you are using versioning or any other format that is not supported then you can be less specific by using simply /*.js without the $ on the end.

Since your code looks correct the issue you have is likely something else or a rule that is conflicting with your current ones you have. You should be as specific as possible and look at any rules that could be conflicting with them.

See:


Order of precedence for group-member records

At a group-member level, in particular for allow and disallow
directives, the most specific rule based on the length of the [path]
entry will trump the less specific (shorter) rule. The order of
precedence for rules with wildcards is undefined.

Vote Up Vote Down