It is possible to use the Allow directive after a Disallow directive to allow crawlers to visit URLs that would otherwise be blocked by the Disallow directive. See: Is it possible to block an entire site to Google then list exceptions?

As John says, the Allow directive is not standard, but it is supported by the major search engine spiders like Googlebot.

10% popularity Vote Up Vote Down

@Kevin317

Allow is not officially part of the robots.txt specifications but the major search engines do honor it. But it is redundant as anything not explicitly blocked is allowed to be crawled.

10% popularity Vote Up Vote Down

Feed

: What does "Allow: /" mean in a robots.txt file? Please help me to understand the following code in robots.txt: Allow: /

More posts by @Samaraweera270

: Why would someone try viewing the site with unsupported languages? Our site is receiving page views with strange browser locales. The most recent, on Sunday, included the following; vi_VN vi_VI

: Most significant content keywords in Google Webmaster Tools are not on my website There are multiple content keywords in the content keyword list of the Google Webmaster Tools that are not on

: Online Email client for a just a domain name with no associated hosting setup? I have a domain name that has its CNAME and A record pointed to a Shopify site. I want to set up an email

: How do I best communicate that my website is shutting down? I am at a crossroads. I have received an offer for my domain name that I am likely going to accept. It's something that has not

Login to post a comment!

2 Comments

Back to top | Use Dark Theme