: In robots.txt, why would there be "Allow: /*" below a Disallow rule? I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent:

@Eichhorn148

The Allow directive is a non-standard extension to robots.txt. The original robots.txt standard only includes Disallow directives. Google's robots do understand Allow directives, so it would be OK to use them in sections targeted at a Google robot. The Allow directive should only be used when it is more specific than Disallow directive. This will counteract the Disallow. It can be used to give finer grained allowance (maybe just a single page) of entire section that would otherwise be disallowed.

In your case, the Allow directive is saying the same thing as the Disallow directive, so it should be omitted.

You say you that you want to allow all Google robots to crawl all of your site. If that is the case, you would need a rule for the main Googlebot as well. I would suggest using this in your robots.txt

# Google Image
User-agent: Googlebot-Image
Disallow:

# Google AdSense
User-agent: Mediapartners-Google
Disallow:

# Googlebot
User-agent: Googlebot
Disallow:

Vote Up Vote Down