: Robots.txt proper way to set it up If you want to allow all search engines to crawl a site and only block one specific folder, is this correct? User-agent: * Disallow: /folder_name/ If you

@Speyer207

Only time you ever need to use allow is for overriding previous rules.

Generally webmasters will never need to use Allow because it is assumed true unless a Disallow is found that overwrites that ruling.

File or Folder within a blocked Folder

User-agent: *
Disallow: /admin/
Disallow: /simon-hayter/
Allow: /simon-hayter/rocks.html


Take a look at the above example, the only time that you will ever need to use allow and disallow together is when you have a situation where you are blocking a file, directory or bot previously, the above situation is when you block a folder but want to allow crawlers access a file within that folder, but keep everything else blocked.

Other Situations

Other situations may be that you want to deny all search engines apart from Google... then your robot.txt might look something like this:

# Block all crawlers
User-agent: *
Disallow: /

# Allow Google
User-agent: Googlebot
Allow: /


WordPress Sites will generally want to use something like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php


The reason why many SEO Guru's recommend that admin-ajax.php is set to allow despite it being located in the administration folder of WordPress is because many WP plugins use AJAX on the front end.

Vote Up Vote Down