Mobile app version of vmapp.org
Login or Join
Nimeshi995

: Robots.txt proper way to set it up If you want to allow all search engines to crawl a site and only block one specific folder, is this correct? User-agent: * Disallow: /folder_name/ If you

@Nimeshi995

Posted in: #RobotsTxt

If you want to allow all search engines to crawl a site and only block one specific folder, is this correct?

User-agent: *
Disallow: /folder_name/


If you don't want to block any folder then is this the correct way?

User-agent: *
Allow:

10.01% popularity Vote Up Vote Down


Login to follow query

More posts by @Nimeshi995

1 Comments

Sorted by latest first Latest Oldest Best

 

@Speyer207

Only time you ever need to use allow is for overriding previous rules.

Generally webmasters will never need to use Allow because it is assumed true unless a Disallow is found that overwrites that ruling.

File or Folder within a blocked Folder

User-agent: *
Disallow: /admin/
Disallow: /simon-hayter/
Allow: /simon-hayter/rocks.html


Take a look at the above example, the only time that you will ever need to use allow and disallow together is when you have a situation where you are blocking a file, directory or bot previously, the above situation is when you block a folder but want to allow crawlers access a file within that folder, but keep everything else blocked.

Other Situations

Other situations may be that you want to deny all search engines apart from Google... then your robot.txt might look something like this:

# Block all crawlers
User-agent: *
Disallow: /

# Allow Google
User-agent: Googlebot
Allow: /


WordPress Sites will generally want to use something like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php


The reason why many SEO Guru's recommend that admin-ajax.php is set to allow despite it being located in the administration folder of WordPress is because many WP plugins use AJAX on the front end.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme