: Correct usage of robots.txt file ("Googlebot-Image" and "Mediapartners-Google") This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress search pages (as well as admin pages).

However, I want the Google Images to be able to crawl anywhere, and I also want the AdSense bot to be able to crawl anywhere.

Have I done it right? I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values. Does the position of things in the file matter?

####################################################
# ALLOW MEDIA BOT TO CRAWL ANYWHERE
#####
User-agent: Mediapartners-Google
Allow: /

####################################################
# ALLOW IMAGE BOT TO CRAWL ANYWHERE
#####
User-agent: Googlebot-Image
Allow: /

####################################################
# ALLOW MAIN GOOGLEBOT SPECIFIC ACCESS
#####
User-agent: Googlebot
Disallow: /forums/calendar/
Disallow: /forums/gallery/
Disallow: /forums/members/
Disallow: /forums/user/
Disallow: /forums/index.php
Disallow: /forums/*&search_tags=
Disallow: /search/
Disallow: /*?s=
Disallow: /wp-admin/
Disallow: /wp-content/

10.02% popularity Vote Up Vote Down

: How should I name my DNS server? Are there any conventions or guidelines I should use to name my DNS server? Does it really matter? On the site I'm using to obtain the domain I desire

@Cofer257

Posted in: #Dns #DnsServers #Name #Nameserver

1 Comments

: Google Translate data tracked with gaTrack is not showing up in Google Analytics? I have a google translate widget on my website with gaTrack set to true and the correct gaId, but no translate

@Cofer257

Posted in: #GoogleAnalytics #GoogleTranslate

0 Comments

: Do all websites in a shared hosting have the same IP address? If a shared webhost server like hostgator (for example) host 100 sites, will all the 100 sites have the same IP address?

@Cofer257

Posted in: #SharedHosting

1 Comments

: Robots.txt: block all webpages except a few number of webpages? I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all

@Cofer257

Posted in: #RobotsTxt

2 Comments

Login to post a comment!

2 Comments

Sorted by latest first Latest Oldest Best

@Jamie184

Does the position of things in the file matter?

No. The groups can be in any order.

I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values.

Any one crawler will only look at one group in the robots.txt file. The crawler will only look at the most specific group, determined by the length of the match against the user-agent record.

So, providing you have a Googlebot-Image group then the Googlebot-Image bot will only match this group. All other Googlebots (inc. News) will match the Googlebot group.

Your robots.txt file looks OK to me.

10% popularity Vote Up Vote Down

@Rivera981

Thing looks fine but why do you disallow /forums/gallery/? Is that a media gallery containing images? If it is then avoid disallowing that directory as a whole.

10% popularity Vote Up Vote Down

Feed

: Correct usage of robots.txt file ("Googlebot-Image" and "Mediapartners-Google") This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum

More posts by @Cofer257

: How should I name my DNS server? Are there any conventions or guidelines I should use to name my DNS server? Does it really matter? On the site I'm using to obtain the domain I desire

: Google Translate data tracked with gaTrack is not showing up in Google Analytics? I have a google translate widget on my website with gaTrack set to true and the correct gaId, but no translate

: Do all websites in a shared hosting have the same IP address? If a shared webhost server like hostgator (for example) host 100 sites, will all the 100 sites have the same IP address?

: Robots.txt: block all webpages except a few number of webpages? I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all

Login to post a comment!

2 Comments

Back to top | Use Dark Theme