Mobile app version of vmapp.org
Login or Join
Shelton105

: How do you add a rule just for a specific bot to robots.txt? I have a small website, for which the current robots.txt looks like so: User-agent: * Disallow: Sitemap: https://www.myawesomesite.com/sitemap.xml

@Shelton105

Posted in: #RobotsTxt #WebCrawlers

I have a small website, for which the current robots.txt looks like so:

User-agent: *
Disallow:
Sitemap: www.myawesomesite.com/sitemap.xml

Now, I want to add another rule like so (with the aim to disallow SomeStupidBot for crawling my website):

User-agent: SomeStupidBot
Disallow: /
Sitemap: www.myawesomesite.com/sitemap.xml

I am unsure how I combine the above two rules into one robots.txt.

10.02% popularity Vote Up Vote Down


Login to follow query

More posts by @Shelton105

2 Comments

Sorted by latest first Latest Oldest Best

 

@Ogunnowo487

To combine them you simply append the second rule block (group) to the file, with a line gap, like so:

User-agent: *
Disallow:

User-agent: SomeStupidBot
Disallow: /

Sitemap: www.myawesomesite.com/sitemap.xml

The order does not matter (the most specific - based on length - wins). Only one block is processed by the bot.

Note that the Sitemap directive is not part of the group. It is an independent directive that often appears at the end of the file (although the location does not really matter).

As @MEGA has already stated, the first group is superfluous in this instance. However, it is up to you whether you keep it (there will usually be something you don't want other bots to crawl).

10% popularity Vote Up Vote Down


 

@Cody1181609

By default, all bots are allowed to crawl all parts of your site with or without the first code you provided.
So to disallow one bot just add the second code:

User-agent: SomeStupidBot
Disallow: /


and remove:

User-agent: *
Disallow:


I am not an expert in robots.txt, but from my understanding this is how its done.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme