: How do you add a rule just for a specific bot to robots.txt? I have a small website, for which the current robots.txt looks like so: User-agent: * Disallow: Sitemap: https://www.myawesomesite.com/sitemap.xml

@Ogunnowo487

To combine them you simply append the second rule block (group) to the file, with a line gap, like so:

User-agent: *
Disallow:

User-agent: SomeStupidBot
Disallow: /

Sitemap: www.myawesomesite.com/sitemap.xml

The order does not matter (the most specific - based on length - wins). Only one block is processed by the bot.

Note that the Sitemap directive is not part of the group. It is an independent directive that often appears at the end of the file (although the location does not really matter).

As @MEGA has already stated, the first group is superfluous in this instance. However, it is up to you whether you keep it (there will usually be something you don't want other bots to crawl).

Vote Up Vote Down

@Cody1181609

By default, all bots are allowed to crawl all parts of your site with or without the first code you provided.
So to disallow one bot just add the second code:

User-agent: SomeStupidBot
Disallow: /


and remove:

User-agent: *
Disallow:


I am not an expert in robots.txt, but from my understanding this is how its done.

Vote Up Vote Down