: Cascading matches in robots.txt I was wondering if it's possible to let bots match two groups of User-Agent: blocks. I have this block of robots.txt User-Agent: AhrefsBot # https://ahrefs.com/robot/

I was wondering if it's possible to let bots match two groups of User-Agent: blocks.

I have this block of robots.txt

User-Agent: AhrefsBot # ahrefs.com/robot/ Crawl-Delay: 5

User-agent: *
Disallow: /error.php
Disallow: /cron.php
... and many more rules ...

Now, I wonder if AhrefsBot will now respect the crawl-delay and also respect the rules in user-agent:*?
If not, then I'd have to duplicate everything under user-agent:* which would be quite unhandy.

10.01% popularity Vote Up Vote Down

: Google Analytics: How can 'Avg time on page' stat be larger than my 'Avg visit duration' stat? I've been reading some about the difference between Google Analytics "Avg Visit Duration" statistic

@Cugini213

Posted in: #GoogleAnalytics

0 Comments

: Can Google see CSS transitions? I heard that Google "rewards" you for styling the links differenty from other text that are not ought to be clicked. It makes sense I guess. But what if my

@Cugini213

Posted in: #Css #Google #Seo

1 Comments

: Do search engines directly penalize bad grammar? Let's say I have a web page with user-contributed content, which has good content but with bad grammar, slang terms, and an inappropriate tone.

@Cugini213

Posted in: #Google #GoogleSearch #Seo #Spellcheck #Writing

3 Comments

: How can you identify duplicate CSS rules? I'm not talking about used/unused here. I have two stylesheets and some rules differ (either in selectors, rules or both) and some are exactly the same.

@Cugini213

Posted in: #Css #Optimization

4 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Martha676

Once an User-Agent is mentioned generally it only reads the rules mentioned below it. Most possibly AhrefsBot will not respect rules in user-agent:* and you need to duplicate the rules. Here is example from twitter and facebook robots.txt to support my opinion. They have duplicated rules for all bot specifically mentioned. twitter.com/robots.txt https://www.facebook.com/robots.txt

10% popularity Vote Up Vote Down

Feed

: Cascading matches in robots.txt I was wondering if it's possible to let bots match two groups of User-Agent: blocks. I have this block of robots.txt User-Agent: AhrefsBot # https://ahrefs.com/robot/

More posts by @Cugini213

: Google Analytics: How can 'Avg time on page' stat be larger than my 'Avg visit duration' stat? I've been reading some about the difference between Google Analytics "Avg Visit Duration" statistic

: Can Google see CSS transitions? I heard that Google "rewards" you for styling the links differenty from other text that are not ought to be clicked. It makes sense I guess. But what if my

: Do search engines directly penalize bad grammar? Let's say I have a web page with user-contributed content, which has good content but with bad grammar, slang terms, and an inappropriate tone.

: How can you identify duplicate CSS rules? I'm not talking about used/unused here. I have two stylesheets and some rules differ (either in selectors, rules or both) and some are exactly the same.

Login to post a comment!

1 Comments

Back to top | Use Dark Theme