: How to make "crawl-delay" works for all "user-agent" in robots.txt or in .htaccess? I have this line: Crawl-delay: 50 in the robots.txt file at the root of my website. But it seems that

@Tiffany637

Posted in: #Htaccess #RobotsTxt #WebCrawlers

I have this line:


Crawl-delay: 50


in the robots.txt file at the root of my website.

But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.

I don't want to disable the user agent but to make it to respect the delay value (50 in my situation).

How to make "crawl-delay" works for all "user-agent" in robots.txt?
I also read somewhere that this can be done in .htaccess, is this correct?

Vote Up Vote Down

Login to post a comment!