: Preventing high bandwidth usage from Yandex Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user

Posted in: #RobotsTxt #SearchEngines #UserAgent #WebCrawlers #Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like this: "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"

But when I check statcounter logs I recently saw that other Yandex related bots frequently crawl my site. They are similar to following. I took this info from my cPanel log:

Beeline (128.69.243.12)
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)
Referer: yandex.ru/yandsearch?text=example.com&lr=213
Beeline (89.178.108.247)
Referer: yandex.ru/yandsearch?text=example.com&lr=213 Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)

How can I block or serve cached pages these bots?

When I check for $_SERVER['HTTP_USER_AGENT'] I can't see "yandex.ru" in referrer. Referrer comes empty. Is it possible to find out referrer in cPanel log, but can't take it from HTTP_USER_AGENT ??

And I also I don't want to ban IPs because there exist too many IPs related with this issue and they are changed periodically. So how can I find out this bot?

Does anybody has similar issue?
Thank you

10.01% popularity Vote Up Vote Down

: Google and 301/302 we have to solve a problem with Google not correctly indexing our multilanguage sites. We'll redesign our urls in the future but until then we need some way to help google

@Cooney921

Posted in: #302Redirect #Google #Seo

2 Comments

: Using modproxy to get around China's Great Firewall I'm using WIX service and I like it very much. However, one big problem is that some of my colleagues are in China but their IP is blocked.

@Cooney921

Posted in: #Apache #Proxy

1 Comments

: Unable to print login-required images in IE I have some images in a section of a site that require the user to be logged in in order to view. These images are served by a PHP script, which

@Cooney921

Posted in: #InternetExplorer #Php #Print

1 Comments

: How can I point wildcard domains to a folder in Apache? I am developing an app using PHP and deploying it on Apache on the Amazon AWS environment. This app requires to be made available to

@Cooney921

Posted in: #Apache #Bitly #Cname #Dns #Virtualhost

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Sue5673885

Use a robots.txt crawl delay as described in help.yandex.com/search/?id=1112639
Example:

User-agent: Yandex
Crawl-delay: 2 # specifies a 2 second timeout

Before you start banning this bot, you should first verify that your logs are actually Yandex and not someone else who is spoofing the user agent to look like they are yandex. A tactic used by competitors to hussle you into blocking or delaying a bot so they can out rank you. Perform a DNS lookup : help.yandex.com/search/?id=1112029
You can serve a cache copy dependant on user agent in a number of ways. If you use apache you can do it with mod_rewrite rules. If you use PHP you can do it by sniffing the $_SERVER['HTTP_USER_AGENT'] variable or even use the function get_browser(). How you build a cache is also varied and can be done in 101 ways. Honestly though for the best performance you should always be using caching.

10% popularity Vote Up Vote Down

Feed

: Preventing high bandwidth usage from Yandex Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user

More posts by @Cooney921

: Google and 301/302 we have to solve a problem with Google not correctly indexing our multilanguage sites. We'll redesign our urls in the future but until then we need some way to help google

: Using modproxy to get around China's Great Firewall I'm using WIX service and I like it very much. However, one big problem is that some of my colleagues are in China but their IP is blocked.

: Unable to print login-required images in IE I have some images in a section of a site that require the user to be logged in in order to view. These images are served by a PHP script, which

: How can I point wildcard domains to a folder in Apache? I am developing an app using PHP and deploying it on Apache on the Amazon AWS environment. This app requires to be made available to

Login to post a comment!

1 Comments

Back to top | Use Dark Theme