: Is Yahoo Search the same as Bing Search now? We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr
We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which was a factor in us blocking them.
[Yahoo's Spider has] been flooding us with 70-200 requests/second since about 5:30 AM EST. I've never seen anything like this, even from their rogue "experimental" crawlers that have abused us in the past. Robots.txt ban isn't enough: I'm having to block them by IP at the load-balancer level. Since they've abused Stack Overflow in the past, I figured you might want to know.
It looks like Yahoo's web search is "Powered by Bing!" now. This is good, because we've never had a problem with Bing's spider (aka MSNBOT).
So, is Yahoo's search the same as Bing search now, or should we be looking at potentially unblocking them if they've somehow, against all odds, stopped sucking so hard?
More posts by @Voss4911412
4 Comments
Sorted by latest first Latest Oldest Best
FYI, On Monday, August 23rd (or within a couple of days of it) Yahoo will officially start serving BING search results.
I raised a search enquiry with Yahoo on Monday - the acknowledgement included the following note on the transition to Bing which you might find relevant:
Yahoo! Search is excited to reach a new stage in the Yahoo! and
Microsoft Search Alliance; the integration of Microsoft algorithmic
results is near complete for the U.S. and Canada marketplace. Yahoo!
Search is currently testing the new integration, and you may notice a
different experience during your searches. Within this test, some basic
search listings will use Microsoft Web, Image, and Video results. All
the enhancements you have come to expect from Yahoo! Search are
preserved, and we will continue to innovate new improvements to the
Search experience.
As this is only a testing stage, not
every search will produce different
results, and the changes may not be
obvious. Results may differ for the
same keyword from one search to the
next. Since the ranking and indexing
algorithms are different, search
results may appear in a different
order. Search results that did not
appear before may be included, and you
may not see search results that were
included before. We appreciate your
patience as we thoroughly test the new
results.
Thank you again for contacting Yahoo!
Search.
I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back.
From the document Joel linked to there is another document here. In it Yahoo states "In mid-July, we began testing the delivery of organic results from Bing to Yahoo!" To me that means that Yahoo is not tightly integrating with the Bing Engine. They are simply calling the Bing Engine API. That also would make the most sense from the standpoint of work required since the Yahoo Engine is going away in the long term.
That means we should see no more Yahoo Bot crawling the web once 100% of the results are coming out of the Bing Engine. Therefore, there is no reason to unblock it.
It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.