: Bing bot following user on password protected pages? I'm seeing some traffic on our classic ASP website from a verified Bing bot that seems to be following a current user of mine. For instance,

@Ann8826881

Maybe that user has a toolbar from Bing installed. This toolbar might submit visited URLs to Bing, and the Bingbot might (try to) crawl these URLs then.

From the searchengineland.com article How The Bing Toolbar May Accidentally Submit Private Pages & Ads:


Microsoft has confirmed that they do discover and index URLs that they find through users surfing the Internet with the Bing Toolbar installed.

[…]

Duane Forrester, a Senior Product Manager at Microsoft, told us:


Yes, as with some other toolbars, the Bing toolbar (when permitted by the user) may record the open (not https) web sites that a user visited to add to our knowledge of the internet in order to improve the search results we provide to users. This in turn (if the site permits) may indeed lead to our crawling links we have discovered. […]



(The article is from 2012. I don’t know if things have changed since then.)

Vote Up Vote Down

@Kristi941

Do you happen to send some of these urls by mail? In that case a nosniff header could help to prevent mail clients from sniffing the page.

Hotmail would use Bing-bot, Gmail would use the Google crawler and so on.

Here is a nosniff snippet for htccess:

# Add headers to all responses.
<IfModule mod_headers.c>
# Disable content sniffing, since it's an attack vector.
Header always set X-Content-Type-Options nosniff
</IfModule>

Vote Up Vote Down

@Angela700

I actually experienced this at one point on my site.

What you can do to solve the problem is to make the links the search engine robots are not supposed to access as POST based links.

Here's how you do it perfectly with good compatibility for just about every browser invented:

Create a form with a method post. For example:

<form action="http://example.com/robots-cant-access-this.asp" method="POST">
<input type="hidden" name="parametertoscript" value="valueforscriptparameter">
<input type="hidden" name="secondparametertoscript" value="valueforsecondscriptparameter">
<input type="hidden" name="nthparametertoscript" value="valuefornthscriptparameter">
<input type="submit" value="button label to show to public">
</form>


Then in your asp code, you need to check that the request method is POST, and if it is not, then don't continue with the cart process. In fact, return a 410 so robots don't continue to try to access urls you don't want them to access.

I'm not fluent enough with ASP, but here's pseudo code to get you started:

Get value of request method
If Request method equals post then
process requested page to client
else
issue HTTP error 410 GONE
print error
end if
exit

Vote Up Vote Down