: Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt? At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI. But until it

@Jamie184

We are seeing requests from both BingBot and YahooSlurp for urls like: /folder/path/data:image/gif;base64,...


If you are actually getting an HTTP request for these resources then you should be able to block them in robots.txt, providing robots.txt is being honoured for these "strange" requests (Bing and Yahoo! should), with something like:

User-agent: *
Disallow /folder/path/data


Or, more generically using a * wildcard (an extension to the original robots.txt "standard" but is reportedly supported by Bing and Yahoo!):

User-agent: *
Disallow /*data:image/


Although, as John suggests in his answer, there really shouldn't be an HTTP request. Nothing is required from the server in order to get the resource (they already have it). I can't actually imagine a server ever responding with a valid response (at least not intentionally)?

Is this perhaps just a bot "glitch"?

Vote Up Vote Down

@Sarah324

It's not supported at all since it is a not a HTTP call. It is embedded in the document and is essentially part of the HTML.

Vote Up Vote Down