: Googlebot and Mediapartners-Google are crawling URLs blocked by robots.txt, why? I have blocked the some of the URLs from robots.txt. I think there is No way to access the blocked URLs from

@Correia994

You say that there is No way to access the locked URL's from your website? I'm guessing you mean that you have no links pointing to those URL's from your own website. So that's what I will assume.

Blocking URL's or part of your website in your robots.txt is actually no guarantee that they will not be crawled. There are circumstances where the disallow directive in the robots.txt-file will be ignored. Usually it's because you have external links (backlinks) pointing to the URL's you wish to hide. If you are trying to prevent the URL to be indexed by Google you should use the X-Robots-Tag HTTP header. If do you that you must remove the disallow directive in your robots.txt-file, otherwise there could be a conflict. You can read about this in Google Search Console help section: support.google.com/webmasters/answer/7424835?hl=en#h12
Google bot is the "normal" bot from Google that will crawl your website and fetch data used for the index. Mediapartners-Google is a bot from Google AdSense or Google Mobile Adsense that will crawl your URL's so they understand what content is on your page(s) and display relevant Ads to your visitors. You can read about this on: www.botreports.com/user-agent/mediapartners-google.shtml

Vote Up Vote Down