: AdsBot-Google visiting domain which no longer has advertisements During analyzing traffic generated by bots to our website, we found the strange issue. We found that AdsBot-Google crawling domain

During analyzing traffic generated by bots to our website, we found the strange issue. We found that AdsBot-Google crawling domain which we stopped to advertise in the middle of 2016.

Right now there are robots.txt to disallow all bots to crawl our site, but ads bot anyway ignore these settings.

Where can I look for further investigations?
Is it safe to just block this bots with 403 http status?

10.01% popularity Vote Up Vote Down

: Leverage Browser caching - not picking up expires header Currently working on improving my page speed score with Google PageSpeed Insights. My biggest issue according to PageSpeed Insights is Leverage

@Voss4911412

Posted in: #AmazonCloudfront #Google #GooglePagespeed #HttpdConf #PageSpeed

0 Comments

: HTTPS or HTTP based on previous records I hope i'm not the wrong guy asking this in webmasters.stackexchange.com Based on google webmaster records, in the previous month, I found improved ranking

@Voss4911412

Posted in: #GoogleRanking #Ranking #Serps

2 Comments

: Can a Web store with .eu domain sell products intended only for UK market? This has nothing to do with Brexit and I am not even sure this is the right place to ask this question. I recently

@Voss4911412

Posted in: #Domains #Legal #Webshop

1 Comments

: SEO: h1 & h2 - what should go into them? I am still not quite sure today what should I put in H1 and H2 for SEO. For instance, should my site title be H1 and my article title be H2?

@Voss4911412

Posted in: #Html #Seo

2 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Megan663

This crawler is unlikely to be hurting your site in any way. I'd continue to allow it. You would need to do so if you want to resume advertising.

According to www.botreports.com/user-agent/adsbot-google.shtml it does not honor user-agent: * directives in robots.txt. You need to give the more specific:

User-agent: AdsBot-Google
Disallow: /

Another thing to check is that the bot is actually from Google. Google has a IP address lookup method for checking that bots are actually from Google: support.google.com/webmasters/answer/80553?hl=en It could be that some other crawler has appropriated the user agent for nefarious purposes.

If you find you do want to block and robots.txt still isn't working, a 403 error is fine.

10% popularity Vote Up Vote Down

Feed

: AdsBot-Google visiting domain which no longer has advertisements During analyzing traffic generated by bots to our website, we found the strange issue. We found that AdsBot-Google crawling domain

More posts by @Voss4911412

: Leverage Browser caching - not picking up expires header Currently working on improving my page speed score with Google PageSpeed Insights. My biggest issue according to PageSpeed Insights is Leverage

: HTTPS or HTTP based on previous records I hope i'm not the wrong guy asking this in webmasters.stackexchange.com Based on google webmaster records, in the previous month, I found improved ranking

: Can a Web store with .eu domain sell products intended only for UK market? This has nothing to do with Brexit and I am not even sure this is the right place to ask this question. I recently

: SEO: h1 & h2 - what should go into them? I am still not quite sure today what should I put in H1 and H2 for SEO. For instance, should my site title be H1 and my article title be H2?

Login to post a comment!

1 Comments

Back to top | Use Dark Theme