: Can I safely block Google AdsBot from spidering specific links? My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison
My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different.
I'm running a price comparison website. I have ads directing users to my home page, and other ads directing users to a product page.
While browsing my server's access logs, I noticed that Adsbot was loading these landing pages as expected, but also links present in this page, even though these are never used as landing pages.
Like many price comparison sites, when a user clicks on a link to go check a merchant's offer, this takes him to a "Redirecting..." page that contains an analytics tag and and Adwords conversion tracker for my statistics, then redirects the user to the actual merchant page.
The problem is, AdsBot is following these links from my product pages.
I can't understand why it's doing this, as it's only supposed to assess the quality of my landing pages. I can definitely understand why it's loading assets (scripts, stylesheets, images) from the page to measure page speed, but why would it follow the damn links on the page afterwards? This is killing my statistics with "fake" clicks.
My robots.txt contains exclusions for all bots for these pages:
User-agent: *
Disallow: /product/*/go/
As we know, AdsBot ignores the blanket exclusion and must be excluded specifically:
In order to avoid decreasing Quality Scores and Ad Rank and increasing CPCs for advertisers who don't intend to restrict AdWords visits to their pages, the system will ignore blanket exclusions and wildcards (for example, User-agent: *) in robots.txt files.
I could specifically forbid AdsBot from accessing these pages:
User-agent: AdsBot-Google
Disallow: /product/*/go/
But I'm very nervous about taking this step, as this same page gives a warning about blocking AdsBot in general:
Important: Blocking the review of your webpages can have serious effects:
Ads not showing [...],
Ads disapproved [...],
Site suspended [...]
... in addition to the decreased QS and increased CPC mentioned above.
Even though I'm not blocking AdsBot from legitimately visiting my landing pages, could it penalize me for disallowing visits to links on my website that my landing pages contain?
Edit
It came to my mind that I'm also using dynamic search ads. This thread suggests that AdsBot data could be used in place of Googlebot data to index pages for paid search results. Could this be the reason why AdsBot is crawling more than my landing pages?
This would still make me wonder why, in this specific case, AdsBot would not follow the blanket exclusions (why would I want paid traffic on pages I want to be excluded from organic search?) but would make me more confident in disallowing access to AdsBot to parts of my website, if it's not there to assess the quality of my website, but rather to discover content to index for dynamic search.
More posts by @Cofer257
1 Comments
Sorted by latest first Latest Oldest Best
The Google AdsBot does not only check one level deep to assess the page and it is not simply for assessing if the page is a good landing page or not. The AdBot has multiple roles, firstly to check the actual landing page for the ad to see what is on the landing page, this improves the quality of ad delivery to end users by more accurately targeting ads to users based on what Google AdBot assesses the subject matter of the page to be (note this is different to the keywords done when creating the ad campaign), secondly it makes sure that the linked to landing page does not violate the terms of service such as being an adult content page, and much like the adsense bot also checks a further link deep past the page to make sure no links on the page link to inappropriate content. They also use the next level deep to further evaluate the subject matter of the page for ad targeting.
The rule of thumb for the terms of service are that Google AdWords and Google AdSense can not be linked to a page which is within 1 click of a page which violates the terms of service.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.