: Google crawl rate dropped after activating CloudFront Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and
Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user experience we decided to deliver a couple of our sites through CloudFront.
We noticed very nice drops in page load time, but when checking Google webmaster tools we noticed that all CloudFront-activated sites got a huge drop in pages crawled per day (from avg ~3500 to ~150). Also one of the sites have issues with the Google sitemaps (just marked as "Pending" in GWT) and no new pages or updated pages seems to be updated in the Google SERP. The rest of the sites gets some updates on the Google SERP, but very few compared to before CloudFront activation.
Is there anybody here who have experience in full site delivery through CloudFront (or other CDNs) and effects on SEO/Google? Would be very glad for any insights or suggestions. The risk is that we need to remove CloudFront if this just continues.
More posts by @Moriarity557
2 Comments
Sorted by latest first Latest Oldest Best
I find it plausible that the addition of cache headers would reduce the amount of crawling that Googlebot has to do. If you are telling Googlebot that the page can be cached for a week, then it may not come back to crawl again until that cache expires.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.