: Significant decrease in crawl rate Below is the graph for my website which has seen a significant decrease in the crawl rate(pages crawled per day) Here is the graph for the time to download

Posted in: #CrawlRate #Googlebot #GoogleSearchConsole #WebTraffic

Below is the graph for my website which has seen a significant decrease in the crawl rate(pages crawled per day)

Here is the graph for the time to download the page

I wanted to know the reason for this continuous decrease in crawl rate since September.
There were around 30 server errors(HTTP Code 500) which were fixed by September end. 404 errors were around 350 and were all valid 404s. The website has around 0.2 million pages out which we selectively submit around 20K URLs via sitemap.
I checked that server connectivity time was 100% since August.
I checked for deployed changes to the website but none is pointing to this issue.
I did do a URL restructuring in August start for which I ensured proper 301s and canonicals.(No crawl errors were reported for URL restructuring)

Any advise in which direction should I look is more than welcome.
I am also experiencing a fall in the organic traffic.

10.01% popularity Vote Up Vote Down

: Webmaster tools not showing clicks but Google Analytics shows traffic from search I have Google Analytics set up on a site and it is tracking traffic correctly. I set up Webmaster tools and

@Kristi941

Posted in: #GoogleSearchConsole

1 Comments

: First time message/popup in site (by cookie), Googlebot and SEO I have a site (based on dynamic content by JS) and I want to trigger a first time popup to users in order to make them register

@Kristi941

Posted in: #Cookie #Googlebot #Seo

2 Comments

: Googlebot requesting for same page within short period I was analysing analyzing week over week pages that GoogleBot is requesting for crawling. I can see that GoogleBot is requesting for same

@Kristi941

Posted in: #Googlebot #Seo #WebCrawlers

0 Comments

: Fetch as Google does not result in child pages getting indexed I work on an exotic pet website which currently has several types/species of reptiles. It has done well in SERP for the first

@Kristi941

Posted in: #Googlebot #Seo #WebCrawlers

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Si4351233

Google uses an algorithm to decide how frequently to recrawl pages. While the exact mechanics are kept confidential the basics are that it is based on how large the site is, how high quality the site is, the number of other sites that link back to it, and how frequently the content on the pages changes. The figures you have posted seem quite reasonable crawl rates for a site your size.

10% popularity Vote Up Vote Down

Feed

: Significant decrease in crawl rate Below is the graph for my website which has seen a significant decrease in the crawl rate(pages crawled per day) Here is the graph for the time to download

More posts by @Kristi941

: Webmaster tools not showing clicks but Google Analytics shows traffic from search I have Google Analytics set up on a site and it is tracking traffic correctly. I set up Webmaster tools and

: First time message/popup in site (by cookie), Googlebot and SEO I have a site (based on dynamic content by JS) and I want to trigger a first time popup to users in order to make them register

: Googlebot requesting for same page within short period I was analysing analyzing week over week pages that GoogleBot is requesting for crawling. I can see that GoogleBot is requesting for same

: Fetch as Google does not result in child pages getting indexed I work on an exotic pet website which currently has several types/species of reptiles. It has done well in SERP for the first

Login to post a comment!

1 Comments

Back to top | Use Dark Theme