: How often are sitemap.xml checked for updates by crawlers? How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?

@Rambettina238

Let's try an empirical approach.

In the access logs for my site, I see 55 sitemap requests over the last 33 days. Out of those 55, 30 are from Googlebot, 21 from msnbot and the remaining four are from Exabot. (I've only submitted the sitemap manually to Google; the others have found it through robots.txt.)

So that's one data point for about "every day", at least for Google and for a smallish site like mine. Although I should note that I regenerate the sitemap daily, so it's possible that Googlebot is simply observing a pattern in the last modification timestamps and adapting to it.

If you want to directly inform search engines that your sitemap has been updated, you can do so by sending an HTTP ping. This may make the search engines reload your sitemap sooner, although of course there are no guarantees.

Vote Up Vote Down

@Odierno851

It's depends on which search engine you refer to. Most people are interested in google so I'll assume that's what you meant:
Generally speaking, the higher your rank is, and if the frequency in which you change your site (add articles etc) is high - your site will be crawled more frequently.

The following article might interest you.

Vote Up Vote Down