: When would Google crawler give up crawling a URL with 5xx error How many times will Google crawler attempt to recrawl a URL that has a 5xx error occurring intermittently? When would it give

@BetL925

When Google determines that your site is having problems, it will slow down the rate at which it crawls. If your site is less responsive, or is giving Google 500 errors, Google will make fewer request per day to your site as a whole.

Googlebot will never give up on any particular URL. Google will come back and try that URL again. How soon it comes back is determined by how important the URL is. It may take weeks for Google to try an unimportant URL again, but it may try again in a couple hours if the URL is well linked from other sites and has a history of being updated frequently.

Vote Up Vote Down

@Shanna517

There is no easy way to measure this as there doesn't appear to be a hard limit and Google is so complex it's difficult to measure.

However, the more intermittent the service the lower an URL will rank as it accrues a performance penalty. This would lead me to guess that it's based on a curve rather than a binary 'indexed' or 'not indexed'.

The only thing you can be certain of is whether an URL is indexed or not.

Vote Up Vote Down