: Average header response time of a website I'm building a link checker to check a huge amount of URLs for uptime and I'm currently thinking about timeout times. Do you know any statistics which

@Miguel251

Analyze HTTPArchive.org's dump of web performance data for thousands of sites to get the average response time.

HTTPArchive.org tracks web performance information such as size of pages, failed requests, and technologies utilized.

While a lot of interesting aggregated statistics drawn from frequently fetched data is presented through charts, response time is not specifically shown. So you may have to dig into the data yourself.

Vote Up Vote Down

@Connie744

30 seconds timeout is my recommendation. And I recommend you not download the entire web page html and instead should be getting just the HEAD (php example) or just the http response (php example) and then bailing. Otherwise you could be downloading megabytes of long pages which increase your time.

At the most I would recommend a max 60 seconds timeout if you trying servers across the world but it should be sufficient to use a 30 second try with a retry every 60 seconds. Like trying for 30 seconds up to a max of 3 times at 60 second intervals to not hammer the server.

There really isn't any stats I can pull for you. I would use a similar value of most web browsers or retriever type commands (like curl, wget). Example:

php max-execution-time: 30 seconds www.php.net/manual/en/info.configuration.php#ini.max-execution-time
Mozilla's old setting was 30 seconds kb.mozillazine.org/Network.http.connect.timeout
Internet explorer is something like an hour, see this stackoverflow question stackoverflow.com/questions/1342310/where-can-i-find-the-default-timeout-settings-for-all-browsers
CURL uses the system's default socket timeout curl.haxx.se/mail/lib-2003-05/0097.html stackoverflow.com/questions/10308915/php-default-curl-timeout-value
Wget doesn't have a default except for a 900 second read timeout www.gnu.org/software/wget/manual/html_node/Download-Options.html

Vote Up Vote Down