: Should we allow HEAD requests? Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD

Posted in: #HttpHeaders #SearchEngines #WebCrawlers #Webserver

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from Squid proxy servers and bots from AddThis, Google and Bing.

My understanding is that most search engines will make GET requests when crawling sites so is there a genuine reason for allowing HEAD requests?

10.01% popularity Vote Up Vote Down

: A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

@Megan663

0 Comments

: In addition to Google's First Click Free, should you whitelist search engine bots past a paywall? Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's

@Megan663

Posted in: #Google #Googlebot #Seo #Subscription

1 Comments

: GoDaddy calls the feature that you are looking for "domain forwarding". Here is the GoDaddy help document about domain forwarding. It states: In the Domain Manager, you can add subdomains

@Megan663

0 Comments

: Why is RPM for mobile is much less than RPM for desktop in Google Adsense? I have about 50% of my website's traffic on mobile devices (iOS and Android.) The RPM for mobile is much worse

@Megan663

Posted in: #GoogleAdsense #Monetize

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Karen161

Some search engines and bots send HEAD request to pages before sending the GET request for reasons like:

Checking if the page size has changed
Checking the last modified date
etc. (Any other info the head would give them!)

This would help large crawlers save a lot of bandwidth if they know a page has not been changed meanwhile and they don't have to crawl it.

10% popularity Vote Up Vote Down

Feed

: Should we allow HEAD requests? Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD

More posts by @Megan663

: A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

: In addition to Google's First Click Free, should you whitelist search engine bots past a paywall? Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's

: GoDaddy calls the feature that you are looking for "domain forwarding". Here is the GoDaddy help document about domain forwarding. It states: In the Domain Manager, you can add subdomains

: Why is RPM for mobile is much less than RPM for desktop in Google Adsense? I have about 50% of my website's traffic on mobile devices (iOS and Android.) The RPM for mobile is much worse

Login to post a comment!

1 Comments

Back to top | Use Dark Theme