: How to find out the referrer of Googlebot's crawling URL? Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer?

@Holmes151

Googlebot doesn't send an HTTP Referer request header when it crawls your site.

However, the 404 report within Google Search Console should tell you from where these URLs are linked from:


Crawl > Crawl Errors > Desktop | Smartphone
Select the "Not found" sub tab.
The URLs that generate 404s should be listed at the bottom of the page.
Click on any one of these URLs to get a popup of more details of the error.
The "Linked from" tab should contain all the URLs that link to the 404 URL.


(Don't "Mark as fixed", since these are genuine 404s that can't be "fixed".)

Vote Up Vote Down