Mobile app version of vmapp.org
Login or Join
Chiappetta492

: How to find out the referrer of Googlebot's crawling URL? Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer?

@Chiappetta492

Posted in: #Google #Googlebot #Serps #WebCrawlers

Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links?

Is there anything like HTTP Referrer?

In detail: Googlebot crawls


example.com/no_such_files.php
example.com/not-there/no_such_files.php


I want to know from where Googlebot got these links?
I recently see lots of URLs like the above.

10.01% popularity Vote Up Vote Down


Login to follow query

More posts by @Chiappetta492

1 Comments

Sorted by latest first Latest Oldest Best

 

@Holmes151

Googlebot doesn't send an HTTP Referer request header when it crawls your site.

However, the 404 report within Google Search Console should tell you from where these URLs are linked from:


Crawl > Crawl Errors > Desktop | Smartphone
Select the "Not found" sub tab.
The URLs that generate 404s should be listed at the bottom of the page.
Click on any one of these URLs to get a popup of more details of the error.
The "Linked from" tab should contain all the URLs that link to the 404 URL.


(Don't "Mark as fixed", since these are genuine 404s that can't be "fixed".)

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme