: Why is Google crawling non-existant URLs? I can see in live traffic of my wordpress website that goggle bot crawl non existing pages. www.example.gr/search/search-results/password-reset%252Fpassword-reset/password-reset%252Fpassword-

@BetL925

Googlebot crawls any URL that it finds:


Links on your own and third party websites
Text on the page that looks like a URL
JavaScript strings that look like they might be URLs


Check your own site to see if there are links to these pages. If not, it is probably some other site. Google may be able to tell you which site in Google Search Console in the crawl error report.

One thing that you can do about it is to use robots.txt to disallow crawling of whole directories. Based on your examples, /search would be a great candidate for disallow:

Disallow: /search


It is also possible that it isn't actually Googlebot doing the crawling. It may be a bot spoofing Googlebot to try to find vulnerabilities on your website. You can verify whether or not it is actually Googlebot by checking the IP address using the procedure here: How to identify if IP address is really google's IP

If it isn't actually Googlebot, you can block the IP addresses used in .htaccess: How to block entire IPs of a VPN server by IP

Vote Up Vote Down