: Google crawler accessing non existing pages I am looking Search Console and found Crawl errors for following URLs on my site: example.com/mobile example.com/m I never mentioned these pages anywhere

@Sent6035632

In mid 2016, we did not have the issue of google bot crawling /mobile/ or /m/ pages that did not exist; however, at the end of 2016, google bot started trying to crawl /mobile/ and /m/, and since they don't exist on our site, Search Console reports a 404 under the SmartPhone tab.

Here is a QA that I found useful, and it links off to other discussions.

From the discussions I've read, the act of google bot attempting to crawl mobile/ and m/ directories suggest that google bot may not think that the site is completely mobile friendly.

Our site is mobile friendly (at least the Mobile-Friendly Test reports it so); however, out of 9,000 URLs, one page is being reported as not being mobile friendly on the Search Console Mobile Friendly report. In Nov. 2016, we introduced the mobile issue with that one page, and Google started trying accessing mobile/ and m/ in Dec., suggesting that the mobile issue with that page has something to do with why they are trying to access those non existing pages.

We will fix that issue soon, and we believe that Google will then stop trying to access mobile/ and m/ pages.

To answer your question, "why Google is accessing it and what should I do?", Google may be trying to access those non existing pages because your site is not completely mobile friendly. To resolve the issue, you could make your site mobile friendly; or, if it is mobile friendly, make sure there are no mobile issues reported within Search Console. After we fix our issue, we will check if Google is still trying to access mobile/ and m/ or not and post a comment with the results.

Vote Up Vote Down

@Heady270

The fact that Googlebot crawls non-existent pages should not be considered a problem. Googlebot tends to be greedy. It checks:


Any link it can find or your site
Any URL in your sitemap
Any link it can find to your site from any other site
Any link it has ever found to your site
JavaScript snippets that it thinks could be links. If you have var foo='some/text' it will probably try to crawl example.com/some/text as a link.
Common places where webmasters put content for mobile such as /m and /mobile


Because Googlebot is programmed to check so many types of URLs, Google knows it will find "404 Not Found" pages. Showing those error pages to Googlbot won't hurt your site in any way. It is the expected site behavior.

Vote Up Vote Down

@Annie201

Google is going to check absolutely everything in relation to the site and try to index it all.

It will scan the home page, discover the links there and scan every one of them and so on and so forth.

It will also check every single URL you have included in all sitemaps that you have submitted to webmaster tools.

What I would suggest (which helped me) is to download a program that scans your entire site like google would but at a better speed. There is a program called screaming frog SEO spider that can check the first 500 URLs for free, and there's a program called Xenu Link sleuth which checks unlimited URLs for free but the caveat with it is that for large sites, you have to scroll the humongous scan list manually to find errors it found.

Here's the link to Xenu Link sleuth:
home.snafu.de/tilman/xenulink.html#Download
I'm not sure if it works for the newest systems, but I have it working on wine in my linux machine

Anyway. Run one of those link scanning softwares and every time it finds an error with a URL in your domain, make changes to your website and scan again and repeat this process until there are zero errors.

If you have submitted a sitemap, modify the sitemap so that the non-existant URLs are removed and resubmit it to google.

Vote Up Vote Down