: Crawl errors and duplicate URL I am getting lots of crawl errors, Google is adding main domain to the URL: sitemapxml: <url><loc>http://www.somedomain.com/whateveryouwant.php</loc><changefreq>monthly</changef

@Connie744

Finally found the error.

The problem was in the canonical link:

I had:

<link href="http:www.somedomain.com/whateveryouwant.php" rel="canonical"/>


It should be

<link href="http://www.somedomain.com/whateveryouwant.php" rel="canonical"/>


The "//" were missing.

The Seo-Spider recommended by @closetnoc did help me to find the problem.

Vote Up Vote Down

@Gretchen104

The link you refer to in your example is not absolute, because it does not contain the www.mysite.com/ part. However, there is nothing wrong in using relative paths in your link (i.e, starting with '/'). This is authorized and ok. It is not an issue.

Browse your page and check the page source for the duplication you are reporting. If you see, then the issue in on your side. If you don't see it, use the Fetch as GoogleBot feature in Google Webmaster Tools. If everything still seems ok on your side, then notify the issue on the Google Webmaster Tools forum.

Vote Up Vote Down

@Jamie184

Your sitemap example looks to be correct. I would make sure that your whole sitemap is correct before taking any further steps.

Otherwise, it has to be something on your website. But there is not enough information here to determine what exactly is going on.

Check your header and footer includes if the pages listed in Google Webmaster Tools appear to be navigational links. I am not sure what lots is. For some it can be thousands, for others it can be a dozen. If the crawl errors exceed your navigational links, then likely the problem exists somewhere else.

But I have an idea!!

If you can hit your live site, I suggest using a site scanner such as screaming frog www.screamingfrog.co.uk/broken-link-checker/ specifically to check for broken links. You can download a free copy from www.screamingfrog.co.uk/seo-spider/ which will let you spider 500 pages which should be more than enough to tell you what is wrong and where.

Vote Up Vote Down

@Heady270

If you are getting crawl errors, then yes it is affecting crawling. I'm guessing how you have your page files and your include files organized on your server is incorrect. Do you have your header.html in a separate directory than the whateveryouwant.php file?

Vote Up Vote Down