: Bing webmaster indicating crawl errors on http to https 301 redirect I moved my site form http to https, now sitemap contains https urls, but Bing webmaster indicates increased "crawl errors"

@Becky754

...my question is why is it looking at http pages when I have https pages in sitemap...

This is the key.

The answer is simple. Sitemaps are not the authority by which search engines submits pages to the fetch queue from. Their index is! Your pages are indexed as HTTP and therefore that is exactly what will be submitted to the fetch queue. Until each page is requested, redirected, and the URL updated in the index, then, and only then, will the search engine not request HTTP URLs. The exception is, of course, following any existing links to your pages made with HTTP. The search engine will always attempt to fetch pages based upon link URLs even if the same page exists within it's index as HTTPS and not HTTP. That would be the right and responsible thing to do.

The sitemap has nothing to do with this process. Not speaking for Bing, I suspect they are exactly like Google in this respect, Google will only use the sitemap to audit that they can properly crawl your site. Nothing more. Generally, sitemaps direct entries into the fetch queue only when the site is so huge that links to all pages are not possible or when pages exist behind a login or paywall. And then only for those pages. That is it. As far as Google is concerned, sitemaps are not used to feed the fetch queue when a site can properly be crawled.

Vote Up Vote Down

@Bryan171

It sounds like all you did was submit a new site map and not actually implement a valid 301 redirect.

If you are on an Apache server, add this to your htaccess file in the root folder of your site.

A simple approach would be;
#Force www:

RewriteEngine on
RewriteCond %{HTTPS_HOST} ^example.com [NC]
RewriteRule ^(.*)$ www.example.com/ [L,R=301,NC]

#Force non-www:

RewriteEngine on
RewriteCond %{HTTPS_HOST} ^www.example.com [NC]
RewriteRule ^(.*)$ example.com/ [L,R=301]


I would pick the one that you are currently indexed under the most in searches.


my question is why is it looking at http pages when I have https pages
in sitemap


Because you did not specify which to use in their webmaster tools and/or no redirects to tell it otherwise.

Most likely, you have at least 4 versions out there either partially of fully indexed.
example.com example.com www.example.com www.example.com

with a possibility of
example.com/ example.com/ www.example.com/ www.example.com/

Vote Up Vote Down