: Google not crawling all pages I have over 200,000 pages but only about 23,000 pages are crawled by Google as is evident when i search for site:mywebsite.com and the result comes as About 22,900

@Barnes591

Posted in: #Googlebot #GoogleSearchConsole #Seo #Sitemap #XmlSitemap

I have over 200,000 pages but only about 23,000 pages are crawled by Google as is evident when i search for site:mywebsite.com and the result comes as About 22,900 results (0.18 seconds)

I have over 100,000 users with profile link such as mywebsite.com/username
My sitemap.xml looks like this:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd >
<url>
<loc>http://mywebsite.com/</loc>
<lastmod>2015-07-20T20:15:54+00:00</lastmod>
<changefreq>daily</changefreq>
</url>
</urlset>


What am I doing wrong? I notice that most of my profile links aren't crawled by Google and hence the issue. How do i make Google crawl all the profiles?

FYI, my robots txt looks like this:

User-agent: *
Disallow:

Vote Up Vote Down

Login to post a comment!