: What URLs should I put into XML Sitemap? We will launch a project with over 1.000.000 products and those are in approx 150 categories. Later on you will find reviews and technical detail pages

@Nimeshi995

With a site the size of yours and especially since it seems that there are many ways to get to each product page, it makes perfect sense to make a sitemap. Create a sitemap that wraps other sitemaps with no more than 45,000 pages each wrapped sitemap. I realize the limit is 50,000. Just make sure there is headroom for error. Create an entry for each product page. Do not worry about categories. Also include home page, about, contact, privacy policy, and so forth. Include all content pages but not navigational (category) pages.
support.google.com/webmasters/answer/183668?hl=en and support.google.com/webmasters/answer/71453
Submit the sitemap using the Google Webmaster Tools. Google will read and index your site extremely fast depending upon the size of the sitemap and the speed of your site. For example, I saw 38,000+ fetches a day from Google. You may see more. The fetches will slow down as it reaches the end of your sitemap.

Google will periodically visit all of your pages and any changes will be noticed. Google uses a TTL (time to live) mechanism for each page. This TTL tells Google how often to visit the page, If the page changes often, ie. rating changes and comments, then the page will be fetched more often and indexed more quickly. It takes Google time to figure this out so be patient. No more than a month or two.

This same TTL applies to the sitemap but I warn you only to create a new sitemap when there are updates and not to try and fool Google. Google will check your sitemap file as often as it seems to be updated. Google can/will also do a HEAD request to see if your sitemap file has changed. If you create new sitemaps, also recreate the wrapper sitemap file.

Use your favorite scripting code to create the sitemap files. I created my update mechanism in about 30 minutes. You can ask the neighbors kid for help. ;-) Or solicit a developer for help. It should not cost much more than an hour of time or an X-Box game or two depending.

Vote Up Vote Down

@Shelton105

The sitemap is really more for helping Google find urls that are not easily found or possibly dynamic. Google won't scan the entire sitemap if it is too large and deep.
I don't think you should worry too much about the sitemap initially. Make sure you have a webmasters account and see what is being indexed.

Also, the sitemap should be created automatically within your system, each time a page is added the sitemap should be updated. Google will check the sitemap relative to your traffic and the sitemap freq tag. It knows if the site is very popular to keep up on the index, it wants to give the best experience to searchers, and that means updated info.
Good Luck.

Vote Up Vote Down

@Caterina187

You want to get to build an XML sitemap for all your pages, it will help to search engine to crawl as easy with crawl boundaries like how many pages are allowed / disallowed for crawling, so you just need to create xml, html and .gz sitemaps for all your pages

Vote Up Vote Down

@Si4351233

You want to get to build an XML sitemap for all your pages, it will help to search engine to crawl as easy, but one thing search engine crawler will take time to index all of your pages and it might not crawl some pages. so you need to create xml site maps for all your pages if there are content is unique.You should update your sitemap for every weeks or every 3 three days

Vote Up Vote Down