: SEO question regarding huge number of URLs I'm wondering the best way to handle this situation. I'm about to release a new section on a website where there are literally hundreds of millions
I'm wondering the best way to handle this situation. I'm about to release a new section on a website where there are literally hundreds of millions of URL combinations that can possibly be crawled and indexed by search engines. This is because I'm giving the users the option to choose their refinements using checkboxes, similar to Zappos, and there are tons of possible combinations.
The website currently has about 100k URLs indexed by Google but releasing this new section of the website will increase that number by a very large magnitude. This could be incorrect, but I've heard websites may be penalized for having a huge sudden increase in crawled pages. Is this true? If so what's the best way to handle this? My goal isn't to "pagerank sculpt" or anything like that, but I don't want to be penalized for any changes I'm about to make.
I thought about adding a "disallow" directive to my robots.txt file, but that will block out the indexing of all of the new pages which I don't want to happen. I think it's best to have the most important ones indexed (one or two filter combinations), but I don't think it's really necessary to have the extremely long-tail combinations indexed. So I guess my question is: is there any good way of doing this or is it best to just let the spiders to their thing?
More posts by @Hamm4606531
2 Comments
Sorted by latest first Latest Oldest Best
I see your point about the URLs being shareable in response to John's answer, and that's understandable, as a user benefit. But here you're talking about the search engines, so: Is there maybe a case for applying noindex to these granular comparison pages, or past some threshold like three selected features?
While single-product pages, and maybe some higher-level categories or whatever are obviously important to have indexed, having every single permutation of features indexed would seem like it could just dilute the site's effectiveness in results and maybe even end up dropping users on them rather than just the product they're looking for.
If the same content can be reached by more then one URL then you will definitely have issues with duplicate content. You will need to specify a canonical URL for each page so Google knows which one is the "main" one and will show that URL in its search results. All other URLs that pull up the same content will be seen as the same page (which means any links to them count as a link to the main page).
There's no real problem with adding that much content at once. Just don't expect to see all of it indexed quickly. Also don't expect it to rank well right away.
Note: If this is customized content that requires a user to login then search engines will not see it as they do not create accounts nor login. So they do not have access to these URLs.
Also, if this is custom content based on user preferences, why not store the preferences in a session and avoid having mangled URLs?
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.