: How to make Google index millions of user profile pages? I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I

@Lengel546

you will surely penalized by Google panda, If you are adding profile pages into sitemap, and you also be penalize by Google Penguin if you have no control on outbound links. So Don't forgot to learn about nofollow attributes.

In my real advice, Don't add every profile url into sitemap, simple disallow your /profile directory from robots.txt

Just look out facebook.com/robots.txt there are many things are blocked from spider, and hence Google will not index all the items, also Google will not crawl that page directly from Facebook.

Now suppose user linked the Facebook profile on own site, Then Google will crawl that page from user site, not from Facebook site, because it is blocked by Facebook robots.txt. Now in reality Facebook blocking profile page to index, But some of people mention Facebook profile on web pages, hence Google index that pages on search result. Just look out Facebook Home Pagerank, It is 9 and just look out someone profile pagerank is 0. I hope you got it me.

Vote Up Vote Down

@Eichhorn148

Google is only going to index millions of pages from a site with lots of reputation. I'd estimate that your site would have to have at least PageRank 6 before Google were willing to index that many pages on your site. That is especially true when the pages are only listed in a sitemap (as tillinberlin suggests).

User profile pages are often low quality. On most of my sites, many users tend to sign up without creating any activity. In many cases I have little more for them than an email address and password. A page for such a user would not be useful in search engines. Furthermore, having lots of low quality pages on your site indexed can cause Google's Panda algorithm to identify your entire site as low quality and penalize your entire site.

I would start with trying to get the 10,000 user pages that have the most content indexed first. If that works well, expand it to 20,000 then 40,000, then 80,000. Stop when you are having trouble getting the pages indexed, or when there isn't enough content on the pages.

Vote Up Vote Down

@Gretchen104

I would suggest you use a sitemap – if you don't use one already:


You can list large number of individual pages.
Google (and other search engines) do like to crawl sitemaps.
Pages listed on sitemaps don't have to be 'exposed' on the website – so you don't have to have a page listing all user profiles.
Building sitemaps can be done programmatically – either you write your own script, or you can use one of many free services to write the actual sitemal file (although I think most free services are restricted to some thousands of pages listed).


This page "Manage your sitemap" at support.google.com/webmasters can be a good starting point. However – using sitemaps doesn't guarantee you anything – and getting google to index literally millions of user pages through sitemaps actually sounds close to impossible for various reasons (as also Stephen Ostermiller points out).

Vote Up Vote Down