: What are common reasons to disallow crawling on pages with unique and rank-able content? I'm trying to figure out the right strategy to differentiate between content I want to appear in search

@Megan663

There are several reasons to keep content out of search engines:


Sensitive content -- Some content needs to be public but may have information in it that you don't want in the search engines. An example could be your resume. You want it available for potential employers to look at, but don't want your address easily locatable in the search engines.
Duplicate content -- Content that is copied from another source is unlikely to get you visitors from search engines. Usually Google would just want to index the original copy. In the worst case, it could get get your entire website penalized. See: What is duplicate content and how can I avoid being penalized for it on my site?
Search Results -- Google does not want to index search results pages. If they did so, users would click from the Google search results to some other search results. That would be a bad user experience. See this article by Google's Matt Cutts: Search results in search results
Auto generated content -- Content that is not written, edited, cultivated, or collected without editorial supervision from a human is usually not appropriate for search engines. Having Google index auto generated content is against the Google webmaster guidelines.
Low quality content -- Any content that isn't as good as the content available from your competitors is likely to do more harm than good. Users that find your site through a search engine will use the back button and click on something else. Few things kill your rankings faster than users who turn back from your site.
Thin content -- Pages with little or no actual information on them are not good to show to search engines. Any page that says something like "Be the first to review this" is never going to rank well. Many such pages like this could hurt your entire website.


Google's Panda Algorithm is used to identify many of these types of content and penalize entire webites that have it.

Vote Up Vote Down

@Bethany197

Their page about stackoverflow could be a good target for seo


That's debatable. On the page you link to I just see some numbers and whole bunch of links. Would I really be searching Google to "analyse" someone's site, or go straight to the dedicated tool?

But I think the main "problem" with allowing Google to crawl this particular page is that it is the result of an arbitrary lookup. There are an infinite number of possibilities and potentially an infinite number of pages to crawl.
example.com/links?site=<anything>

Vote Up Vote Down