: Impact of blocking site search from googlebot, but listing content pages in sitemap.xml We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search

@BetL925

You should not allow Googlebot to crawl site search pages. In addition to putting undue stress on your server, Google doesn't want to crawl them. Here is Google's Matt Cutts blog post about the issue: Search results in search results
by Matt Cutts on March 10, 2007 Google now actively penalizes sites that allow their site search results to be crawled and appear in Google's SERPs. By allowing googlebot to crawl your search result pages, you are risking all of your Google referral traffic. One favorite trick of a google reviewer is to use your site search for a spam terms such as "Viagra". When they see a crawlable page as the result (even if it says "no results for Viagra found") they will apply a manual penalty against your site as a spam site.

You should put your site search into robots.txt. Just make sure that googlebot can still crawl your content pages.

Having a sitemap that lists all your content files IS NOT ENOUGH to get all your content files indexed. Here is a very related question The Sitemap Paradox in which Jeff Atwood from Stack Overflow notes that pages in the sitemap which can't be crawled, don't get indexed. The question is answered by Google's John Mueller. He states in no uncertain terms:


The Sitemap file isn't meant to "fix" crawlability issues. If your site can't be crawled, fix that first. We don't use Sitemap files for ranking.


I would recommend that every piece of content on your site is available within 3 or 4 clicks from the home page. That is admittedly a tough task to do well, especially because neither users nor googlebot react well to large lists of links in your pages. If you do add links to your pages, try to make them as useful to users as possible and keep lists of links to fewer than 10. You can use dimensions out of your faceted navigation to create useful links on products like


Also by this manufacturer
Similar items by feature
Similar items by price
Users who bought this also bought

Vote Up Vote Down

@Merenda212

Honestly, google should NEVER search your site. Not having a sitemap is the problem. WIth that amount of request you just need it.

But its not necessary to add noindex. Just make sure you specify on google webmaster tools whats the search parameter so google wont hit it.

Vote Up Vote Down