: How can I prevent Googlebot from indexing web service URL's? I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated

Posted in: #Bing #Googlebot #Iis #WebCrawlers

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal code or city through drupal and search query is outputted through aspx. The problem is that google/bing keeps indexing the aspx returned query pages and then we receive 404 errors.

How can I prevent googlebot from indexing the web service. We are on IIS 7.5. Do we setup a rule on the server? I have tried blocking it in robots.txt but am unsure if I have the correct syntax.

Disallow: /*?
User-agent: Googlebot
Noindex: /office/default.aspx
Noindex: /?keyword
Noindex: /electronics/locationdetails.aspx?
Noindex: /electronics/communitydetails.aspx?
Noindex: /electronics/Communitylist.aspx?
Noindex: /office/communitysearch.aspx?
Noindex: /Electronics/Communitylist.aspx?
Noindex: /Office/LocationDetails.aspx?

10.02% popularity Vote Up Vote Down

: "Product" for used items on a classifieds site? I have a classified ads website at http://www.gastonia.com. We are redoing the architecture to be HTML5 compliant as well as heavily use Microdata

@Lee4591628

Posted in: #SchemaOrg

3 Comments

: Pagination for product listing, what to use? "canonical" or "rel-prev-next" or do nothing? I want to make sure my product listing is 10 products per page which are not in a series (link). They

@Lee4591628

Posted in: #CanonicalUrl #Google #Pagination #Rel #Seo

3 Comments

: Hosting Dynamic Content on GitHub? Is it possible to host dynamic content through GitHub:Pages? More specifically, we are using GitHub Enterprise, so we have a little more control, but I'm not

@Lee4591628

Posted in: #Dynamic #Github #ServerSideScripting #WebHosting

1 Comments

: 301 redirects.... www. > http ... the entire sitemap I submitted a duplicate sitemap some time back to incorporate the www. addresses when I saw my site was not directing there when I typed

@Lee4591628

Posted in: #301Redirect #Htaccess #Redirects

1 Comments

Login to post a comment!

2 Comments

Sorted by latest first Latest Oldest Best

@Angela700

Use "NoINDEX, NOFOLLOW" Meta Robots tag in each pages under the head tag or certain directives under robots.txt will do the job for you.

10% popularity Vote Up Vote Down

@XinRu657

Your robots.txt file does not appear to conform to the robots.txt specification (you must specify the user agent before the directives and I do not believe that Noindex is a valid directive).

Consider the following updates:

User-agent: googlebot
Disallow: /Office/LocationDetails.aspx
Disallow: /office/default.aspx
Disallow: /Electronics/Communitylist.aspx
Disallow: /electronics/locationdetails.aspx
Disallow: /electronics/Communitydetails.aspx
Disallow: /electronics/communitydetails.aspx

... alternatively, you could configure your webserver (or application) to add noindex/none X-Robots-Tag headers where appropriate as described in detail at Google's Robots meta tag and X-Robots-Tag HTTP header specifications document.

10% popularity Vote Up Vote Down

Feed

: How can I prevent Googlebot from indexing web service URL's? I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated

More posts by @Lee4591628

: "Product" for used items on a classifieds site? I have a classified ads website at http://www.gastonia.com. We are redoing the architecture to be HTML5 compliant as well as heavily use Microdata

: Pagination for product listing, what to use? "canonical" or "rel-prev-next" or do nothing? I want to make sure my product listing is 10 products per page which are not in a series (link). They

: Hosting Dynamic Content on GitHub? Is it possible to host dynamic content through GitHub:Pages? More specifically, we are using GitHub Enterprise, so we have a little more control, but I'm not

: 301 redirects.... www. > http ... the entire sitemap I submitted a duplicate sitemap some time back to incorporate the www. addresses when I saw my site was not directing there when I typed

Login to post a comment!

2 Comments

Back to top | Use Dark Theme