: Does htpasswd authentication prevent Google from crawling my site? I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as

Posted in: #Authentication #Googlebot #WebCrawlers

I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as there will be a large amount of garbage content during testing.

I currently have authentication set up using htpasswd, will this prevent site crawlers or robots from crawling my site?

10.01% popularity Vote Up Vote Down

: The /l.php URI is Facebook's exit/redirect page (Google strips the query string Facebook uses to redirect from links in messages, statuses, comments, etc) and the /ajax/emu/end.php referrer is

@XinRu657

0 Comments

: Is it ok to use 301's this way? I installed a forum, which has a page for member info. I want this to link to my main sites member info page which is a lot more detailed and interactive.

@XinRu657

Posted in: #301Redirect

1 Comments

: I also found that the availability of Tomcat hosting was limited last time I checked. I'm using Amazon AWS now as it's affordable and easy to scale up and out if needed. They're offing a

@XinRu657

0 Comments

: This Community Wiki answer has been added to address multiple ways in which one can locate links to one's site. Referrer Logs Check your web server's logs for referring page URL's. (Most analytics

@XinRu657

0 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Barnes591

Yes, if the site requires authentication then robots will not be able to crawl it.

You can also specify a robots.txt file so that the entire site is disallowed

10% popularity Vote Up Vote Down

Feed

: Does htpasswd authentication prevent Google from crawling my site? I have a site that I want to continue testing before I make public, so I don't want Google to start crawling the site as

More posts by @XinRu657

: The /l.php URI is Facebook's exit/redirect page (Google strips the query string Facebook uses to redirect from links in messages, statuses, comments, etc) and the /ajax/emu/end.php referrer is

: Is it ok to use 301's this way? I installed a forum, which has a page for member info. I want this to link to my main sites member info page which is a lot more detailed and interactive.

: I also found that the availability of Tomcat hosting was limited last time I checked. I'm using Amazon AWS now as it's affordable and easy to scale up and out if needed. They're offing a

: This Community Wiki answer has been added to address multiple ways in which one can locate links to one's site. Referrer Logs Check your web server's logs for referring page URL's. (Most analytics

Login to post a comment!

1 Comments

Back to top | Use Dark Theme