: Googlebot not respecting HTTP basic auth I have basic auth set up and it has always worked. Suddenly Google started crawling my pages. The auth is still there (I have checked it using different

@Jessie594

Google has a feature in webmaster tools where you can add login information if you want Google to crawl content behind a user login form. If you have provided Google this information in the past and have not changed the login information since then Google will have access to the content accessible to the login information you provided to it. Google does not however try to "guess" login information even if this can be parsed from the URL of the page.

Google can also try to crawl a page that is protected by a user login if the page exists in your sitemap file or has been linked to from another website or a link on your site that Google has access to. In this case Google will try to crawl the page and will detect that it is protected by HTTP basic authentication and so won't list it in the index but it a crawl attempt will still have been attempted.

Vote Up Vote Down