: Do I need a robots.txt file if most of my site's pages require authorization? I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization.

@Welton855

Depending on the website, its content, and how it's linked, disallowing crawling may be something to watch out for. By disallowing crawling, Google won't be able to see that the content requires authentication. This means that it may end up indexing the URLs and showing them in search (based on things like links to those URLs). So if having those URLs visible in search is a problem, then I'd recommend not blocking them in the robots.txt file. On the other hand, if having those URLs visible in search is helpful (eg if you know that your users are searching for your content), then disallowing them via robots.txt might be a possibility (better would be to just have an indexable landing page, and to put the rest behind authentication.)

Vote Up Vote Down

@Kristi941

You don't have to block those pages via robots.txt, but it is still a good idea to do so. You never know if an error on your part, or changes in how crawlers deal with authentication, will result in crawlers being able to successfully find and crawl those pages in the future. And since explicitly blocking those pages is very easy to do, it is never a bad idea to block them via robots.txt (and x-robots-tag, too).

Vote Up Vote Down