: Right way to allow all access to website in robots.txt? user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?

@Welton855

To use robots.txt and the functionality in a nutshell: www.robotstxt.org/robotstxt.html.
As for a sitemap.xml / sitemap.xml.gz you don't need specific a robots.txt.
If i.e. Google is allowed (and so to see you want that, why else Disallow without any) to crawl the site you can give Google the exact path. (create Google account and use Webmaster Tools, it is free and imho usefull. It even has option to create robots.txt for you)

Anyway, still want to use the robots.txt and sitemap path then this could be sample:

User-agent: *
Disallow:
sitemap: name of your website/sitemap.xml


Be aware that robots.txt file is publicly available so don't use it with the intention to hide information.
A .htaccess file is a much more better please to hide/protect "stuff". (if there is no access on/at server level itself)

Vote Up Vote Down

@YK1175434

If you want to allow access to all content of your website, just don't create a robots.txt, or create a blank one (to avoid 404 errors).

Vote Up Vote Down

@Tiffany637

Yes. The wild card * after user agent means it applies to all crawlers and as there is nothing defined after the Disallow, the entire site is free to be crawled.

Vote Up Vote Down