: If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file? If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt

@Rivera981

The lack of a robots.txt file leaves it up to the crawler to decide what it can and can not do. Since it takes only seconds to avoid any kind of ambiguity, why not just make one that allows all agents to access everything?

Vote Up Vote Down

@Lengel546

Depending on your content there should be no issues with not having a robots file as long as you hapy to have every page on your site indexed by search engines.

Vote Up Vote Down

@Welton855

Lack of a robots.txt file will not be harmful. From the robotstxt.org website:


To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)


However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the location of your XML Sitemap. You can do this by adding a line at the top of your robots.txt file that looks something like:


Sitemap: www.example.com/sitemap-host1.xml

You should also be aware that not having it will create a lot of 404 entries in your web logs.

Vote Up Vote Down

@Karen161

Well, since robots.txt contains the address of your sitemap, not having one is potentially harmful.

Vote Up Vote Down

@Voss4911412

I think it would have to be OK, otherwise huge swaths of the web would be un-indexable by web spiders.

No robots.txt is the same as an "allow indexing by everyone" robots.txt almost by definition.

Vote Up Vote Down