: Do you really need robots.txt and sitemap.xml if I want bot to follow every link? I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website

@Twilah146

The only thing that you need for Google to follow every link is to let Google know that your site exists. This can happen "naturally" if your site is linked from sites that already exist in Google index, alternatively you can use their site submission form to explicitly add your site to the list of sites to index here.

robots.txt and sitemap.xml are not needed, but sitemap.xml will help guide the bots through your site.

Vote Up Vote Down

@BetL925

Of course not. These two files are not required all the time.

In Google Webmaster Tools, Google says that a robots.txt is not necessary if you want that all webpages being crawled. If you want to put only these following lines in a robots.txt:

User-agent: *
Allow: /


Just don't create a robots.txt.

Moreover, a sitemap.xml helps indexing of your website but it's not obliged to use it. Googlebot is smart enough to follow links on the web and when it finds a link to one of your webpage, it follows it and crawls your webpage to index it. For example, if you have many backlinks, Googlebot will crawl often your webpages and you don't need to create a sitemap.xml.

Vote Up Vote Down

@Pierce454

No, neither is required.

By default your site will be indexed by Google. The robots.txt file is useful for preventing it from accessing specific directories or files. It's not a security mechanism, however, and if you don't want the public to access those pages/files, you should block access another way, such as an .htaccess file.

The sitemap.xml is also not required, but it does help Google to find all of your content. If all of your pages are cross-linked, that won't matter. Sitemaps also contain page ranking that you can suggest to Google, effectively letting the search engine know what you think is the most important content on your site.

Vote Up Vote Down

@Holmes151

I think in order for your page to get index by search engines like Google, Bing etc. you need to have both robots.txt and sitemap.xml files.

A quick Google shows that your robots.txt file has to be placed on root directory for your website. This link should give you more details about how-to create that file and what to do with it.

And for sitemap.xml, there are many websites that will let your generate it for free. This Google webmaster help should give you more details about how Google understands it. And how to create it.

I am using WordPress so I know that there are already plugins available that does it for you. Not sure about other blogging platforms, but they too should have it.

Vote Up Vote Down