: Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines? I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically

@Ogunnowo487

If you think creating an XML sitemap is too complicated, you can also create a sitemap in a plain text file. This format doesn’t offer all additional features, but it’s sufficient if you only want to list all URLs to be indexed.

See www.sitemaps.org/protocol.html#otherformats
UTF-8. One absolute URL per line. Nothing else.

So it might look like:
example.com/ http://example.com/videos example.com/videos/video-id1 http://example.com/videos/video-id2 example.com/videos/video-id3

Vote Up Vote Down

@Pierce454

you should use both:


crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag tells crawlers what you consider most important. Google accepts sitempas following the XML-Sitemap-Protokol 0.9 which is explained in detail at sitemaps.org crawling discouragement using a robots.txt – Providing a robots.txt you can try to tell a spider which directories and files you do not want to have crawled. This may first sound a bit strange, but it can indeed make sense to try to exclude pages like "sign up" or "subscribe to newsletter" from crawling. Here's some more information about excluding directories and files at tools.seobook.com/robots-txt/


Both techniques can help because they can be respected by search engines – but they don't have to follow your instructions listed in sitemap.xml and robots.txt.. So don't rely on either – but still do use both.

One addition: I would also recommend generating the sitemap.xml using php whenever a new file is uploaded – or whenever you actually trigger it. This gives you a) 100% control of what is listed, in what order and what tags are used and b) I consider a static file superior to one that the crawler has to render itself every time it visits.

PS: you'll probably want to use a robots.txt anyways – even if you don't want to exclude anything from crawling – since it can be so useful in many different situations…

Vote Up Vote Down

@Smith883

A robots file tells search engines and others where to go and, more so, what to avoid. Sitemaps tell search engines the structure of your site and how to find, possibly, hard to discover pages and paths. In your case, it sounds like you need to set up a sitemap.

Vote Up Vote Down

@Shelton105

I thought using a sitemap.xml would be a good choice to get those
files indexed in search engines because using PHP I can edit the
sitemap dynamically after each page is created. But this is quite hard
to do (since it's an XML document).


It's not as difficult as you think. Using PHP, you can create the XML in a dynamic fashion and then output it with the following header:

header("Content-type: text/xml");


i.e. Select your dynamic pages from your database, create an XML structure and output them with the header above. Then, provide Google Webmasters with a link to your PHP script.

No need to edit any XML files with PHP.

Vote Up Vote Down