: Google video bot and video sitemap I'm looking to understand a sitemap created a long time ago. Is there a good reason to exclude Google's video crawler in robots.txt when you have a video

@Frith620

Sitemap – list of pages of a web site accessible to crawlers or users
video sitemap - A video sitemap is a text file packed with useful data about the content of your video
robots.txt are signboards on your site giving signals to search engines that they are allowed to enter and crawl your site in order for them to collect and gather data.


However there are some robots.txt setup or configuration that will deny or won't allow search engine robots to crawl on specific pages of your site probably for privacy purposes, but this is configurable

Vote Up Vote Down

@Sarah324

No, don't do that. Look's like you misunderstand about robots.txt and sitemap.

Video sitemap and other sitemap tells about your new content to Googlebot, so they come to your websites and start crawling, but before they crawl, they always check your robots.txt first. So If you disallow them, then they can't crawl anything.

Let me talk more about some of Google crawlers like Googlebot image and video. When Googlebot come to your site, and find out you have embed images then next time, Google image bot crawl that kind of data.

So, in general Googlebot crawl anything, but when they found specific content like images and videos, then next time they crawl with their specific crawler. It makes process fast for them.

Vote Up Vote Down

@Berumen354

Why would you exclude video?

You can use the Google Video extension to the Sitemap protocol to give Google information about video content on your site. You can create a new video Sitemap or add video information to an existing Sitemap.

Vote Up Vote Down