You may also find 404's in your error logs if you don't have a robots.txt in place, if you want to avoid those just create a blank file named robots.txt in your sites root.

10% popularity Vote Up Vote Down

@RJPawlick198

If you don't have a robots.txt your website will get indexed. The robots.txt file are just some directions for (nice) crawlers on what not to crawl.
www.robotstxt.org/robotstxt.html
Please also note that there are aggressive crawlers out there that just ignore it.

10% popularity Vote Up Vote Down

Feed

: Do crawlers require a robots.txt file before indexing content? Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want

More posts by @Hamaas447

: Simplest way to create a theme I am about to create my first ever WordPress theme, based on the excellent tips provided in this answer. Initially, all I want to do is to modify the CSS only,

: Looking for a CMS for specific requirements (Javascript, import/export...) I heard about Joomla, Drupal, Wordpress, etc. but I don't know about their detailed features. I am looking for a content

: Is PayPal (Website Payments Standard) appropriate for invoicing? Here's what I want to accomplish: Agent sends email invoice to client containing payment link. Client clicks link, is brought to

: Does putting more ads on a page necessarily generate more revenues? It seems 'logic' that if you put more ads on a page, it 'should' generate more revenues. However, someone could argue that

Login to post a comment!

3 Comments

Back to top | Use Dark Theme