: To allow crawling of all but a specific folder, do I need to include an empty disallow directive in robots.txt? If I want my website to be crawled, do I need an "empty" disallow? Is there

If I want my website to be crawled, do I need an "empty" disallow?

Is there any difference between

User-agent: *
Disallow:
Disallow: /folder/

and

User-agent: *
Disallow: /folder/

I have seen a robots.txt where the first option is used but I don´t understand why.

10.01% popularity Vote Up Vote Down

: Adding location and category to Name of NAP Is it okay to add location and business category to the Name of the local citation (NAP)? for ex: Drive Digital – Houston Web Design and

@Michele947

Posted in: #LocalSeo #Seo

0 Comments

: How to make Apache/cPanel recognise all/any domains pointed to its wildname CNAME record? Setup: Server Domain: redir.example.com subdomain of above: * CNAME value of *.redir.example.com = redir.example.com

@Michele947

Posted in: #Apache #Cname #Cpanel #Dns

1 Comments

: Cloudflare not redirecting to HTTPS I set it up, and set "Automatic HTTPS Rewrites" to "ON", BUT this still not redirecting me to the https by default when the page is requested via http...

@Michele947

Posted in: #Cloudflare #Https #WebHosting

1 Comments

: Closing ftp-connection in filezilla If I am connected to a server, is it ok to just shut down filezilla? Does filezilla make a disconnect in a proper way?

@Michele947

Posted in: #Ftp

2 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Si4351233

An empty disallow matches nothing and is ignored. The second example is what you're looking for.

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

www.robotstxt.org/robotstxt.html

10% popularity Vote Up Vote Down

Feed

: To allow crawling of all but a specific folder, do I need to include an empty disallow directive in robots.txt? If I want my website to be crawled, do I need an "empty" disallow? Is there

More posts by @Michele947

: Adding location and category to Name of NAP Is it okay to add location and business category to the Name of the local citation (NAP)? for ex: Drive Digital – Houston Web Design and

: How to make Apache/cPanel recognise all/any domains pointed to its wildname CNAME record? Setup: Server Domain: redir.example.com subdomain of above: * CNAME value of *.redir.example.com = redir.example.com

: Cloudflare not redirecting to HTTPS I set it up, and set "Automatic HTTPS Rewrites" to "ON", BUT this still not redirecting me to the https by default when the page is requested via http...

: Closing ftp-connection in filezilla If I am connected to a server, is it ok to just shut down filezilla? Does filezilla make a disconnect in a proper way?

Login to post a comment!

1 Comments

Back to top | Use Dark Theme