: Does a "Disallow:" rule with nothing following it in robots.txt block my entire domain? If my website's robots.txt has the following code, does that mean my entire site is being blocked from

If my website's robots.txt has the following code, does that mean my entire site is being blocked from indexing, or do I need to append the code with a /?

User-Agent: *
Disallow:

10.01% popularity Vote Up Vote Down

: 404 status code for HTTP/2 How does a correct 404 status code look like on using HTTP/2 protocol? HTTP/1.1 404 Not Found seems to be for the previous protocol version, but I can't find anywhere

@BetL925

Posted in: #Http #HttpHeaders #Https

1 Comments

: 404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment My site lists blogs like this example.com/?status=blog&id=number I only have

@BetL925

Posted in: #302Redirect #Googlebot #Seo #WebCrawlers

1 Comments

: Thousands of links from Russian Sites I've recently checked into my Google Webmasters account and found that I have several thousand links from .ru domains. Almost half of all links from one

@BetL925

Posted in: #Backlinks #Google #GoogleSearchConsole #NegativeSeo #Seo

1 Comments

: What is better: one page or some pages? What do you mean is better in terms of SEO (all aspects of it: internal linking, content, index management): to have one page with unique content or

@BetL925

Posted in: #CanonicalUrl #Content #Links #RelCanonical #Seo

2 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Hamaas447

The rule as you have it up there allows all user agents to access everything. If you wanted to block all user agents from accessing anything, you would add the forward slash:

User-Agent: *
Disallow: /

The forward slash stands for your root directory. This command disallows all user agents from accessing your root directory or anything that it contains - which is everything.

10% popularity Vote Up Vote Down

Feed

: Does a "Disallow:" rule with nothing following it in robots.txt block my entire domain? If my website's robots.txt has the following code, does that mean my entire site is being blocked from

More posts by @BetL925

: 404 status code for HTTP/2 How does a correct 404 status code look like on using HTTP/2 protocol? HTTP/1.1 404 Not Found seems to be for the previous protocol version, but I can't find anywhere

: 404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment My site lists blogs like this example.com/?status=blog&id=number I only have

: Thousands of links from Russian Sites I've recently checked into my Google Webmasters account and found that I have several thousand links from .ru domains. Almost half of all links from one

: What is better: one page or some pages? What do you mean is better in terms of SEO (all aspects of it: internal linking, content, index management): to have one page with unique content or

Login to post a comment!

1 Comments

Back to top | Use Dark Theme