: Unreachable robots.txt error I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue.

But I also use Google Page Speed test, site is working correctly than getting 91/100 score.
how can I fix it?

10.01% popularity Vote Up Vote Down

: Web hosting on another continent? My client from the US wants to have a website for Australian users. I am wondering if the web host really needs to be in Australia for this site. Does it

@Samaraweera270

Posted in: #WebHosting

3 Comments

: Resized background images blurred in desktop Safari The website I'm creating (test6.lazyfoxdesign.co.uk) is responsive and makes heavy use of images for the background. I use high resolution images

@Samaraweera270

Posted in: #Images #Safari

5 Comments

: Is it OK to use translations obtained via crawling a translation service with advertising? Google and Bing not only have their API-based translation, but also their websites which allow for one-off

@Samaraweera270

Posted in: #GoogleTranslate #Legal #TermsOfUse #Translation #WebCrawlers

1 Comments

: Robots signing up on website A couple of days ago, some robots started to sign up on the website (about 30 in a couple of days), they don't do anything on the website (as I'm guessing they

@Samaraweera270

Posted in: #Botattack #Spam #SpamPrevention

3 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Hamaas447

Check if your robots.txt bear any empty Disallow like below:

User-agent: *
Disallow:
Disallow: /cgi-bin/

Sitemap: www.yoursite.com/sitemap.xml

It should be updated to:

User-agent: *
Disallow: /cgi-bin/

Sitemap: www.yoursite.com/sitemap.xml

10% popularity Vote Up Vote Down

Feed

: Unreachable robots.txt error I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page

More posts by @Samaraweera270

: Web hosting on another continent? My client from the US wants to have a website for Australian users. I am wondering if the web host really needs to be in Australia for this site. Does it

: Resized background images blurred in desktop Safari The website I'm creating (test6.lazyfoxdesign.co.uk) is responsive and makes heavy use of images for the background. I use high resolution images

: Is it OK to use translations obtained via crawling a translation service with advertising? Google and Bing not only have their API-based translation, but also their websites which allow for one-off

: Robots signing up on website A couple of days ago, some robots started to sign up on the website (about 30 in a couple of days), they don't do anything on the website (as I'm guessing they

Login to post a comment!

1 Comments

Back to top | Use Dark Theme