sitemap.xml and when I use the Webmaster Tools to

Posted in: #GoogleSearchConsole #RobotsTxt

I have the following robots.txt file:

User-agent: *
Sitemap: m.mysite.com.au/sitemap.xml

and when I use the Webmaster Tools to validate it I get it showing that the user agent line is incorrect.

However when I validate it, it comes back as "Allowed".

In both cases it is the same file. So my questions are:

What is wrong with the User-Agent line?
I am being asked to remove the User-Agent line. While I understand that * means all agents I don't understand why it has to be removed.
Any ideas why Google is giving inconsistent results?

10.01% popularity Vote Up Vote Down

: Google mobile friendly resources which are blocked by robots.txt I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath

@Kevin317

Posted in: #RobotsTxt #Seo

1 Comments

: File does not exist from wacky websites? Our LAMPs website has been running slow as of late, I suspect its a SQL problem but I want to check all the logs. Can somebody explain what the following

@Kevin317

Posted in: #Apache #ApacheLogFiles

0 Comments

: After change to new software stack, Chinese pages require setting encoding in browser We changed to a new server with a completely upgraded software stack, including apache 2.4 and php 5.6.9.

@Kevin317

Posted in: #Browsers #ContentEncoding #Html

1 Comments

: How to remove Analytics' DoubleClick (for demographic data)? Over a week ago I enabled demographic reports in Google Analytics. A day later, I changed my mind and disabled this feature in my

@Kevin317

Posted in: #GoogleAnalytics

2 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Sherry384

This is because...

User-agent: *
Sitemap: m.mysite.com.au/sitemap.xml

...would be incorrect.

The correct syntax for the robots.txt is:

To disallow all:

User-agent: *
Disallow: /

To allow all:

User-agent: *
Disallow:

To disallow specific directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Examples taken from here: www.robotstxt.org/robotstxt.html
The correct syntax for the site map is:

Sitemap: m.mysite.com.au/sitemap.xml

You cannot restrict sitemaps using the robots.txt file. You should place the sitemap last though I am not sure that is a requirement.

If you wish to allow all and specify a sitemap, it would be:

User-agent: *
Disallow:

Sitemap: m.mysite.com.au/sitemap.xml

10% popularity Vote Up Vote Down

Feed

: Why is Google saying that the User-Agent line is invalid? I have the following robots.txt file: User-agent: * Sitemap: https://m.mysite.com.au/sitemap.xml and when I use the Webmaster Tools to

More posts by @Kevin317

: Google mobile friendly resources which are blocked by robots.txt I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath

: File does not exist from wacky websites? Our LAMPs website has been running slow as of late, I suspect its a SQL problem but I want to check all the logs. Can somebody explain what the following

: After change to new software stack, Chinese pages require setting encoding in browser We changed to a new server with a completely upgraded software stack, including apache 2.4 and php 5.6.9.

: How to remove Analytics' DoubleClick (for demographic data)? Over a week ago I enabled demographic reports in Google Analytics. A day later, I changed my mind and disabled this feature in my

Login to post a comment!

1 Comments

Back to top | Use Dark Theme