: The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix? We have a website: www.example.com which redirects to https://www.example.com.
We have a website: example.com which redirects to www.example.com.
Therefore when you type example.com/robots.txt you are redirected to www.example.com/robots.txt.
Another point to note:
The main mirror in Yandex will be decided by the crawler, it won't allow us to set example.com or set www.example.com as the main mirror.
Please note: This error that I am getting
The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used is from this page : webmaster.yandex.com/robots.xml
Update:
We updated our Robots.txt to the following after the answers.
User-agent: Yandex
Sitemap: www.example.com/sitemap.xml Disallow: /some_pages/
Host: www.example.com
User-agent: *
Sitemap: www.example.com/sitemap.xml Disallow: /some_pages/
More posts by @Sherry384
1 Comments
Sorted by latest first Latest Oldest Best
You can set a main mirror in Yandex.
I think you need something like this:
Host: myhost.ru # uses
User-agent: *
Disallow: /cgi-bin
User-agent: Yandex
Disallow: /cgi-bin
Host: myhost.ru # is not used
or even:
# domen.myhost.com is the main mirror
# domen.myhost.com, is the correct use of
# the host Host directive:User-Agent: *
Disallow:Host: domen.myhost.com
The Host directive should contain:
An HTTPS protocol indication if the mirror is only available via a protected channel (Host: myhost.com).
One concrete domain name that corresponds to RFC 952 and is not an IP-address.
The port number, if necessary (Host: myhost.com:8080).
More information : yandex.com/support/webmaster/controlling-robot/robots-txt.xml
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.