: Alexa SEO audit reports that all crawlers are blocked despite "Allow: /" for specific crawlers in robots.txt I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers
I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like
User-agent: *
Disallow: /
User-Agent: GoogleBot
Allow: /
User-Agent: Bingbot
Allow: /
User-Agent: Slurp
Allow: /
User-agent: ia_archiver
Allow: /
Sitemap : [Sitemap URL]
I have also checked these search engines and they do show indexed results from my website. Also before I had added permission for Alexa's bot
User-agent: ia_archiver
Allow: /
without which Alexa was not able to do the audit. I am puzzled as to why Alexa is still reporting so, that all crawlers are blocked even when Alexa itself used that robots.txt permission to crawl my website.
I had already used Google's Robots.txt testing tool.
It reports that GoogleBot is not blocked.
Result when I feed a URL to a page in my website to be tested against selection of GoogleBot
I am using Allow operator as a way of having only selective bots be able to crawl my website.
I would also like to highlight the fact that Alexa's own bot was not able to crawl my website for audit till I had added it to website's robots.txt using the same selective allow method.
More posts by @Murray155
1 Comments
Sorted by latest first Latest Oldest Best
The default directive for spiders is allow and when Google visits your robots.txt and sees the disallow command, it immediately knows that it isn't allowed to crawl the website anymore. The subsequent allows don't matter.
Do note that when there are conflicting codes within the robots file or between the robots.txt and the Meta Robots tag of a page, spiders must follow the most restrictive rule.
If you want to block specific bots from crawling your website, you need to setup individual disallow rules for each of those bots.
On the other hand, if you want to allow some bots and disallow all the rest, then you can put the following:
User-agent: Googlebot
Disallow:
User-agent: Slurp
Disallow:
User-agent: *
Disallow: /
You can analyze your current robots.txt file with the built in checker in Google Search Console (former Webmaster Tools)
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.