Mobile app version of vmapp.org

Sue5673885

: Effect of incomplete Disallow rule in robots.txt file Solved: Pages were being blocked by meta robots deliberately A lot of pages are being blocked in the robots.txt file and when I checked

Posted in: #RobotsTxt

Solved: Pages were being blocked by meta robots deliberately

A lot of pages are being blocked in the robots.txt file and when I checked the file there were no indications of a rule that blocks these.

The robots.txt file is structured:

Sitemap: domain.com/sitemap.xml User-agent: *
Disallow: /directory-1/
Disallow: /directory-2/
Disallow: /directory-3/
Disallow: /directory-4/
Disallow: /directory-5/

User-agent: Googlebot-Image
Disallow:

None of the directories in the robots file match the URLs that are being blocked.

I was wondering if the incomplete Disallow could be the issue, despite it being only applicable to Googlebot-Image?

10.01% popularity Vote Up Vote Down

Login to follow query

More posts by @Sue5673885

Sue5673885

: Should my canonical URL contain the language parameter? As I understand correct a canonical URL should point to the original URL of the website. Our website provides different languages that can

Posted in: #CanonicalUrl #GoogleSearch #Seo #Url #UrlParameters

Sue5673885

: Redirecting all old URLs in same domain I'm relaunching a new site on a current domain, currently there are hundreds of URLs. With the relaunch, all of the content will have new URLs and there

Posted in: #Redirects #Url

Sue5673885

: Prevent International Subdomain Ranking Higher than National Website I have a national website and an international website. The international website covers the rest of the world so not country

Posted in: #GoogleRanking #International #Ranking #Seo

Sue5673885

: Block third domain from being indexed We have a web application that handles many websites that have test and production environment. This application has only one folder and it's hosted with

Posted in: #Iis #Index #RobotsTxt #Seo

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

I have just found that an incomplete Disallow allows all robots complete access.

Found the answer here: www.robotstxt.org/robotstxt.html

10% popularity Vote Up Vote Down

Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.

Back to top | Use Dark Theme