: Google crawling/indexing PHP files All of a sudden I had about 819 errors pop up in my Webmaster Tools accounts, all of which are PHP files. Google has never indexed these before, because they
All of a sudden I had about 819 errors pop up in my Webmaster Tools accounts, all of which are PHP files. Google has never indexed these before, because they aren't URLs. The main folder they are indexing happens to be the my /content/ folder which contains basically the relevant information of my site.
My question is: if I block these directories via robots.txt will it affect Google crawling my real pages?
More posts by @Ann8826881
1 Comments
Sorted by latest first Latest Oldest Best
No they won't, unless you have relevant pages under /content, you'll have no problem with Google crawling you current pages.
Add the entry Disallow: /content/ to robots.txt.
Google tries to crawl subdirectories from your URLs, i.e. if Google finds an internal or external link pointing to a page in your site at /content/foo/foo.html, it will try to crawl /content/foo and /content.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.