: How can I set my robots.txt file to let bots not crawl certain folders in root directory? I have a small website with about 20 pages with .html extensions, containing unique articles I wrote
I have a small website with about 20 pages with .html extensions, containing unique articles I wrote consisting of about 1000 words or more. These 20 pages are the main feature of my site.
Root Directory
My root directory is structured in such a way that all 20 pages of my HTML files are in the root directory, not contained in any folder. I also have a forum installed on my website, and a PHP/jQuery poll plugin which is kept in the root directory in their two respective folders and this is where the problem comes in
My Problem - Question 1
I mentioned earlier I only have 20 HTML files which makes up my site. However, when I do a scan of my site to generate a site map more than 70 pages shows up, yet only 20 of those 70+ pages are my own original pages I want crawled and indexed.
The other 50+ files contained in the PHP poll folders and PHP forum folders are all source code files and not unique and do not contain any real content.
Should I or can I block Google from scanning those two folders containing the poll and forum files? If yes, how would I do this in my robots.txt file?
Question 2
If I have these poll and forum folders in my root directory, and a scan of my site shows 70+ files but only 20 pages of those are my own would it have any effect on my SEO?
I have purchased GoDaddy SEO visibility service with my hosting, a crappy but basic product service. When I started my site and only had those 20 HTML files, I got a SEO score of 85+ after installing my forum and poll and re-evaluating my sites SEO score the pages increased to 70+ and my SEO score dropped from 85+ to the mid 50s. I realize these tools are not always accurate but it did prickle my curiosity, leading me to ask this question.
I would also like to add that my forum is rather empty at this stage, but I do enjoy the poll plugin.
Conclusion
So what should I do:
Block robots.txt from scanning those folders if so how?
Should I just remove the forum and poll folder?
Should I not worry about it, and accept products giving you a SEO score is crap?
More posts by @Miguel251
1 Comments
Sorted by latest first Latest Oldest Best
To prevent robots indexing folders or files, create a robots file (called Robots.txt) with the following content and upload it to your root directory
User-agent: *
Disallow: /myFolder1/
Disallow: /myFolder2/
Disallow: /myFolder3/
Or, you can do it by files as well.
User-agent: *
Disallow: /MyFolder/other.html
Disallow: /that.html
Disallow: /MyFOlder/OtherFolder/this.html
Please note, the robots file only works if the robot listens to this file!
In regards to SEO performance, that is very hard to answer. The SEO tools should be used with a pinch of salt - no one knows how to get to position 1, so I'm not sure what the tools are promising they can do. However, use them alongside analytical tools and see how it is affected.
Since your site is new, personally, I wouldn't block anything and see how it does over the next few months. If within 6 months, then consider altering the content.
I've never had to block certain pages (other than error message pages or login pages etc) and I don't think doing so is the answer. Personally, I'd leave it exposed to the engines and let them decide...
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.