: What is the right value in robots.txt for wordpress blog ? I migrated my wordpress site from godaddy to AWS and converted to a php site with wordpress into a sub directory, blog. My main
I migrated my wordpress site from godaddy to AWS and converted to a php site with wordpress into a sub directory, blog.
My main site has few pages. When I look at content keywords after few weeks in google webmaster , to my surprise instead of keywords which my site was ranking, I see these below
If I click on any of these I can see that the crawler is going to these sub directories in wordpress blog and counting image extension .jpg in keywords.
To avoid this situation, I added a robots.txt in blog sub directory to avoid this happening
user-agent: *
Disallow: /blog/wp-admin/
Disallow: /blog/wp-includes/
Disallow: /blog/wp-content/
Disallow: /blog/comments/feed/
It has been more than a month and I still see the content keywords to be same.
I wanted to review my robots.txt and few sites like yoast suggested that the robots.txt should allow /blog/wp-content/
I am kind of confused as to what should go in robots.txt for wordpress. If I allow /blog/wp-content/ , then I end up situation like above.
My question is if this is the right approach and is my robots.txt for wordpress correct ?
More posts by @Alves908
3 Comments
Sorted by latest first Latest Oldest Best
Try below robots.txt
User-agent: *
Disallow: /blog/cgi-bin/
Disallow: /blog/wp-admin/
Disallow: /blog/archives/
Disallow: /blog/*?*
Disallow: /blog/*?replytocom
Disallow: /blog/comments/feed/
Disallow: /blog/*/trackback/*
Disallow: /blog/*/xmlrpc.php
Disallow: /blog/wp-content/uploads/*
sitemap: www.yourdomain.com/sitemap.xml
Most posts about WordPress robots.txt are totally wrong.
You should not block this in robots.txt
I suggest to use only this robots.txt for WordPress blog:
User-agent: *
Disallow: /blog/?s=*
If you block WordPress directories like admin and includes and content, then you're going to block CSS, JavaScript and other assets for Googlebot, and that will result in it not rendering correctly.
By the way, directories like wp-admin will not be crawled because it is password-protected, you can read my answer on other question in regards to this.
Content keywords which are displayed in Google's Search Console indicates that you're using these keywords the most in your website, but it does not have any relationship with keyword ranking or traffic. In general I only checkout that data to see whether my site contains any incorrect keywords because that can happen when your site is hacked and someone inserts other keyword stuff, but if you're using this for ranking analytics, then you're probably using it incorrectly.
From www.wpbeginner.com/wp-tutorials/how-to-optimize-your-wordpress-robots-txt-for-seo/
User-Agent: *
Allow: /?display=wide
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /readme.html
Disallow: /refer/
Sitemap: domain.com/post-sitemap.xml Sitemap: domain.com/page-sitemap.xml Sitemap: domain.com/deals-sitemap.xml Sitemap: domain.com/hosting-sitemap.xml
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.