: How should a robots.txt be for a wordpress site in order to be mobile-friendly ? I run a site built on wordpress and my question kind of sums up what I am looking for. There has been an
I run a site built on wordpress and my question kind of sums up what I am looking for. There has been an update on google according to which "mobile-friendliness" of a site is now considered as a ranking factor. Since my site traffic has taken a significant dive I took a mobile-friendly-test. Although my site resizes pretty well in mobile devices, the test result still says my site is NOT mobile-friendly.
On a closer look, the result also says - This page uses 20 resources which are blocked by robots.txt. And on further reading of this page I found that blocked resources are a part of the Mobile-SEO. It seems like googlebot needs access to css and js files of a site in order to determine the mobile-friendliness of a site.
On the other hand I read long back that for a wordpress site "wp-content" folder should be blocked. So currently I have my robots.txt as given below. I am wondering on what would be the correct contents in a robots.txt if it is a wp-site, so my site could be read by the mobile-friendly tool as a complying site. Also is it okay to allow access to entire wp-content folder ? This is what my robots.txt looks like.
User-agent: *
Disallow: /cgi-bin/
Disallow: /jwp/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /xmlrpc.php
Disallow: /*?s
Disallow: /search/
Allow: /wp-content/themes/mysite/style.css
Sitemap: mywebsite.com/sitemap_index.xml
More posts by @Gail5422790
2 Comments
Sorted by latest first Latest Oldest Best
The robots.txt doesn't define mobile friendliness, it is supposed to define what robots are not allowed to access certain resources. In your case, you're preventing every robot from accessing a set of folders except for one CSS file.
When google checks a page, it tries to access all assets in that page. Go on that page in a browser and choose to view the source code and look for all tags that begin with anything similar to:
<link href
<script src
Then you can see all the URL's to the resources your page uses to make it completely loaded. Then if you need to leave your robots.txt file intact, then specifically add allow lines in your robots.txt file to include the resources you find in the HTML.
To make a site mobile friendly, you need to send users with some user agents to one page, and other users to another page, or modify your page so it becomes responsive so it works on all devices. If you choose the first option, you can get source code for your favorite language that you can put in at the beginning before the rest of the site is executed. go to:
detectmobilebrowsers.com/
I would pare that down to:
Disallow: /search/
Sitemap: example.com/sitemap_index.xml
You should certainly remove wp-content from that list. That has your themes, plugins, and images. For mobile friendly you want all these crawled.
If you want more info on what each of the other pieces do, here is a guide to the WordPress directory structure.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.