: Wordpress Layered Nav pages - Noindex I have been trying to figure out how to Noindex the pages created by Woocommerce widget - Latered Nav. I use YOAST and have used that to noindex other

@Bryan171

I would suggest going to Configuration > robots.txt > settings and check your screaming frog robots configuration and make sure "ignore robots.txt" is unchecked. Also Go to Configuration > HTTP Headers > User Agent, and select "GoogleBot Regular" as the preset user agent. This way you will have a better idea about how Googlebot will crawl your website.



Then, go to your website robots.txt file and make sure you add instructions as to how bots should crawl the pages of your website. For example, if you would like to block the provided URL add the following line:

.
.
.
Disallow: /*?*
Disallow: /whishlistpage
Disallow: /userloginpage


For example, the Disallow:/*?* will prevent Bots, including Googlebot, to crawl any page or URL that contains the “?” character. However the pages could still get indexed if you are linking any of those pages from another page. The most effective way to get those pages not to be indexed is by implementing the meta robots tag. Using the Yoast SEO plugin make sure you define those pages to be noindex.

You need to set the meta robots tag for all the filtered result pages and the areas of the website you wish to block, like user login page

<meta name=“robots” content=“noindex”>


Wait until Google crawl again your webpages with the meta robots tag and then, and only then, edit robots.txt file. This way they will know they do not need to crawl the pages again in the future.

Note: I would also like to point out that as long as the pages have the noindex (and sometimes nofollow) meta tag your pages won’t get indexed. Making your pages crawlable or not by screaming frog does not determine if the pages will be indexed, screaming frog is just reporting back to you and it has nothing to do with Google, Google will only take in this case just the noindex meta robots tag and will not index your pages accordantly. In other words, if screaming frog finds and report those URLs back to you does not mean that Googlebot won’t get the noindex meta robots tag instructions.

Vote Up Vote Down