: How to block pagination when the URL is just one number? I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter
I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. Example:
/category/sub-category/product/
/category/sub-category/product/1
/category/sub-category/product/2
/category/sub-category/product/3
In the example above, the URLs that end with 1, 2, and 3 is pagination. Any idea on how I can block these? I don't want to just block all numbers, as this is not an ideal solution.
More posts by @Dunderdale272
2 Comments
Sorted by latest first Latest Oldest Best
If you wanted to block just the page's listed, this is what you would need to add to your robots.txt:
Disallow: /category/sub-category/product/1$
Disallow: /category/sub-category/product/2$
Disallow: /category/sub-category/product/3$
You could also use robots 'NOINDEX, FOLLOW' meta tag in the head section of the paginated source code. This would still allow Google to crawl these URLs (and still discover links on them), but will stop Google indexing them. If just blocking them via robots.txt, Google will not crawl the URLs any longer, and wont find other links that may be on them.
As you mentioned rel next and prev tags could also be a solution here, but using simply canonical tags would be incorrect if the content differs from one page to another.
If you have possible duplicate content with your pagination, the idea is not to block pagination URLs with robots.txt but just defining the rel="canonical" tag for your different URLs.
For your four URLs, define this tag in your <head> section:
<link rel="canonical" href="http://www.example.com/category/sub-category/product/"/>
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.