: Using robots.txt to block sessionID URLs I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something

Posted in: #RobotsTxt #Url #UrlParameters #WebCrawlers

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment).

An example URL is:

catalog/product-category/product/;jsessionid=Mf87s+Xw2P8ByQYz2CyQjEJh.prod-14?f=p%3A100-200

Can I update my robots.txt to say:

Disallow: /;jsessionid=*

Does anyone see an issue with doing that? I can also canonicalize the pages, but I feel that using robots is a better solution so Googlebot won't have to waste any resources crawling the URLs in the first place.

10.01% popularity Vote Up Vote Down

: What does the tag do in wordpress? I saw that using <hr /> before and after some content sort of lifts the content. Could anybody share what does the tag do and does it come/have

@Megan663

Posted in: #Html #Tags #Wordpress

3 Comments

: Session ID Creating 404 Errors Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created

@Megan663

Posted in: #Googlebot #GoogleSearchConsole #UrlParameters

1 Comments

: How to have domain point to github pages? So I have a domain registered with 1&1 and I recently decided to create my website using GitHub pages. I added as GitHub told me to, but how

@Megan663

Posted in: #1and1 #Cname #Domains #Github #WebHosting

1 Comments

: Note: There's now an entire Stack Exchange site devoted to the subject of User Experience, so please consider whether your question would be better asked there. Questions on Stack Overflow should

@Megan663

0 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Eichhorn148

you should exclude such urls from crawling, because you spend your whole crawling budget for those useless urls. Read this topic at Google Product Forum.

set the robots rule like:

disallow: /*jsessionid*

10% popularity Vote Up Vote Down

Feed

: Using robots.txt to block sessionID URLs I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something

More posts by @Megan663

: What does the tag do in wordpress? I saw that using <hr /> before and after some content sort of lifts the content. Could anybody share what does the tag do and does it come/have

: Session ID Creating 404 Errors Problem: Website contains URLs with parameters Session ID gets inserted into URLs when Googlebot crawls page with parameter (sorting) options Session ID is created

: How to have domain point to github pages? So I have a domain registered with 1&1 and I recently decided to create my website using GitHub pages. I added as GitHub told me to, but how

: Note: There's now an entire Stack Exchange site devoted to the subject of User Experience, so please consider whether your question would be better asked there. Questions on Stack Overflow should

Login to post a comment!

1 Comments

Back to top | Use Dark Theme