#review-form The rules I've tried are:

I have a similar URL in my website that I want to stop all robots from crawling /review/dir/dir/dir/dir/#review-form

The rules I've tried are:

Disallow:/review/*review-form$
Disallow:/*#review-form$
Disallow:review-form$
Disallow:/#review-form

The bottom rule works but also blocks the URL without the hash fragment because the # symbol starts a comment.

How can I disallow a URL that has a hash fragment in the robots.txt file?

10.01% popularity Vote Up Vote Down

: Eve of 2018: It is possible for third parties to know our web stats? As per today at the end of 2017, is it possible for someone (like a 3rd party organization or a competitor) to know

@Candy875

Posted in: #Competitors #WebTraffic

1 Comments

: Add custom domain on Azure without application restart? Few months ago Azure started restarting applications when change occurs in custom domains. This behavior is, of course, quite bothersome as,

@Candy875

Posted in: #Azure #Domains #Iis

0 Comments

: How to suppress google page speed Minify JS recommenation Google page speed recommend minifying my JS file which will save a few 100 bytes. Its low priority right now and will have negligible

@Candy875

Posted in: #Google #GooglePagespeed #PageSpeed

0 Comments

: Google DFP - Responsive ads don't render I have https://www.iq-mag.net/dfp.php and it doesn't seem to work - by that I mean no ads are loading at all I checked the console and am getting the

@Candy875

Posted in: #GoogleDfp #Javascript

0 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Nimeshi995

SOURCE
Comments can be included at any location in the file using the "#" character.

You should be able to get around the issue by using URL Encoding and using the code %23, like so:

Disallow: /%23your-url/

10% popularity Vote Up Vote Down

Feed

: How to disallow hash fragments in robots.txt file I have a similar URL in my website that I want to stop all robots from crawling /review/dir/dir/dir/dir/#review-form The rules I've tried are:

More posts by @Candy875

: Eve of 2018: It is possible for third parties to know our web stats? As per today at the end of 2017, is it possible for someone (like a 3rd party organization or a competitor) to know

: Add custom domain on Azure without application restart? Few months ago Azure started restarting applications when change occurs in custom domains. This behavior is, of course, quite bothersome as,

: How to suppress google page speed Minify JS recommenation Google page speed recommend minifying my JS file which will save a few 100 bytes. Its low priority right now and will have negligible

: Google DFP - Responsive ads don't render I have https://www.iq-mag.net/dfp.php and it doesn't seem to work - by that I mean no ads are loading at all I checked the console and am getting the

Login to post a comment!

1 Comments

Back to top | Use Dark Theme