: Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter In our pages we have internal links that are urlencoded because the URL may sometime contain

@Pierce454

I think that the problem is that you have non-encoded forms of the links somewhere in your site. Googlebot has found those links and is trying to access those.

So, try to doublecheck that all links in your page are actually URL encoded.

The post you were referring to talks about double-encoding URLs in Giigle sitemaps, which is a separate issue.

Vote Up Vote Down

@Heady270

Section 2.2 of RFC 3986 addresses reserved "sub delimiters" such as !:

It says that when you produce URLs with these characters you should url encode them:


URI producing applications should percent-encode data octets that
correspond to characters in the reserved set unless these characters
are specifically allowed by the URI scheme to represent data in that
component.


However, it also says that you should accept the characters either encoded on unencoded unless the unescaped version of the character is used as a delimiter in that particular portion of the URL:


If a reserved character is found in a URI component and
no delimiting role is known for that character, then it must be
interpreted as representing the data octet corresponding to that
character's encoding in US-ASCII.


! has no delimiting role in the path portion of a http URL. You should not be returning a 400 status when you encounter it unencoded there. You should be treating %21 and ! the same and showing the same page in either case.

Vote Up Vote Down