: CMS text encoding and common browser decoding; recent change? I'm managing a CMS-based website. The primary language of the CMS is English. Our content is mostly in English, but user postings

@Murphy175

Resume

Your server had and haven't now AddDefaultCharset direcive (namely - AddDefaultCharset utf-8).

This is single reason for described past and current view.

Return (try) it back over htaccess

Bonus answer on q.4


charset-dependent code changes to text-manipulation exist always - most string functions have pairs as mb*()
when you start to think about 8bit-texts in (My)SQL, you have to think about DB-charset|client charset|connection charset|natural sorting|string size. Closed eyes and bad memory seems better choice ("English rule the world" mantra)


Bonus answer on q.5

No, you miss nothing. Old good results was just spoofed, you must got headache with 8bit from first strings on site

Vote Up Vote Down

@Jamie184

As I understand it, the character encoding used by the browser is decided in the following order:


The Content-Type response header as sent from the server.
If not #1 then the Content-Type META tag.
If neither of the above then the browser default, which I assume is based initially on the default language on the system.


AFAIK the default encoding in the browser has not changed. And I very much doubt that simply upgrading to the latest version of a browser would touch the default that was already set.

Also, I don't think it is even possible to accurately deduce the correct character encoding by analysing the page content. It needs to be told, or it falls back to the browser default. Incidentally, on the machine I'm sat at, both Firefox 3.6 and Chrome 14 both default to ISO-8859-1.


Question 1: iso-8859-1 is effectively a subset of UTF-8, right?


Unfortunately, that's not the case - US-ASCII is a subset of utf-8, but non-ASCII characters in iso-8859-1 are encoded differently than in utf-8.


Question 2: Has the standard/practice changed, so that recent browser
releases obey the header declaration of text encoding and won't
override it no matter what encoding is actually present?


I'm unaware of any change. This would surely break a lot of sites? A browser cannot override the character encoding, as specified by the site, by what it thinks should be the character encoding! Can it?!


Question 3: If not, what other factors could account for the sudden
appearance of garbage where there was none before? (For example, could
a subtle change to Apache made by the hosting provider have this
effect?)


Yes, I would guess a change in server config could be causing this. The Content-Type response header (as mentioned above) might have changed.


Question 4: (bonus!) If there is no encoding-dependent code in the
CMS, what kind of technical reasons might make the CMS developers
reluctant to move their primary distribution to UTF-8?


How old is the CMS? What is it written in? Historically PHP does not handle multi-byte strings too well. There are many functions in PHP that only work on single byte strings. Some of the multi-byte functions are only available with PHP 5.

Vote Up Vote Down

@Shelley277

Utf-8 and non English chars do not display well in iso-8859-1 and iso-8859-1 chars do not display well in utf-8.

If you want to know the encoding your browser has detected and is using on a page:

In Firefox just go to the menu "View" > "Character encoding".
support.mozilla.com/en-US/kb/Menu%20Reference

Make sure UF-8 is automatically selected when you read a page and when you enter text in the CMS to be posted.
If possible configure the cms to use html character entities.
Try replacing the meta tag to utf-8, or to the char encoding that displays well in your site.
Please note that invisible chars may appear if you are using the wrong encoding

Vote Up Vote Down