6 ms·

A while back, Firefox removed the ability to override the text encoding of pages. If Firefox gets the encoding wrong, you are screwed. The thing is, according t

A while back, Firefox removed the ability to override the text encoding of pages. If Firefox gets the encoding wrong, you are screwed. The thing is, according to the developers themselves, detecting various single-byte Latin encodings is very unreliable, and they have indicated that making Firefox detect one encoding more reliably nearly always means introducing failures at detecting another one, so there will be no progress in this area. Despite this, they have seen it fit to remove the encoding menu, because telemetry shows that most users don’t use it, and because when they do, it often takes them multiple tries to guess the correct encoding. Their solution? To replace it with a “guess again” button, completely removing the ability to choose manually. Henri, one of the developers responsible for this change, has argued that as long as both encodings are Latin-script encodings with a common ASCII subset, it is not catastrophic for the user to be stuck with the wrong encoding, because the text is likely “still legible”. To give an example of what he is calling an “acceptable” level of mojibake, consider this text from the Polish Wikipedia, encoded as Latin-2 and decoded as Latin-1. > Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. Personally, I think this change is ridiculous. It has been a few versions already since it was rolled out, but people are still complaining in the issue tracker. I know one Russian guy who has resorted to using an extension which replaces arbitrary strings in order to correct common mojibake sequences in order to deal with the regression in functionality brought about by this change. If you feel strongly about this like I do, I strongly encourage you to comment in the Bugzilla thread. Yes, the web should be using Unicode these days, but if it isn’t, that is not your fault as a user, and making the experience miserable for the end user is not justifiable.

hsivonen 5y ago
Your example is not an example of actual Firefox failure mode: If I encode your text as ISO-8859-1 and run chardetng on the bytes, it says ISO-8859-2. It’s generally unlikely that chardetng would misdetect ISO-8859-2 Polish as windows-1252. When you claim that something isn’t working for you (and it’s completely plausible that you have encountered a case where chardetng doesn’t work for you), it would be polite to post the actual failure and not something that’s not an actual example of failure.
deleted 5y ago
[deleted]
phoe-krk 5y ago
> Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. As a Polish person who has been seeing this sort of mis-encoded Polish text for over two decades now, my gut instinct is to immediately reach for the encoding menu. That menu is gone now. We live in the era of almost omnipresent UTF-8, but it simply feels wrong to remove backwards compatibility with older documents on the Polish web that are mis-encoded like that - and there are still some of them out there.
- ajsnigrutin 5y ago
  yep, as someone from the balkans with old sites and iso8859-2 vs cp1250, I used this feature sometimes too... well.. I guess not anymore.
  - goalieca 5y ago
    I wonder how possible it would be to make an extension
    - nneonneo 5y ago
      It turned out to be pretty easy; I hacked one up in a few hours and got it published on AMO: https://addons.mozilla.org/en-US/firefox/addon/override-text-encoding/ https://addons.mozilla.org/en-US/firefox/addon/override-text... It's open-sourced on GitHub in case you're curious. It just overrides the Content-Type header on pages where you've turned it on, setting the `charset` parameter to whatever you select.
- jopsen 5y ago
  Would it be impossible to put this in an extension? What is the compatibility worth if people have to manually figure out how to switch it on?
RunSet 5y ago
Good luck but I am not hopeful Firefox's behavior will be corrected. See also this 13 year old pearl courtesy of Bugzilla: "Set screen coordinates during HTML5 drag event" > The current HTML5 spec describes that all DragEvent properties should be available during all the events - according to editor Ian Hickson. >> Note though that it doesn't specify what the properties should be set to, just that they should be set and we currently set them to 0. https://bugzilla.mozilla.org/show_bug.cgi?id=505521 https://bugzilla.mozilla.org/show_bug.cgi?id=505521
Asooka 5y ago
- dan-robertson 5y ago
  Do other browsers offer character encoding menus or are they from improper companies? (Or were the developers fired but their changes kept?)
- baybal2 5y ago
- nneonneo 5y ago
  This person makes a decision supported by reasonable evidence and argumentation, confirmed through discussion with other members of the organization and wider community, and shipped after a thorough review (all happening in the open on a bug tracker!); and you're calling for them to be fired? I sure hope you don't make any mistakes at your company!
  - xg15 5y ago
    > all happening in the open on a bug tracker! I find it suspicious that there was not a single comment from a user before the change shipped - but immediately after, there were various complaints. That's not the first time I see this pattern in a bug. This looks more like the "Hitchhiker's Guide to the Galaxy" definition of "open" - where the discussion is technically public, but you heavily rely on the fact that no one knows it exists.
    - zozbot234 5y ago
      That's a good point. The removal of such user-critical features should at the very least be announced well in advance via a clear deprecation notice, to allow for the collection of relevant feedback.
      - wolverine876 5y ago
mnd999 5y ago
mnd999 5y ago
Seems like they restricted the comments, because, you know, head in the sand is the standard Mozilla approach.
- d3nj4l 5y ago
  Or, maybe, hear me out here: deliberately posting a link to an issue to a widely-visited community with an inflammatory title, with the explicit intent of dogpiling the maintainer led to them locking the thread? You know, something like what HN itself does to prevent vote manipulation?
vesinisa 5y ago
It seems he did extensive background research to investigate and find optimal solution for this issue: https://hsivonen.fi/chardetng/ https://hsivonen.fi/chardetng/ Goal was apparently to bring FF to feature parity with Chrome. Sad to see it's not working as well as intended. (Even sadder to see his person being attacked by some commenters below.)
- mnd999 5y ago
  Extensive maybe, correct apparently not so much.
- serentty 5y ago
  I agree that good detection is important and I am glad that he is working on it, but I don’t think that the level of detection which is possible is a good replacement for the menu, based on what he himself has indicated he thinks is possible to achieve in Firefox.
- fami-com 5y ago
  The problem is that it's impossible to guess correctly 100% of the time. You can improve the guessing algorithm as much as you want to (not that I'm saying it's useless - it's very good for regular users who only rarely encounter such issues), but you can never achieve total accuracy, which is why some sort of manual override is absolutely necessary in cases when it fails.
dollop43 5y ago
I seldom use this feature, but when I do and I can't find the encoding menu, then it will be a very frustrating experience. Their logic seems to be like dialing emergency service is a rarely use feature based on telemetry, so we will just remove that. No discussion is allowed, if you challenged, then they ask you about do you have data about the use case of dialing emergency service? People can still send text to emergency service, it is fine. And because of this post, I just checked if chromium supported encoding menu. Yep, they removed it long ago. Basically, when you need to read some very old Japanese site where their encoding is pretty non standard, you are screwed.
- CrendKing 5y ago
  > when you need to read some very old Japanese site where their encoding is pretty non standard, you are screwed If you need to read them frequently, yes. Otherwise, if this is just one-off, maybe user can download the page source, extract text content, paste into text editor and select encoding? Do you have an example CJK site I may try?
  - dollop43 5y ago
    I rarely need to do that, so I cannot find an example for the moment. It is usually a one-off thing, then yes I could do the conversion manually, but it was obviously an usability degraded.
- nneonneo 5y ago
  Per https://hsivonen.fi/encoding-telemetry/ https://hsivonen.fi/encoding-telemetry/, it looks like they run a special encoding detector on .jp sites, which is presumably designed specifically to choose between the various Japanese encodings (Shift-JIS, EUC-JP, etc). I think your example of emergency services is a bit hyperbolic. This is a feature that is really not often used, whose omission is not fatal, which often requires several tries to get right, and which is increasingly less useful thanks to gradual changes on the Web. Much more widely-used features like FTP and Flash have been deprecated; people howl and yell every time, but yet things still seem to work.
kevin_b_er 5y ago
Too late, code merged. Bugzilla locked. You the user do not matter really to Firefox anymore. If the encoding is broken, it cannot be fixed by Henri Sivonen auto-detection code, but he can claim he "simplified the menus" as a bullet point to project managers. This is the gradual but inevitable backslide of Firefox into Chrome.
- doublepg23 5y ago
  Chrome has marketshare.
vvillena 5y ago
Simplifying a submenu by replacing it with a quick option that works in most cases is a great improvement, but removing the underlying feature that lets users select the text encoding manually is bad. There must be a good way to keep the menu bar simple while keeping the option. Firefox already shows page-specific info in the left side of the address bar. There's usually a shield icon and a lock icon, and permission requests appear there too. Why not add an "encoding icon" that shows whenever the encoding repair feature is active? Such a marker would also be a great place to put a "override with custom encoding" menu.
- xg15 5y ago
  The quick option was already there before though - it was labeled "automatic" before. The only thing he did was remove all other options.
deleted 5y ago
[deleted]

hsivonen 5y ago

Your example is not an example of actual Firefox failure mode: If I encode your text as ISO-8859-1 and run chardetng on the bytes, it says ISO-8859-2. It’s generally unlikely that chardetng would misdetect ISO-8859-2 Polish as windows-1252. When you claim that something isn’t working for you (and it’s completely plausible that you have encountered a case where chardetng doesn’t work for you), it would be polite to post the actual failure and not something that’s not an actual example of failure.

deleted 5y ago

[deleted]

phoe-krk 5y ago

> Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. As a Polish person who has been seeing this sort of mis-encoded Polish text for over two decades now, my gut instinct is to immediately reach for the encoding menu. That menu is gone now. We live in the era of almost omnipresent UTF-8, but it simply feels wrong to remove backwards compatibility with older documents on the Polish web that are mis-encoded like that - and there are still some of them out there.

ajsnigrutin 5y ago
yep, as someone from the balkans with old sites and iso8859-2 vs cp1250, I used this feature sometimes too... well.. I guess not anymore.
- goalieca 5y ago
  I wonder how possible it would be to make an extension
  - nneonneo 5y ago
    It turned out to be pretty easy; I hacked one up in a few hours and got it published on AMO: https://addons.mozilla.org/en-US/firefox/addon/override-text-encoding/ https://addons.mozilla.org/en-US/firefox/addon/override-text... It's open-sourced on GitHub in case you're curious. It just overrides the Content-Type header on pages where you've turned it on, setting the `charset` parameter to whatever you select.
jopsen 5y ago
Would it be impossible to put this in an extension? What is the compatibility worth if people have to manually figure out how to switch it on?

RunSet 5y ago

Good luck but I am not hopeful Firefox's behavior will be corrected. See also this 13 year old pearl courtesy of Bugzilla: "Set screen coordinates during HTML5 drag event" > The current HTML5 spec describes that all DragEvent properties should be available during all the events - according to editor Ian Hickson. >> Note though that it doesn't specify what the properties should be set to, just that they should be set and we currently set them to 0. https://bugzilla.mozilla.org/show_bug.cgi?id=505521 https://bugzilla.mozilla.org/show_bug.cgi?id=505521

Asooka 5y ago

dan-robertson 5y ago
Do other browsers offer character encoding menus or are they from improper companies? (Or were the developers fired but their changes kept?)
baybal2 5y ago
nneonneo 5y ago
This person makes a decision supported by reasonable evidence and argumentation, confirmed through discussion with other members of the organization and wider community, and shipped after a thorough review (all happening in the open on a bug tracker!); and you're calling for them to be fired? I sure hope you don't make any mistakes at your company!
- xg15 5y ago
  > all happening in the open on a bug tracker! I find it suspicious that there was not a single comment from a user before the change shipped - but immediately after, there were various complaints. That's not the first time I see this pattern in a bug. This looks more like the "Hitchhiker's Guide to the Galaxy" definition of "open" - where the discussion is technically public, but you heavily rely on the fact that no one knows it exists.
  - zozbot234 5y ago
    That's a good point. The removal of such user-critical features should at the very least be announced well in advance via a clear deprecation notice, to allow for the collection of relevant feedback.
    - wolverine876 5y ago

mnd999 5y ago

Seems like they restricted the comments, because, you know, head in the sand is the standard Mozilla approach.

d3nj4l 5y ago
Or, maybe, hear me out here: deliberately posting a link to an issue to a widely-visited community with an inflammatory title, with the explicit intent of dogpiling the maintainer led to them locking the thread? You know, something like what HN itself does to prevent vote manipulation?

vesinisa 5y ago

It seems he did extensive background research to investigate and find optimal solution for this issue: https://hsivonen.fi/chardetng/ https://hsivonen.fi/chardetng/ Goal was apparently to bring FF to feature parity with Chrome. Sad to see it's not working as well as intended. (Even sadder to see his person being attacked by some commenters below.)

mnd999 5y ago
Extensive maybe, correct apparently not so much.
serentty 5y ago
I agree that good detection is important and I am glad that he is working on it, but I don’t think that the level of detection which is possible is a good replacement for the menu, based on what he himself has indicated he thinks is possible to achieve in Firefox.
fami-com 5y ago
The problem is that it's impossible to guess correctly 100% of the time. You can improve the guessing algorithm as much as you want to (not that I'm saying it's useless - it's very good for regular users who only rarely encounter such issues), but you can never achieve total accuracy, which is why some sort of manual override is absolutely necessary in cases when it fails.

dollop43 5y ago

I seldom use this feature, but when I do and I can't find the encoding menu, then it will be a very frustrating experience. Their logic seems to be like dialing emergency service is a rarely use feature based on telemetry, so we will just remove that. No discussion is allowed, if you challenged, then they ask you about do you have data about the use case of dialing emergency service? People can still send text to emergency service, it is fine. And because of this post, I just checked if chromium supported encoding menu. Yep, they removed it long ago. Basically, when you need to read some very old Japanese site where their encoding is pretty non standard, you are screwed.

CrendKing 5y ago
> when you need to read some very old Japanese site where their encoding is pretty non standard, you are screwed If you need to read them frequently, yes. Otherwise, if this is just one-off, maybe user can download the page source, extract text content, paste into text editor and select encoding? Do you have an example CJK site I may try?
- dollop43 5y ago
  I rarely need to do that, so I cannot find an example for the moment. It is usually a one-off thing, then yes I could do the conversion manually, but it was obviously an usability degraded.
nneonneo 5y ago
Per https://hsivonen.fi/encoding-telemetry/ https://hsivonen.fi/encoding-telemetry/, it looks like they run a special encoding detector on .jp sites, which is presumably designed specifically to choose between the various Japanese encodings (Shift-JIS, EUC-JP, etc). I think your example of emergency services is a bit hyperbolic. This is a feature that is really not often used, whose omission is not fatal, which often requires several tries to get right, and which is increasingly less useful thanks to gradual changes on the Web. Much more widely-used features like FTP and Flash have been deprecated; people howl and yell every time, but yet things still seem to work.

kevin_b_er 5y ago

Too late, code merged. Bugzilla locked. You the user do not matter really to Firefox anymore. If the encoding is broken, it cannot be fixed by Henri Sivonen auto-detection code, but he can claim he "simplified the menus" as a bullet point to project managers. This is the gradual but inevitable backslide of Firefox into Chrome.

doublepg23 5y ago
Chrome has marketshare.

vvillena 5y ago

Simplifying a submenu by replacing it with a quick option that works in most cases is a great improvement, but removing the underlying feature that lets users select the text encoding manually is bad. There must be a good way to keep the menu bar simple while keeping the option. Firefox already shows page-specific info in the left side of the address bar. There's usually a shield icon and a lock icon, and permission requests appear there too. Why not add an "encoding icon" that shows whenever the encoding repair feature is active? Such a marker would also be a great place to put a "override with custom encoding" menu.

xg15 5y ago
The quick option was already there before though - it was labeled "automatic" before. The only thing he did was remove all other options.

deleted 5y ago

[deleted]