10 ms·

Replace the Text Encoding menu with a single override item

88 points by serentty 5y ago

bombcar 5y ago
Encoding privilege is not something I expected to be enforced by Firefox. I guess I’m lucky to not be hindered by this - because I can’t ready any language that uses non English characters to the point where I can’t guess what they would be.
serentty 5y ago
A while back, Firefox removed the ability to override the text encoding of pages. If Firefox gets the encoding wrong, you are screwed. The thing is, according to the developers themselves, detecting various single-byte Latin encodings is very unreliable, and they have indicated that making Firefox detect one encoding more reliably nearly always means introducing failures at detecting another one, so there will be no progress in this area. Despite this, they have seen it fit to remove the encoding menu, because telemetry shows that most users don’t use it, and because when they do, it often takes them multiple tries to guess the correct encoding. Their solution? To replace it with a “guess again” button, completely removing the ability to choose manually. Henri, one of the developers responsible for this change, has argued that as long as both encodings are Latin-script encodings with a common ASCII subset, it is not catastrophic for the user to be stuck with the wrong encoding, because the text is likely “still legible”. To give an example of what he is calling an “acceptable” level of mojibake, consider this text from the Polish Wikipedia, encoded as Latin-2 and decoded as Latin-1. > Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. Personally, I think this change is ridiculous. It has been a few versions already since it was rolled out, but people are still complaining in the issue tracker. I know one Russian guy who has resorted to using an extension which replaces arbitrary strings in order to correct common mojibake sequences in order to deal with the regression in functionality brought about by this change. If you feel strongly about this like I do, I strongly encourage you to comment in the Bugzilla thread. Yes, the web should be using Unicode these days, but if it isn’t, that is not your fault as a user, and making the experience miserable for the end user is not justifiable.
- hsivonen 5y ago
  Your example is not an example of actual Firefox failure mode: If I encode your text as ISO-8859-1 and run chardetng on the bytes, it says ISO-8859-2. It’s generally unlikely that chardetng would misdetect ISO-8859-2 Polish as windows-1252. When you claim that something isn’t working for you (and it’s completely plausible that you have encountered a case where chardetng doesn’t work for you), it would be polite to post the actual failure and not something that’s not an actual example of failure.
- deleted 5y ago
  [deleted]
- phoe-krk 5y ago
  > Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. As a Polish person who has been seeing this sort of mis-encoded Polish text for over two decades now, my gut instinct is to immediately reach for the encoding menu. That menu is gone now. We live in the era of almost omnipresent UTF-8, but it simply feels wrong to remove backwards compatibility with older documents on the Polish web that are mis-encoded like that - and there are still some of them out there.
  - ajsnigrutin 5y ago
    yep, as someone from the balkans with old sites and iso8859-2 vs cp1250, I used this feature sometimes too... well.. I guess not anymore.
    - goalieca 5y ago
      I wonder how possible it would be to make an extension
zerocrates 5y ago
Just another casualty of telemetry. I'm sure it doesn't help that Chrome killed its own encoding menu years ago.
- Gigachad 5y ago
  Isn't this issue fixed by websites fixing their encoding? If no browser is able to properly display the content, I imagine they will be fixed pretty quickly if they are at all maintained.
  - roughly 5y ago
    Yes, but, they’re not. Huge swaths of the web are effectively abandoned, and will not get fixed.
    - Gigachad 5y ago
      Then the best option would be to copy out the useful content to new maintained sites and let the old ones die. Rather than keeping browsers tied to supporting long long dead features. UTF-8 has been standards for over a decade now. I think there has been plenty of migration time. More than flash had.
- rjh29 5y ago
  Chrome's position was "use an extension for this", but then users have to risk privacy by installing a third-party extension. What harm does having the option tucked somewhere in advanced settings do?
  - sodality2 5y ago
    Every release they probably have to test that it still works. Not that I would ever justify removing something like this - it's clear that for a percentage of the userbase, this is necessary for them to browse. But I can see a product manager trying to cut "fat" if they see that 0.01% of people use it.
- 5y ago
kevingadd 5y ago
The telemetry-driven desire to redesign everything and remove lots of 'infrequently used' features is a real plague on software and it really sucks that it seems to have infested the web because of the amount of novelty-obsessed designers, PMs and engineers who move from one company to another. Lots of chrome, firefox and safari devs are people who previously worked on a different browser or a company with this culture and eventually your team has enough people like that to push through things like 'remove this important low-cost feature because only 1% of our billion users are impacted', as if 1% of a billion isn't a significant number of people. It used to be that even minor web platform changes wouldn't go through if they would break 0.1% of webpages, but we don't live in that world anymore.
- Gigachad 5y ago
  >but we don't live in that world anymore. And I'm glad. I'm sick of using software that has 2 trillion toolbars, menus and settings. I want something which sets the defaults to be pretty much what I want and makes the settings all the common things I might need to change. Telemetry driven design lets designers build what the majority like rather than the most vocal bug tracker using power users who want exactly the opposite of the average user.
  - vvillena 5y ago
    Must be nice to always live in the middle of the bell curve.
    - Gigachad 5y ago
      The defaults aren’t always exactly what I want but I find it is preferable to just accept them as they are than to have a billion customisation options. I pick the best bit of software that works for me and then just change my workflow to fit it.
- PragmaticPulp 5y ago
  > The telemetry-driven desire to redesign everything and remove lots of 'infrequently used' features is a real plague on software I disagree. Maintaining features requires engineering overhead. It sucks to have to maintain features that barely anybody uses when you have a backlog of features that are in demand by a large number of your users.
dan-robertson 5y ago
It’s pretty unclear to me whether people use this feature much. Two possibilities seem likely to me: 1. Very few people use this feature but a few vocal users who are particularly good at navigating bugtracking bureaucracy do use it and care deeply. 2. Of all Firefox users, a small percentage speak languages for which there are many non-ascii letters and a relatively large amount of pre-Unicode content online, and of those users a small percentage read niche (eg non-facebook and old) sites with broken character encoding. Both situations could probably lead to few uses of the menu in telemetry. Perhaps we should care more about users in the second category as they are perhaps more correlated with other things like location whereas the first set may be more uniformly distributed across the population. I think other points are also being disregarded: - ordinary users may find the feature cryptic and confusing. Or at least undiscoverable - dealing with character encoding and supporting the feature is painful (though the will still deal with it under the hood). Mozilla have limited resources and they (and Firefox users) want the browser to be successful and sustainable, and this means focusing resources on things that will make the browser easier to adopt and less confusing rather than chasing the short-lived gains from appeasing a small and stagnant pool of ‘power users’.
- baybal2 5y ago
  > Mozilla have limited resources and they (and Firefox users) want the browser to be successful and sustainable, To begin with, near nobody uses Firefox today. Very succesful, and sustainable. I have to pour heavy sarcasm here. It's honestly justly needed. Firefox went from 50%+ to sub 1% under the current leadership.
  - yoasif_ 5y ago
    > To begin with, near nobody uses Firefox today. Very succesful, and sustainable. 215 million MAU is still a lot of people: https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/user-activity
- zozbot234 5y ago
  > dealing with character encoding and supporting the feature is painful This feature is in no way "painful". It's a legitimate facility for dealing with non-Unicode, legacy content, and AFAICT it's not clear that it can be reliably replaced, even by a plugin. Any argument that the menu entry is "confusing" for most users can be addressed by hiding it behind an advanced preference setting, and I think this was done already.
nfriedly 5y ago
Could the menu be restored by an add-on?
butz 5y ago
There are still documents on the internet that use ASCII encoding, and while English speaking users doesn't have this problem, in some countries there were several different encoding systems used back in the past. While most believe that web content is resilient and we will still be able to browse old websites forever, recent web browser developments show otherwise. Please do not remove working features, especially if they have no effect on security or privacy.
- wolverine876 5y ago
  > There are still documents on the internet that use ASCII encoding Probably a great many because ASCII is a subset of UTF-8, and afaict is common for English when the site doesn't use more sophisticated typography such as curly quotes, em dashes, etc.
- hsivonen 5y ago
  > Please do not remove working features, especially if they have no effect on security or privacy. As it happens, part of what made the menu back end cause other changes to take more effort than they should have taken on their face were the security aspects of the menu in Firefox. Try https://hsivonen.com/test/moz/never-show-user-supplied-content-as-utf-16.htm16le https://hsivonen.com/test/moz/never-show-user-supplied-conte... in Firefox 90 (which had the menu) and in another browser that has a menu or an extension that adds a menu.
mnau 5y ago
Seems like clickbait title. I have seen this issue few times over the years and it was always a game of "Which encoding should I choose?" Having an automated way to solve this seems pretty good to me. Especially if the solution works pretty much 100% of time (especially for non-ASCII languages, like Japanese or Chinese) - see paragraph Document-length Accuracy in https://hsivonen.fi/chardetng/ https://hsivonen.fi/chardetng/
- serentty 5y ago
  East Asian languages are not the main concern here, actually. It’s the various single-byte Latin script encodings, which are incredibly similar. The issue of these being confused is addressed in the thread itself, where he says that such issues will not be fixed, because they are so confusable that fixing the detection of one means breaking the detection of another.
  - mnau 5y ago
    I view it as a firefox fixing incorrectly written pages. It's nice that FF has a "fix-it" option, but ultimately, the author shoudl fix her site. The single-byte Latin scripts are semi legible, so that is enough for me. When a website breaks because 3rd party cookies go way of dodo or the page relies on ActiveX, shoudl Firefox provide an option to run it? I understand that some people have different opinion, but any page past 2005 should have encoding in order and older sites should fix their sites.
    - serentty 5y ago
      “Should” is not “is”. As a user, I run into broken sites regularly enough. I still want to be able to see them. I get the “let’s bury all broken legacy and hacks as quickly as possible” mindset, but when it comes to the issue of the main text of a page being unreadable, I think there is a high bar. > older sites should fix their sites. A lot of the reason they are considered “older sites” in the first place is because they are not heavily maintained. The thing to remember about the web is this: not all websites are “software” that is maintained. Some of them are static works similar to a movie, with static content. > The single-byte Latin scripts are semi legible, so that is enough for me. Did you see the example that I gave of Polish text? I am sorry, but if you feel like reading multiple paragraphs this way is bearable, then we will just be stuck in disagreement.
twhb 5y ago
I like this change. If a website misconfigures encoding and users have an encoding override menu, the result is 99% of users see a broken website, 1% know how to fix it. The menu doesn’t meaningfully improve the situation, but it does keep the people who could improve it from caring. Take that away and I predict 10%+ of currently misconfigured websites will get fixed for 100% of visitors, and that’s a big improvement over 100% of websites being fixed for 1% of visitors.
- serentty 5y ago
  So you think that making things miserable for the remaining 1% is good because it gets them to care, and therefore for the website to be fixed? I’m sorry, but most of the time when a website has a borked encoding, it is some website where I doubt I would ever get in contact with the webmaster anyway. It’s mostly small websites. But small websites are important to maintaining a healthy web where things are not dominated by a few big players. But perhaps most importantly, I have never agreed with any mindset of that encourages punishing some subset of users in order to get someone to change their ways, even more so if that isn’t even the users themselves.
  - yoasif_ 5y ago
    > I’m sorry, but most of the time when a website has a borked encoding, it is some website where I doubt I would ever get in contact with the webmaster anyway. So you wouldn't bother to try to get the webmaster to fix the issue, you would just resolve it on your own, letting everyone else be miserable? > But perhaps most importantly, I have never agreed with any mindset of that encourages punishing some subset of users in order to get someone to change their ways, even more so if that isn’t even the users themselves. It sounds like you don't care if people are punished, as long as it isn't you.
    - serentty 5y ago
      > So you wouldn't bother to try to get the webmaster to fix the issue, you would just resolve it on your own, letting everyone else be miserable? I mean that I would not be able to. Huge swaths of the web are static content that was just put there and then just abandoned. The web will remain this way unless everything is moved onto SPA social networks, and I hope that we agree that that would not a good thing. > It sounds like you don't care if people are punished, as long as it isn't you. If the added benefit of causing people more inconvenience is that they push for problems to be fixed, I don’t think it’s fair to cause that inconvenience with the intent of using that to solve the problem. I think it’s a manipulative tactic. That is my idea of what “punishment” we’re talking about. What kind of punishment are you thinking of if you think I’m fine with punishment, and who is getting punished by it?
jcranmer 5y ago
I'm of a few different minds on this. First off, what needs to be acknowledged is that any website that requires you to override the charset is broken and the greatest part of your ire should be directed at the people who write and operate such websites for expecting that everybody else should be responsible for cleaning up their messes. Let's also acknowledge that avoiding this breakage has been possible for (checks notes) 22 years. Since this is very legacy stuff, at what point can we say that it is no longer the end consumer's job to deal with legacy cruft? The motivation for why a user-level override was ever necessary in the first place--the fact that once upon a time computers couldn't simultaneously represent content in different locales--is quite ancient history. I don't think an answer of "never" is compelling: why then aren't we complaining that browsers don't let us select EBCDIC or UTF-7 as an encoding (both of which were once supported, but have been dropped)? At the same time, this change does somehow still feel premature; dropping the ability to override charset encoding does feel very "WTF?" to me. But... if that's the case, then what can be done to hasten the moment where we live in a world where dropping this feature doesn't feel like that? (This makes me glad I work in compilers, where no one attacks you for deciding that some inputs are just too wrong to attempt to deal with it.)
- zozbot234 5y ago
  Unicode encoding was far from universally adopted in the early 2000s. Some legacy content will always exist, and the moment where dropping the feature is harmless is also the moment where there's no reason to drop it in the first place, since it only ever comes into play with broken legacy content; otherwise, it is entirely hidden and has no harmful effect whatsoever.
  - jcranmer 5y ago
    Unicode encoding isn't what I was referring to--it was the ability to specify <META http-equiv="Content-Type" content="text/html; charset=EUC-JP"> (copied from HTML 4.01 specification, dated December 24, 1999). That might not be the first time it was possible to declare the charset of an HTML document within HTML itself, but I don't see any mention of it in HTML 3.2. Also, it's wrong to say that there's "no harmful effect whatsoever"--the ability to do charset overrides requires code to support it, and that code could force awkward compromises in (say) your HTTP layer. All features have costs, even if they're invisible.
TazeTSchnitzel 5y ago
This decision is offensively user-hostile. It breaks a critical feature for dealing with broken and legacy webpages (which, for better or worse, are never going to stop existing) without a good justification. Even for English-speaking users this sucks; there's lots of mailing list archives and the like that mishandle their encoding.
nightpool 5y ago
This headline is unfairly editorialized. What the Firefox dev said was that the automatic encoding detecter confusing some ISO/IEC 8859 encodings for each other is an intractable problem, and fixing one Latin text encoding being auto-detected for a different Latin text encoding is a very low-priority use of time. Also, removing the encoding override is completely separate from that, since it's (apparently) a very complicated feature to keep around that 1) breaks webpages (e.g. etrade) and 2) keeps confusing users and leading them think Firefox is worse at text encoding then Chrome. Regardless of your opinion of the change in question, these two statements are completely different from what the headline implies.
- wolverine876 5y ago
  > This headline is unfairly editorialized. I'm not sure why it doesn't follow HN guidelines to use the page title: "Replace the Text Encoding menu with a single override item Repair Text Encoding"
- serentty 5y ago
  He straight-up said that it’s not as much of an issue because it’s still legible because not all of the letters are borked. I don’t see how this is a misrepresentation of what he said. > Also, removing the encoding override is completely separate from that, It is not separate at all. If the detector cannot do it, and the ability for the user to do it is removed, these issues are intrinsically linked. Sure, there are the other issues you mentioned around keeping it around, but you can’t act like these issues can be considered separately.
- zozbot234 5y ago
  The etrade breakage was a failure of the previous detector, which expected an encoding declaration within the first few page KB and introduced a page reload otherwise. Switching the current encoding manually does not reload the page per se.
wolverine876 5y ago
The title, by HN guidelines, is "Replace the Text Encoding menu with a single override item Repair Text Encoding"
rtpg 5y ago
All the discussion in this thread and in the bugzilla thread feel very odd to me. Like arguing that it's not very used, is a maintenance burden.... and it was already hidden off in a submenu! I'm, of course, glad that the guesser can work more effectively. But isn't this a bit different from something like FTP (which can introduce nasty security bugs?)? And you still end up with "change encoding of page" with the guesser, so it's like... it's not like you are removing some lower-level functionality. It's unfortunate because yeah people mention Flash getting removed in a similar tone, but at least with Flash there are things like ruffle that can pick up the slack. I don't really want to have to install a plugin for the couple of times I land on the page that miss this encoding, but when I do hit the page I'll be frustrated. Of course the original sin here was browsers not throwing warnings or something on every page without the encoding set, and having users become the people who complain to webmasters about it like... 15 years ago or whatever. But meanwhile we had a really _fine_ place for the encoding picker. Maybe first click is "Guess", second reveals the menu... or something. I don't want to hold too much malice towards the devs here cuz getting yelled at by all the internet sucks. But I hope there can be some resolution here that doesn't lead to massive burnout. I will be very happy if there is some backup solution inserted (even if it's behind a conf flag), and I would see this as an overall positive to FF if that happens (listening to feedback!)
emiliocobos 5y ago
To people thinking that this was removed gratuitously please read the bug and linked bugs carefully. This got in the way of implementing various improvements to <meta charset> and interop improvements to XHR.
xg15 5y ago
In addition to the concrete case at hand, this is another instance where explicit user abilities are taken away and replaced by an algorithm. I really hope the tech industry will see the light at some point and stop this trend.
SSLy 5y ago
I use it few times a year to fix websites sending ISO-8859-2 (or Windows-1250) and the guess mode worked. As an user affected by this change, it is an improvement (less need to click around trying 5 different encodings)
- serentty 5y ago
  I am definitely not against the addition of the guesser. It was added a bit before the encoding menu was removed, though, so there was a time when they were both there. I mainly wanted to draw attention to the removal of the menu.
nneonneo 5y ago
After all the heated discussion here, I decided to make this into a plugin: https://addons.mozilla.org/en-US/firefox/addon/override-text-encoding/ https://addons.mozilla.org/en-US/firefox/addon/override-text... It's pretty bare-bones and probably has some bugs (after all, I hacked it up in a few hours...) but it does what it says on the tin: it lets you set the text encoding for any webpage manually, just like the good ol' Encoding menu.
- serentty 5y ago
  I hugely appreciate this! I will definitely install it.

88 points by serentty 5y ago

Encoding privilege is not something I expected to be enforced by Firefox. I guess I’m lucky to not be hindered by this - because I can’t ready any language that uses non English characters to the point where I can’t guess what they would be.

serentty 5y ago

A while back, Firefox removed the ability to override the text encoding of pages. If Firefox gets the encoding wrong, you are screwed. The thing is, according to the developers themselves, detecting various single-byte Latin encodings is very unreliable, and they have indicated that making Firefox detect one encoding more reliably nearly always means introducing failures at detecting another one, so there will be no progress in this area. Despite this, they have seen it fit to remove the encoding menu, because telemetry shows that most users don’t use it, and because when they do, it often takes them multiple tries to guess the correct encoding. Their solution? To replace it with a “guess again” button, completely removing the ability to choose manually. Henri, one of the developers responsible for this change, has argued that as long as both encodings are Latin-script encodings with a common ASCII subset, it is not catastrophic for the user to be stuck with the wrong encoding, because the text is likely “still legible”. To give an example of what he is calling an “acceptable” level of mojibake, consider this text from the Polish Wikipedia, encoded as Latin-2 and decoded as Latin-1. > Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. Personally, I think this change is ridiculous. It has been a few versions already since it was rolled out, but people are still complaining in the issue tracker. I know one Russian guy who has resorted to using an extension which replaces arbitrary strings in order to correct common mojibake sequences in order to deal with the regression in functionality brought about by this change. If you feel strongly about this like I do, I strongly encourage you to comment in the Bugzilla thread. Yes, the web should be using Unicode these days, but if it isn’t, that is not your fault as a user, and making the experience miserable for the end user is not justifiable.

hsivonen 5y ago
Your example is not an example of actual Firefox failure mode: If I encode your text as ISO-8859-1 and run chardetng on the bytes, it says ISO-8859-2. It’s generally unlikely that chardetng would misdetect ISO-8859-2 Polish as windows-1252. When you claim that something isn’t working for you (and it’s completely plausible that you have encountered a case where chardetng doesn’t work for you), it would be polite to post the actual failure and not something that’s not an actual example of failure.
deleted 5y ago
[deleted]
phoe-krk 5y ago
> Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku. As a Polish person who has been seeing this sort of mis-encoded Polish text for over two decades now, my gut instinct is to immediately reach for the encoding menu. That menu is gone now. We live in the era of almost omnipresent UTF-8, but it simply feels wrong to remove backwards compatibility with older documents on the Polish web that are mis-encoded like that - and there are still some of them out there.
- ajsnigrutin 5y ago
  yep, as someone from the balkans with old sites and iso8859-2 vs cp1250, I used this feature sometimes too... well.. I guess not anymore.
  - goalieca 5y ago
    I wonder how possible it would be to make an extension

zerocrates 5y ago

Just another casualty of telemetry. I'm sure it doesn't help that Chrome killed its own encoding menu years ago.

Gigachad 5y ago
Isn't this issue fixed by websites fixing their encoding? If no browser is able to properly display the content, I imagine they will be fixed pretty quickly if they are at all maintained.
- roughly 5y ago
  Yes, but, they’re not. Huge swaths of the web are effectively abandoned, and will not get fixed.
  - Gigachad 5y ago
    Then the best option would be to copy out the useful content to new maintained sites and let the old ones die. Rather than keeping browsers tied to supporting long long dead features. UTF-8 has been standards for over a decade now. I think there has been plenty of migration time. More than flash had.
rjh29 5y ago
Chrome's position was "use an extension for this", but then users have to risk privacy by installing a third-party extension. What harm does having the option tucked somewhere in advanced settings do?
- sodality2 5y ago
  Every release they probably have to test that it still works. Not that I would ever justify removing something like this - it's clear that for a percentage of the userbase, this is necessary for them to browse. But I can see a product manager trying to cut "fat" if they see that 0.01% of people use it.
5y ago

kevingadd 5y ago

The telemetry-driven desire to redesign everything and remove lots of 'infrequently used' features is a real plague on software and it really sucks that it seems to have infested the web because of the amount of novelty-obsessed designers, PMs and engineers who move from one company to another. Lots of chrome, firefox and safari devs are people who previously worked on a different browser or a company with this culture and eventually your team has enough people like that to push through things like 'remove this important low-cost feature because only 1% of our billion users are impacted', as if 1% of a billion isn't a significant number of people. It used to be that even minor web platform changes wouldn't go through if they would break 0.1% of webpages, but we don't live in that world anymore.

Gigachad 5y ago
>but we don't live in that world anymore. And I'm glad. I'm sick of using software that has 2 trillion toolbars, menus and settings. I want something which sets the defaults to be pretty much what I want and makes the settings all the common things I might need to change. Telemetry driven design lets designers build what the majority like rather than the most vocal bug tracker using power users who want exactly the opposite of the average user.
- vvillena 5y ago
  Must be nice to always live in the middle of the bell curve.
  - Gigachad 5y ago
    The defaults aren’t always exactly what I want but I find it is preferable to just accept them as they are than to have a billion customisation options. I pick the best bit of software that works for me and then just change my workflow to fit it.
PragmaticPulp 5y ago
> The telemetry-driven desire to redesign everything and remove lots of 'infrequently used' features is a real plague on software I disagree. Maintaining features requires engineering overhead. It sucks to have to maintain features that barely anybody uses when you have a backlog of features that are in demand by a large number of your users.

dan-robertson 5y ago

It’s pretty unclear to me whether people use this feature much. Two possibilities seem likely to me: 1. Very few people use this feature but a few vocal users who are particularly good at navigating bugtracking bureaucracy do use it and care deeply. 2. Of all Firefox users, a small percentage speak languages for which there are many non-ascii letters and a relatively large amount of pre-Unicode content online, and of those users a small percentage read niche (eg non-facebook and old) sites with broken character encoding. Both situations could probably lead to few uses of the menu in telemetry. Perhaps we should care more about users in the second category as they are perhaps more correlated with other things like location whereas the first set may be more uniformly distributed across the population. I think other points are also being disregarded: - ordinary users may find the feature cryptic and confusing. Or at least undiscoverable - dealing with character encoding and supporting the feature is painful (though the will still deal with it under the hood). Mozilla have limited resources and they (and Firefox users) want the browser to be successful and sustainable, and this means focusing resources on things that will make the browser easier to adopt and less confusing rather than chasing the short-lived gains from appeasing a small and stagnant pool of ‘power users’.

baybal2 5y ago
> Mozilla have limited resources and they (and Firefox users) want the browser to be successful and sustainable, To begin with, near nobody uses Firefox today. Very succesful, and sustainable. I have to pour heavy sarcasm here. It's honestly justly needed. Firefox went from 50%+ to sub 1% under the current leadership.
- yoasif_ 5y ago
  > To begin with, near nobody uses Firefox today. Very succesful, and sustainable. 215 million MAU is still a lot of people: https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/user-activity
zozbot234 5y ago
> dealing with character encoding and supporting the feature is painful This feature is in no way "painful". It's a legitimate facility for dealing with non-Unicode, legacy content, and AFAICT it's not clear that it can be reliably replaced, even by a plugin. Any argument that the menu entry is "confusing" for most users can be addressed by hiding it behind an advanced preference setting, and I think this was done already.

nfriedly 5y ago

Could the menu be restored by an add-on?

butz 5y ago

There are still documents on the internet that use ASCII encoding, and while English speaking users doesn't have this problem, in some countries there were several different encoding systems used back in the past. While most believe that web content is resilient and we will still be able to browse old websites forever, recent web browser developments show otherwise. Please do not remove working features, especially if they have no effect on security or privacy.

wolverine876 5y ago
> There are still documents on the internet that use ASCII encoding Probably a great many because ASCII is a subset of UTF-8, and afaict is common for English when the site doesn't use more sophisticated typography such as curly quotes, em dashes, etc.
hsivonen 5y ago
> Please do not remove working features, especially if they have no effect on security or privacy. As it happens, part of what made the menu back end cause other changes to take more effort than they should have taken on their face were the security aspects of the menu in Firefox. Try https://hsivonen.com/test/moz/never-show-user-supplied-content-as-utf-16.htm16le https://hsivonen.com/test/moz/never-show-user-supplied-conte... in Firefox 90 (which had the menu) and in another browser that has a menu or an extension that adds a menu.

mnau 5y ago

Seems like clickbait title. I have seen this issue few times over the years and it was always a game of "Which encoding should I choose?" Having an automated way to solve this seems pretty good to me. Especially if the solution works pretty much 100% of time (especially for non-ASCII languages, like Japanese or Chinese) - see paragraph Document-length Accuracy in https://hsivonen.fi/chardetng/ https://hsivonen.fi/chardetng/

serentty 5y ago
East Asian languages are not the main concern here, actually. It’s the various single-byte Latin script encodings, which are incredibly similar. The issue of these being confused is addressed in the thread itself, where he says that such issues will not be fixed, because they are so confusable that fixing the detection of one means breaking the detection of another.
- mnau 5y ago
  I view it as a firefox fixing incorrectly written pages. It's nice that FF has a "fix-it" option, but ultimately, the author shoudl fix her site. The single-byte Latin scripts are semi legible, so that is enough for me. When a website breaks because 3rd party cookies go way of dodo or the page relies on ActiveX, shoudl Firefox provide an option to run it? I understand that some people have different opinion, but any page past 2005 should have encoding in order and older sites should fix their sites.
  - serentty 5y ago
    “Should” is not “is”. As a user, I run into broken sites regularly enough. I still want to be able to see them. I get the “let’s bury all broken legacy and hacks as quickly as possible” mindset, but when it comes to the issue of the main text of a page being unreadable, I think there is a high bar. > older sites should fix their sites. A lot of the reason they are considered “older sites” in the first place is because they are not heavily maintained. The thing to remember about the web is this: not all websites are “software” that is maintained. Some of them are static works similar to a movie, with static content. > The single-byte Latin scripts are semi legible, so that is enough for me. Did you see the example that I gave of Polish text? I am sorry, but if you feel like reading multiple paragraphs this way is bearable, then we will just be stuck in disagreement.

twhb 5y ago

I like this change. If a website misconfigures encoding and users have an encoding override menu, the result is 99% of users see a broken website, 1% know how to fix it. The menu doesn’t meaningfully improve the situation, but it does keep the people who could improve it from caring. Take that away and I predict 10%+ of currently misconfigured websites will get fixed for 100% of visitors, and that’s a big improvement over 100% of websites being fixed for 1% of visitors.

serentty 5y ago
So you think that making things miserable for the remaining 1% is good because it gets them to care, and therefore for the website to be fixed? I’m sorry, but most of the time when a website has a borked encoding, it is some website where I doubt I would ever get in contact with the webmaster anyway. It’s mostly small websites. But small websites are important to maintaining a healthy web where things are not dominated by a few big players. But perhaps most importantly, I have never agreed with any mindset of that encourages punishing some subset of users in order to get someone to change their ways, even more so if that isn’t even the users themselves.
- yoasif_ 5y ago
  > I’m sorry, but most of the time when a website has a borked encoding, it is some website where I doubt I would ever get in contact with the webmaster anyway. So you wouldn't bother to try to get the webmaster to fix the issue, you would just resolve it on your own, letting everyone else be miserable? > But perhaps most importantly, I have never agreed with any mindset of that encourages punishing some subset of users in order to get someone to change their ways, even more so if that isn’t even the users themselves. It sounds like you don't care if people are punished, as long as it isn't you.
  - serentty 5y ago
    > So you wouldn't bother to try to get the webmaster to fix the issue, you would just resolve it on your own, letting everyone else be miserable? I mean that I would not be able to. Huge swaths of the web are static content that was just put there and then just abandoned. The web will remain this way unless everything is moved onto SPA social networks, and I hope that we agree that that would not a good thing. > It sounds like you don't care if people are punished, as long as it isn't you. If the added benefit of causing people more inconvenience is that they push for problems to be fixed, I don’t think it’s fair to cause that inconvenience with the intent of using that to solve the problem. I think it’s a manipulative tactic. That is my idea of what “punishment” we’re talking about. What kind of punishment are you thinking of if you think I’m fine with punishment, and who is getting punished by it?

jcranmer 5y ago

I'm of a few different minds on this. First off, what needs to be acknowledged is that any website that requires you to override the charset is broken and the greatest part of your ire should be directed at the people who write and operate such websites for expecting that everybody else should be responsible for cleaning up their messes. Let's also acknowledge that avoiding this breakage has been possible for (checks notes) 22 years. Since this is very legacy stuff, at what point can we say that it is no longer the end consumer's job to deal with legacy cruft? The motivation for why a user-level override was ever necessary in the first place--the fact that once upon a time computers couldn't simultaneously represent content in different locales--is quite ancient history. I don't think an answer of "never" is compelling: why then aren't we complaining that browsers don't let us select EBCDIC or UTF-7 as an encoding (both of which were once supported, but have been dropped)? At the same time, this change does somehow still feel premature; dropping the ability to override charset encoding does feel very "WTF?" to me. But... if that's the case, then what can be done to hasten the moment where we live in a world where dropping this feature doesn't feel like that? (This makes me glad I work in compilers, where no one attacks you for deciding that some inputs are just too wrong to attempt to deal with it.)

zozbot234 5y ago
Unicode encoding was far from universally adopted in the early 2000s. Some legacy content will always exist, and the moment where dropping the feature is harmless is also the moment where there's no reason to drop it in the first place, since it only ever comes into play with broken legacy content; otherwise, it is entirely hidden and has no harmful effect whatsoever.
- jcranmer 5y ago
  Unicode encoding isn't what I was referring to--it was the ability to specify <META http-equiv="Content-Type" content="text/html; charset=EUC-JP"> (copied from HTML 4.01 specification, dated December 24, 1999). That might not be the first time it was possible to declare the charset of an HTML document within HTML itself, but I don't see any mention of it in HTML 3.2. Also, it's wrong to say that there's "no harmful effect whatsoever"--the ability to do charset overrides requires code to support it, and that code could force awkward compromises in (say) your HTTP layer. All features have costs, even if they're invisible.

TazeTSchnitzel 5y ago

This decision is offensively user-hostile. It breaks a critical feature for dealing with broken and legacy webpages (which, for better or worse, are never going to stop existing) without a good justification. Even for English-speaking users this sucks; there's lots of mailing list archives and the like that mishandle their encoding.

nightpool 5y ago

This headline is unfairly editorialized. What the Firefox dev said was that the automatic encoding detecter confusing some ISO/IEC 8859 encodings for each other is an intractable problem, and fixing one Latin text encoding being auto-detected for a different Latin text encoding is a very low-priority use of time. Also, removing the encoding override is completely separate from that, since it's (apparently) a very complicated feature to keep around that 1) breaks webpages (e.g. etrade) and 2) keeps confusing users and leading them think Firefox is worse at text encoding then Chrome. Regardless of your opinion of the change in question, these two statements are completely different from what the headline implies.

wolverine876 5y ago
> This headline is unfairly editorialized. I'm not sure why it doesn't follow HN guidelines to use the page title: "Replace the Text Encoding menu with a single override item Repair Text Encoding"
serentty 5y ago
He straight-up said that it’s not as much of an issue because it’s still legible because not all of the letters are borked. I don’t see how this is a misrepresentation of what he said. > Also, removing the encoding override is completely separate from that, It is not separate at all. If the detector cannot do it, and the ability for the user to do it is removed, these issues are intrinsically linked. Sure, there are the other issues you mentioned around keeping it around, but you can’t act like these issues can be considered separately.
zozbot234 5y ago
The etrade breakage was a failure of the previous detector, which expected an encoding declaration within the first few page KB and introduced a page reload otherwise. Switching the current encoding manually does not reload the page per se.

wolverine876 5y ago

The title, by HN guidelines, is "Replace the Text Encoding menu with a single override item Repair Text Encoding"

rtpg 5y ago

All the discussion in this thread and in the bugzilla thread feel very odd to me. Like arguing that it's not very used, is a maintenance burden.... and it was already hidden off in a submenu! I'm, of course, glad that the guesser can work more effectively. But isn't this a bit different from something like FTP (which can introduce nasty security bugs?)? And you still end up with "change encoding of page" with the guesser, so it's like... it's not like you are removing some lower-level functionality. It's unfortunate because yeah people mention Flash getting removed in a similar tone, but at least with Flash there are things like ruffle that can pick up the slack. I don't really want to have to install a plugin for the couple of times I land on the page that miss this encoding, but when I do hit the page I'll be frustrated. Of course the original sin here was browsers not throwing warnings or something on every page without the encoding set, and having users become the people who complain to webmasters about it like... 15 years ago or whatever. But meanwhile we had a really _fine_ place for the encoding picker. Maybe first click is "Guess", second reveals the menu... or something. I don't want to hold too much malice towards the devs here cuz getting yelled at by all the internet sucks. But I hope there can be some resolution here that doesn't lead to massive burnout. I will be very happy if there is some backup solution inserted (even if it's behind a conf flag), and I would see this as an overall positive to FF if that happens (listening to feedback!)

emiliocobos 5y ago

To people thinking that this was removed gratuitously please read the bug and linked bugs carefully. This got in the way of implementing various improvements to <meta charset> and interop improvements to XHR.

xg15 5y ago

In addition to the concrete case at hand, this is another instance where explicit user abilities are taken away and replaced by an algorithm. I really hope the tech industry will see the light at some point and stop this trend.

SSLy 5y ago

I use it few times a year to fix websites sending ISO-8859-2 (or Windows-1250) and the guess mode worked. As an user affected by this change, it is an improvement (less need to click around trying 5 different encodings)

serentty 5y ago
I am definitely not against the addition of the guesser. It was added a bit before the encoding menu was removed, though, so there was a time when they were both there. I mainly wanted to draw attention to the removal of the menu.

nneonneo 5y ago

After all the heated discussion here, I decided to make this into a plugin: https://addons.mozilla.org/en-US/firefox/addon/override-text-encoding/ https://addons.mozilla.org/en-US/firefox/addon/override-text... It's pretty bare-bones and probably has some bugs (after all, I hacked it up in a few hours...) but it does what it says on the tin: it lets you set the text encoding for any webpage manually, just like the good ol' Encoding menu.

serentty 5y ago
I hugely appreciate this! I will definitely install it.