3 ms·

Seems like clickbait title. I have seen this issue few times over the years and it was always a game of "Which encoding should I choose?" Having an automated w

by mnau 5y ago

Seems like clickbait title. I have seen this issue few times over the years and it was always a game of "Which encoding should I choose?" Having an automated way to solve this seems pretty good to me. Especially if the solution works pretty much 100% of time (especially for non-ASCII languages, like Japanese or Chinese) - see paragraph Document-length Accuracy in https://hsivonen.fi/chardetng/ https://hsivonen.fi/chardetng/

serentty 5y ago
East Asian languages are not the main concern here, actually. It’s the various single-byte Latin script encodings, which are incredibly similar. The issue of these being confused is addressed in the thread itself, where he says that such issues will not be fixed, because they are so confusable that fixing the detection of one means breaking the detection of another.
- mnau 5y ago
  I view it as a firefox fixing incorrectly written pages. It's nice that FF has a "fix-it" option, but ultimately, the author shoudl fix her site. The single-byte Latin scripts are semi legible, so that is enough for me. When a website breaks because 3rd party cookies go way of dodo or the page relies on ActiveX, shoudl Firefox provide an option to run it? I understand that some people have different opinion, but any page past 2005 should have encoding in order and older sites should fix their sites.
  - serentty 5y ago
    “Should” is not “is”. As a user, I run into broken sites regularly enough. I still want to be able to see them. I get the “let’s bury all broken legacy and hacks as quickly as possible” mindset, but when it comes to the issue of the main text of a page being unreadable, I think there is a high bar. > older sites should fix their sites. A lot of the reason they are considered “older sites” in the first place is because they are not heavily maintained. The thing to remember about the web is this: not all websites are “software” that is maintained. Some of them are static works similar to a movie, with static content. > The single-byte Latin scripts are semi legible, so that is enough for me. Did you see the example that I gave of Polish text? I am sorry, but if you feel like reading multiple paragraphs this way is bearable, then we will just be stuck in disagreement.
    - mnau 5y ago
      I guess we are stuck at disagreement. I did see the text. If it bothered me that much, I would write a proxy that passed HTML through iconv tool before serving it to user (with a dictionary to remember setting for each site). But more likely, I would stop visiting such site or complain to the owner. Situation doesn't get better by covering up errors.
- hsivonen 5y ago
  As a matter of usage and severity of failure mode, East Asian languages are very much the main concern. Japanese specifically is the primary motivator for keeping the feature in any form. If there's more elaborate UI surface in order to address Latin-script cases where autodetection fails (I estimate 1% failure rate for Polish), it risks getting the whole thing removed completely even for Japanese users (as happened with the hamburger menu entry point in Firefox when the hamburger menu was rewritten, as happened in Chrome previously, and as happened with Safari for iOS in the sense of the menu never making it to iOS).
Jach 5y ago
The old menu did have an automatic automated way to solve this. I'm not sure if it goes all the way back to Firefox 1.0, but it's pretty old. (At least Waterfox Classic exists and you can install it and see the menu there.) That it wasn't very good (as detailed in the post) suggests replacing it with a better version, but this goes beyond that in removing the ability to pick something specifically when the automatic detection is wrong.

by mnau 5y ago

serentty 5y ago

East Asian languages are not the main concern here, actually. It’s the various single-byte Latin script encodings, which are incredibly similar. The issue of these being confused is addressed in the thread itself, where he says that such issues will not be fixed, because they are so confusable that fixing the detection of one means breaking the detection of another.

mnau 5y ago
I view it as a firefox fixing incorrectly written pages. It's nice that FF has a "fix-it" option, but ultimately, the author shoudl fix her site. The single-byte Latin scripts are semi legible, so that is enough for me. When a website breaks because 3rd party cookies go way of dodo or the page relies on ActiveX, shoudl Firefox provide an option to run it? I understand that some people have different opinion, but any page past 2005 should have encoding in order and older sites should fix their sites.
- serentty 5y ago
  “Should” is not “is”. As a user, I run into broken sites regularly enough. I still want to be able to see them. I get the “let’s bury all broken legacy and hacks as quickly as possible” mindset, but when it comes to the issue of the main text of a page being unreadable, I think there is a high bar. > older sites should fix their sites. A lot of the reason they are considered “older sites” in the first place is because they are not heavily maintained. The thing to remember about the web is this: not all websites are “software” that is maintained. Some of them are static works similar to a movie, with static content. > The single-byte Latin scripts are semi legible, so that is enough for me. Did you see the example that I gave of Polish text? I am sorry, but if you feel like reading multiple paragraphs this way is bearable, then we will just be stuck in disagreement.
  - mnau 5y ago
    I guess we are stuck at disagreement. I did see the text. If it bothered me that much, I would write a proxy that passed HTML through iconv tool before serving it to user (with a dictionary to remember setting for each site). But more likely, I would stop visiting such site or complain to the owner. Situation doesn't get better by covering up errors.
hsivonen 5y ago
As a matter of usage and severity of failure mode, East Asian languages are very much the main concern. Japanese specifically is the primary motivator for keeping the feature in any form. If there's more elaborate UI surface in order to address Latin-script cases where autodetection fails (I estimate 1% failure rate for Polish), it risks getting the whole thing removed completely even for Japanese users (as happened with the hamburger menu entry point in Firefox when the hamburger menu was rewritten, as happened in Chrome previously, and as happened with Safari for iOS in the sense of the menu never making it to iOS).

Jach 5y ago

The old menu did have an automatic automated way to solve this. I'm not sure if it goes all the way back to Firefox 1.0, but it's pretty old. (At least Waterfox Classic exists and you can install it and see the menu there.) That it wasn't very good (as detailed in the post) suggests replacing it with a better version, but this goes beyond that in removing the ability to pick something specifically when the automatic detection is wrong.