15 ms·

How about assume utf-8, and if someone has some binary file they'd rather a program interpret as some other format, they turn it into utf-8 using a standalone p

How about assume utf-8, and if someone has some binary file they'd rather a program interpret as some other format, they turn it into utf-8 using a standalone program first. Instead of burning this guess-what-bytes-they-might-like nonsense into all the software. We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

zarzavat 2y ago
Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players. It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.
- fl7305 2y ago
  > It’s much easier to tell the people with old cassette tapes to rip them I assume you mean "rip them", as in transcode to a different format? In that case, you need a tool that takes the old input format(s) and convert them to the new format. For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8. Isn't the point of the article to describe how an engineer would create such a tool?
  - 0xffff2 2y ago
    A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
    - fl7305 2y ago
      > should be asking the user to select explicit input and output formats It depends on the requirements. If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
    - tempaccount1234 2y ago
      Unless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive. It becomes easier to just use https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ on the input.
  - jcranmer 2y ago
    > Isn't the point of the article to describe how an engineer would create such a tool? Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job. If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.
    - fl7305 2y ago
      I agree 100% on the technical issues. But the point of the article is not the same thing as how well it achieved its goals.
    - javajosh 2y ago
      >universal charset detection isn't really a thing that works all that well. This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
      - fl7305 2y ago
        Yeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text. In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|". So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
        extraduder_ire 2y ago
        I'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
  - taneq 2y ago
    No you don’t, they (the ones with the piles of old mixtapes) do.
    - fl7305 2y ago
      I'm an engineer. I'm sometimes hired by they to create tools like that.
- gwervc 2y ago
  UTF-8 uses 50% more bytes than UFT-16 to encode Chinese or Japanese texts.
  - bawolff 2y ago
    Which basically never matters and in any case where it actually does, gzip will make it equal again.
    - deleted 2y ago
      [deleted]
    - iraqmtpizza 2y ago
      zip-then-encrypt leaks information about the plaintext. if it's life or death, better not to compress at all
      - tnmom 2y ago
        Huh, never heard that before. Does it leak more information than just encrypting without zipping? Struggling to imagine how this attack works.
        iraqmtpizza 2y ago
        Encrypting without zipping doesn't leak any information about the content. You can't rule out certain byte sequences (other than by total length) just by looking at the ciphertext length. If "oui" compresses to two bytes and "non" compresses to one byte, and then you go over them with a stream cipher, which is which: A: ; B: *&
        lisper 2y ago
        This has nothing to do with compression. If you use "yes" and "no" instead of "oui" and "non" (which just happen to be three characters each) and you compress "yes" to "T" and "no" to "F" then the uncompressed text will be the leaky one.
amarshall 2y ago
> some software probably does. Browsers do, kind of https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-unknown-mime-type https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-...
fl7305 2y ago
> they turn it into utf-8 using a standalone program first I took the article to be for people who would be writing that "standalone program"? I have certainly been in a position where I was the person who had to deal with input text files with unknown encodings. There was no-one else to hand off the problem to.
- 0xffff2 2y ago
  In that case, you should be explicitly asking the user what the input format is.
  - PaulDavisThe1st 2y ago
    "I dunno, got it from a friend on a USB stick"
  - fl7305 2y ago
    > you should be explicitly asking the user what the input format is. Me: "I'll make the program stop and ask the user." Customer: "No. We have 10 million files. Start by using heuristics for batch processing."
    - Spivak 2y ago
      And it's usually worse, we have 10 million files coming from sources neither we nor our users control. Or "what do you mean ask them, they barely know what a jpeg is?!"
      - MichaelZuo 2y ago
        There's someone that must be paying your invoices? Charge them extra for the extra work that needs to be done to sort things out.
        Spivak 2y ago
        ??? The work to use one of the many encoding guessing tools https://github.com/Joungkyun/libchardet https://github.com/Joungkyun/libchardet and then get it correct for almost every document? You just look bad if you can't do what every other software is able to do. Charging for it takes that to another level.
ryandrake 2y ago
> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does. What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser." 1: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
- jerf 2y ago
  We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. This is in some sense relatively recent, so word is still getting around, as the programming world does not move as quickly as it fancies itself to. If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.
  - tomsmeding 2y ago
    Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec? Or is it genuinely unexplainable without experiencing something first-hand?
    - fl7305 2y ago
      >>> the best software accepts many formats and "deals with it," >> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. >> If you'd like to see why, read the HTML 5 parsing portion of the spec. > Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?
ezoe 2y ago
I doubt you can handle UTF-8 properly with that attitude. The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows. It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you. Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC. These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.
- dgellow 2y ago
  I’ve been sharing it multiple times but I love it: WTF-16 spec https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
- int_19h 2y ago
  > If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you. For a library, yeah.
  - ezoe 2y ago
    Don't be surprised such UTF-8 locale program will break so bad on Windows with default language set to Japanese.
    - int_19h 2y ago
      Why would it break? If you just assume that the system codepage is UTF-8, then sure. If you specifically say in your manifest that you want UTF-8, then Windows (10+) will give you UTF-8 regardless of which locale it is: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...
SuperNinKenDo 2y ago
Not every encoding can make a round trip through Unicode without you writing ad hoc handling code for every single one. There's a number of reasons some of these are still in use and Unicode destroying information is one of them.
998244353 2y ago
Non-technical users don't want to do that, and won't understand any of that. That's the unfortunate reality of developing software for people. If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.
- unclebucknasty 2y ago
  >A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding... What if you just gave them instructions?
  - nkrisc 2y ago
    They’ll probably switch to the application that does it for them and just works, instead of the one telling them to do something they don’t really understand.
    - unclebucknasty 2y ago
      Selecting UTF-8 from a dropdown on the export doesn't seem too onerous an ask. If that's the differentiator between you and your competitors, then you might have bigger problems.
      - nkrisc 2y ago
        I have sat in on and observed many in-person usability sessions with various applications and websites. In can tell you right now anyone reading this site is in a completely different universe of tech competency than the general public, and even professionals who aren’t tech-focused. You would have lost many of them simply with then jumble of letters “UTF-8”.
kelnos 2y ago
Solutions that require lots of unrelated people to start doing something a different way are not really solutions.
thaumasiotes 2y ago
> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does. Browsers used to have a menu option to choose the encoding you wanted to use to decode the page. In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this. They seem to be in the process of disabling that option too: > Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported. ( https://support.mozilla.org/en-US/kb/text-encoding-no-longer-available-firefox-menu https://support.mozilla.org/en-US/kb/text-encoding-no-longer... ) This note is logical gibberish; encoding isn't something that has to be supported by the page. Decoding is a choice by the browser!
- n_plus_1_acc 2y ago
  Probably because most websites now send a correct encoding header or meta Tag, so the user changing can only make it wrong. (Assuming no encoding header is wrong, which happens in reality)
  - layer8 2y ago
    It does happen a lot that old text content in non-UTF-8 encoding is mistakenly served explicitly marked as UTF-8. It is precisely in such circumstances that the Repair Text option is useful.
- shiomiru 2y ago
  It seems the decision was made in the name of security: https://hsivonen.fi/no-encoding-menu/ https://hsivonen.fi/no-encoding-menu/ > Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run. > Elaborate UI surface for a niche feature risks the whole feature getting removed > Telemetry [...] suggested that users aren’t that good at choosing correctly manually. In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)

zarzavat 2y ago

Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players. It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.

fl7305 2y ago
> It’s much easier to tell the people with old cassette tapes to rip them I assume you mean "rip them", as in transcode to a different format? In that case, you need a tool that takes the old input format(s) and convert them to the new format. For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8. Isn't the point of the article to describe how an engineer would create such a tool?
- 0xffff2 2y ago
  A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
  - fl7305 2y ago
    > should be asking the user to select explicit input and output formats It depends on the requirements. If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
  - tempaccount1234 2y ago
    Unless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive. It becomes easier to just use https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ on the input.
- jcranmer 2y ago
  > Isn't the point of the article to describe how an engineer would create such a tool? Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job. If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.
  - fl7305 2y ago
    I agree 100% on the technical issues. But the point of the article is not the same thing as how well it achieved its goals.
  - javajosh 2y ago
    >universal charset detection isn't really a thing that works all that well. This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
    - fl7305 2y ago
      Yeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text. In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|". So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
      - extraduder_ire 2y ago
        I'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
- taneq 2y ago
  No you don’t, they (the ones with the piles of old mixtapes) do.
  - fl7305 2y ago
    I'm an engineer. I'm sometimes hired by they to create tools like that.
gwervc 2y ago
UTF-8 uses 50% more bytes than UFT-16 to encode Chinese or Japanese texts.
- bawolff 2y ago
  Which basically never matters and in any case where it actually does, gzip will make it equal again.
  - deleted 2y ago
    [deleted]
  - iraqmtpizza 2y ago
    zip-then-encrypt leaks information about the plaintext. if it's life or death, better not to compress at all
    - tnmom 2y ago
      Huh, never heard that before. Does it leak more information than just encrypting without zipping? Struggling to imagine how this attack works.
      - iraqmtpizza 2y ago
        Encrypting without zipping doesn't leak any information about the content. You can't rule out certain byte sequences (other than by total length) just by looking at the ciphertext length. If "oui" compresses to two bytes and "non" compresses to one byte, and then you go over them with a stream cipher, which is which: A: ; B: *&
        lisper 2y ago
        This has nothing to do with compression. If you use "yes" and "no" instead of "oui" and "non" (which just happen to be three characters each) and you compress "yes" to "T" and "no" to "F" then the uncompressed text will be the leaky one.

amarshall 2y ago

> some software probably does. Browsers do, kind of https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-unknown-mime-type https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-...

fl7305 2y ago

> they turn it into utf-8 using a standalone program first I took the article to be for people who would be writing that "standalone program"? I have certainly been in a position where I was the person who had to deal with input text files with unknown encodings. There was no-one else to hand off the problem to.

0xffff2 2y ago
In that case, you should be explicitly asking the user what the input format is.
- PaulDavisThe1st 2y ago
  "I dunno, got it from a friend on a USB stick"
- fl7305 2y ago
  > you should be explicitly asking the user what the input format is. Me: "I'll make the program stop and ask the user." Customer: "No. We have 10 million files. Start by using heuristics for batch processing."
  - Spivak 2y ago
    And it's usually worse, we have 10 million files coming from sources neither we nor our users control. Or "what do you mean ask them, they barely know what a jpeg is?!"
    - MichaelZuo 2y ago
      There's someone that must be paying your invoices? Charge them extra for the extra work that needs to be done to sort things out.
      - Spivak 2y ago
        ??? The work to use one of the many encoding guessing tools https://github.com/Joungkyun/libchardet https://github.com/Joungkyun/libchardet and then get it correct for almost every document? You just look bad if you can't do what every other software is able to do. Charging for it takes that to another level.

ryandrake 2y ago

> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does. What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser." 1: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle

jerf 2y ago
We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. This is in some sense relatively recent, so word is still getting around, as the programming world does not move as quickly as it fancies itself to. If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.
- tomsmeding 2y ago
  Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec? Or is it genuinely unexplainable without experiencing something first-hand?
  - fl7305 2y ago
    >>> the best software accepts many formats and "deals with it," >> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. >> If you'd like to see why, read the HTML 5 parsing portion of the spec. > Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?

ezoe 2y ago

I doubt you can handle UTF-8 properly with that attitude. The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows. It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you. Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC. These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.

dgellow 2y ago
I’ve been sharing it multiple times but I love it: WTF-16 spec https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
int_19h 2y ago
> If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you. For a library, yeah.
- ezoe 2y ago
  Don't be surprised such UTF-8 locale program will break so bad on Windows with default language set to Japanese.
  - int_19h 2y ago
    Why would it break? If you just assume that the system codepage is UTF-8, then sure. If you specifically say in your manifest that you want UTF-8, then Windows (10+) will give you UTF-8 regardless of which locale it is: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...

SuperNinKenDo 2y ago

Not every encoding can make a round trip through Unicode without you writing ad hoc handling code for every single one. There's a number of reasons some of these are still in use and Unicode destroying information is one of them.

998244353 2y ago

Non-technical users don't want to do that, and won't understand any of that. That's the unfortunate reality of developing software for people. If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.

unclebucknasty 2y ago
>A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding... What if you just gave them instructions?
- nkrisc 2y ago
  They’ll probably switch to the application that does it for them and just works, instead of the one telling them to do something they don’t really understand.
  - unclebucknasty 2y ago
    Selecting UTF-8 from a dropdown on the export doesn't seem too onerous an ask. If that's the differentiator between you and your competitors, then you might have bigger problems.
    - nkrisc 2y ago
      I have sat in on and observed many in-person usability sessions with various applications and websites. In can tell you right now anyone reading this site is in a completely different universe of tech competency than the general public, and even professionals who aren’t tech-focused. You would have lost many of them simply with then jumble of letters “UTF-8”.

kelnos 2y ago

Solutions that require lots of unrelated people to start doing something a different way are not really solutions.

thaumasiotes 2y ago

> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does. Browsers used to have a menu option to choose the encoding you wanted to use to decode the page. In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this. They seem to be in the process of disabling that option too: > Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported. ( https://support.mozilla.org/en-US/kb/text-encoding-no-longer-available-firefox-menu https://support.mozilla.org/en-US/kb/text-encoding-no-longer... ) This note is logical gibberish; encoding isn't something that has to be supported by the page. Decoding is a choice by the browser!

n_plus_1_acc 2y ago
Probably because most websites now send a correct encoding header or meta Tag, so the user changing can only make it wrong. (Assuming no encoding header is wrong, which happens in reality)
- layer8 2y ago
  It does happen a lot that old text content in non-UTF-8 encoding is mistakenly served explicitly marked as UTF-8. It is precisely in such circumstances that the Repair Text option is useful.
shiomiru 2y ago
It seems the decision was made in the name of security: https://hsivonen.fi/no-encoding-menu/ https://hsivonen.fi/no-encoding-menu/ > Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run. > Elaborate UI surface for a niche feature risks the whole feature getting removed > Telemetry [...] suggested that users aren’t that good at choosing correctly manually. In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)