19 ms·

You can't just assume UTF-8

195 points by calpaterson 2y ago

bhaney 2y ago
I'm just gonna assume UTF-8
- calpaterson 2y ago
  榥\ue0af侬펭懃䒥亷
  - bhaney 2y ago
    And a good day to you too, my friend whose input I'm going to discard
    - eqvinox 2y ago
      It's garbage anyway, which you can (non-reliably) guess by there being a Korean character in the middle of C/J kanji. (Kanji are not completely gone from Korean, but mostly.)
      - calpaterson 2y ago
        It's mojibake for "probably a bad idea" [in Chinese]
- duskwuff 2y ago
  I'm disappointed that the article doesn't discuss this in more detail. Most byte sequences are not valid UTF-8. If you can decode a message as UTF-8 with no errors, that is almost certainly the correct encoding to use; it's extremely unlikely that some text in another encoding just happened to be perfectly valid as UTF-8. (The converse is not true; most 8-bit text encodings will happily decode UTF-8 sequences to nonsense strings like ðŸš©.) If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding. But that should be a fallback, not the first thing you try.
  - actionfromafar 2y ago
    Not extremely unlikely. Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII.
    - PaulDavisThe1st 2y ago
      such encodings are also UTF-8 then, are they not?
      - tsimionescu 2y ago
        I think there are a bunch of encodings that just repurposed a few ASCII characters as different characters - someone on this page was giving the example of some Swedish encoding where {}| were replaced with three accented Swedish letters. There are probably a bunch of others. In those cases, the text will decode fine as UTF-8, but it will display the wrong thing.
    - srj 2y ago
      A distinction without a difference?
      - actionfromafar 2y ago
        I meant, some messages will decode fine as UTF-8, but in some other messages there may be letters which don't fit in 7 bits. So some simple testing, especially with English words, will show it to work fine. But as soon as a non-7-bit characters creeps, it will stop working fine.
    - duskwuff 2y ago
      > Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII. At which point the message is effectively ASCII. UTF-8 is a superset of ASCII, so "decoding" ASCII as UTF-8 is fine. (Yes, I know there are some Japanese text encodings where 0x5c is decoded as "¥" instead of "\". But they're sometimes treated as backslashes even though they look like ¥ symbols so handling them "correctly" is complicated.)
- klysm 2y ago
  You can’t just do that! /s
jujube3 2y ago
Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is no need to have dozens of incompatible systems.
- qwertox 2y ago
  You can't assume it if you're handed a random sample of text files.
  - duskwuff 2y ago
    You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements. (Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
    - ranger_danger 2y ago
      Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.
      - duskwuff 2y ago
        Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
        ranger_danger 2y ago
        I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
pronoiac 2y ago
Archive copy: https://web.archive.org/web/20240429061925/https://csvbase.com/blog/9 https://web.archive.org/web/20240429061925/https://csvbase.c...
o11c 2y ago
Default UTF-8 is better than the linked suggestion of using a heuristic, but failing catastrophic when old data is encountered is unacceptable. There must be a fallback. (Note that the heuristic for "is this intended to be UTF-8" is pretty reliable, but most other encoding-detection heuristics are very bad quality)
dandigangi 2y ago
Except I can
Veserv 2y ago
Off-topic, but the bit numbering convention is deliciously confusing. Little-endian bytes (lowest byte is leftmost) and big-endian bits (bits contributing less numerical value are rightmost) are normal, but the bits are referenced/numbered little-endian (first bit is leftmost even though it contributes the most numerical value). When I first read the numbering convention I thought it was going to be a breath of fresh air of someone using the much more sane, but non-standard, little-endian bits with little-endian bytes, but it was actually another layered twist. Hopefully someday English can write numbers little-endian, which is objectively superior, and do away with this whole mess.
- kstrauser 2y ago
  > Hopefully someday English can write numbers little-endian, which is objectively superior Upon reading this, I threw my laptop out the window.
  - Veserv 2y ago
    It actually is if we did not need to consider historical baggage. Especially in programming where we already use in-band encoding like 0x to denote a hex string or 0b to denote a binary string. I like using 1{s}, e.g. 1x to denote a little-endian hex string and 1b to denote a little-endian binary string, to denote little-endian encoding. But, even ignoring programming, it is still better in normal use. The Arabic language got it right writing little-endian (Arabic numbers are written the same, but Arabic is a right-to-left language so it is actually little-endian), and the European languages just stole it stupid by copying the form instead of the function.
    - filmor 2y ago
      Arabian numerals are derived from Indian scripts, which are written left to right, yet the order of digits is already the same we have today.
      - pezezin 2y ago
        Not to mention that in most (all?) spoken languages numbers are big-endian.
bandyaboot 2y ago
> In the most popular character encoding, UTF-8, character number 65 ("A") is written: > 01000001 > Only the second and final bits are 1, or "on". Isn’t it more accurate to say that the first and penultimate bits are 1, or “on”?
- fl7305 2y ago
  It depends on whether your bit numbering is like x86 (your description), or PowerPC (left most bit is 0).
  - duskwuff 2y ago
    Basically everyone uses x86 bit numbering. It has the pleasant property that the place value of every bit is always 2^n (or -2^n for a sign bit), and zero-extending a value doesn't change the numbering of its bits.
    - fl7305 2y ago
      Sure, it is by far the industry standard. It works much better for handling discrete integers. Once you get into bitfield instructions it is nice to have bit 0 be the "left most bit".
    - mark-r 2y ago
      It's a mistake to call it x86 bit numbering, it's way older than that.
  - bandyaboot 2y ago
    The more I thought it through, even assuming x86, I guess there’s just no “correct” way to casually reference bit positions when we read them in the opposite order from the machine. Are they being referenced from the perspective of a human consumer of text, or the machine’s perspective as a consumer of bits? If I were writing that content, I’d have a difficult time deciding on which to use. If I were writing for a lay person, referencing left-to-right seems obvious, but in this case where the audience is primarily developers, it becomes much less obvious.
vitaut 2y ago
This is so spectacularly outdated. KOI-8 has been dead for ages.
JonChesterfield 2y ago
How about assume utf-8, and if someone has some binary file they'd rather a program interpret as some other format, they turn it into utf-8 using a standalone program first. Instead of burning this guess-what-bytes-they-might-like nonsense into all the software. We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.
- zarzavat 2y ago
  Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players. It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.
  - fl7305 2y ago
    > It’s much easier to tell the people with old cassette tapes to rip them I assume you mean "rip them", as in transcode to a different format? In that case, you need a tool that takes the old input format(s) and convert them to the new format. For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8. Isn't the point of the article to describe how an engineer would create such a tool?
    - 0xffff2 2y ago
      A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
      - fl7305 2y ago
        > should be asking the user to select explicit input and output formats It depends on the requirements. If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
dublin 2y ago
Make your life easy. Assume 7-bit ASCII. No one needs all those other characters, anyway...
- AlienRobot 2y ago
  Do we really need 128 permutations just to express an alphabet of 26 letters? I think we should use a 4 bit encoding. 0 - NUL 1-7 - aeiouwy 8 - space 9-12 - rst 13-15 - modifiers When modifier bits are set, the values of the next half-byte change to represent the rest of the alphabet, numbers, symbols, etc. depending on the bits set.
  - apantel 2y ago
    R S T L N E
  - Dylan16807 2y ago
    I like it, but what if we use 2 to pick symbols, 3 to pick numbers, and 4 and 5 to pick capital letters.
- jonathankoren 2y ago
  As someone who has repeatedly had to deal with Unicode nonsense; I wholeheartedly agree. Also, you don’t need accents. You just need to know how to read and have context. See: live and live, read and read, etc.
drdaeman 2y ago
Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024" then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun. Or, you know, just say "nah, I can, those ancient stuff don't matter (outside of obligatory exceptions, like software archeology) anymore." If someone wants to feed me a KOI8-R or JIS X 0201 CSV heirloom, they should convert it into something modern first.
- deleted 2y ago
  [deleted]
- skissane 2y ago
  > Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024" I have a hobby interest in IBM mainframes and IBM i, so yes to EBCDIC for me. (I have encountered them professionally too, but only to a very limited extent.) In practice, I find looking for 0x40 (EBCDIC space) a useful heuristic. Even in binary files, since many mainframe data structures are fixed length space padded. > then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun. Actual use of UTF-EBCDIC, while not nonexistent, has always been extremely rare. A person could spend an entire career dedicated to IBM mainframes and never encounter it EBCDIK, at first I wondered if that was a joke, now I realise it is a name used for non-IBM Japanese EBCDIC code pages. Again, something one can spend a whole career in mainframes and never encounter – if one never works in Japan, if one works for a vendor whose products aren't sold in Japan, probably even if you work for a vendor whose products are sold in Japan but only to IBM sites (as opposed to Fujitsu/Hitachi sites)
smeagull 2y ago
I absolutely can. If it's not UTF-8, I assume it's worthless.
AlienRobot 2y ago
"You can't assume a 32 bit integer starts from 0"
Karellen 2y ago
Don't worry, I never assume UTF-8. I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, and I get non-UTF-8 input, I'm fine bailing on that with a "malformed input - please correct" error.
- lmm 2y ago
  [flagged]
  - pavel_lishin 2y ago
    Does the entirety of Japan shun utf8?
    - koito17 2y ago
      Many Japanese websites have migrated from Shift-JIS to UTF-8, but this still ignores the fact that e.g. television captioning uses special characters[1] that are not found in UTF-8 or Shift-JIS. Windows itself has a habit of using its own Windows-932 encoding, which frequently causes problems in the Unix software I use. (e.g. Emacs fails at auto-detecting this format, and instructing Emacs to use Shift-JIS will result in decoding issues) [1] section 1.2 in https://www.fontucom.com/pdf/AFSARIBR-Map.pdf https://www.fontucom.com/pdf/AFSARIBR-Map.pdf
      - cheeze 2y ago
        If Emacs fails to do it correctly, end developers are screwed.
        koito17 2y ago
        Windows-932 is the text encoding I dread the most. I wish literally everything could use Unicode, but we're not quite there yet. There's a reason encoding-japanese[1] on NPM has nearly 1 million weekly downloads. Unfortunately, I have to use it in one of my React Native applications since the one of servers I speak to returns text encoded as Shift-JIS and Hermes does not implement JavaScript's TextDecoder API. Shift-JIS is marginally better than the Windows code page, but I'd really prefer UTF-8. [1] https://www.npmjs.com/package/encoding-japanese https://www.npmjs.com/package/encoding-japanese
groestl 2y ago
I will assume it, I will enforce it where I can, and I will fight tooth and nail should push come to shove. I got 99 problems, but charsets aint one of them.
zadokshi 2y ago
Better to assume UTF8 and fail with a clear message/warning. Sure you can offer to guess to help the end user if it fails, but as other people have pointed out, it’s been standard for a long time now. Even python caved and accepted it as the default: https://peps.python.org/pep-0686/ https://peps.python.org/pep-0686/
koito17 2y ago
The comments in this thread are a bit amusing. I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world". Many Japanese documents and sites still use Shift JIS. Windows has this strange Windows-932 format that you will frequently encounter in CUE files outputted by some CD ripping software. ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. These special characters are mostly icons used in traffic and weather reports, but transcoding to UTF-8 still causes trouble with these icons.
- bongodongobob 2y ago
  Most people on this site probably live in the world where everything is done in English. That's the norm for the vast majority of businesses and people in the US.
  - fl7305 2y ago
    Even for those people, there is still a ton of old text files in Windows-1252 etc floating around. You can choose to never work on projects where you have to deal with files like that. But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.
    - pixl97 2y ago
      Why would you write a tool that does that instead of just digging up one that's already written? This sounds like the folly of writing ones own encryption library.
      - fl7305 2y ago
        Which library do you pick? Which ones are good? Bad? What are the common pitfalls that the library must deal with? How do you know how to evaluate them if all you know is "not UTF-8? Abort with error" ?
norir 2y ago
If it's turtles all the way down and at every level you use utf-8, it's hard to see how any input with a different encoding (for the same underlying text) will not be detected before any unintended side effects were invoked. At this point, I don't see any sufficiently good reason to not use utf-8 exclusively in any new system. Conversions to and from other encodings would only be done at well defined boundaries when I'm calling into dependencies that require non utf-8 input for whatever reason.
kstrauser 2y ago
If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, and the sender doesn't like the results, that's on them. I'm willing to spend approximately zero effort trying to guess what nonstandard thing they're trying to send me unless they're paying me or my company a whole lot of money, in which case I'll convert it to UTC upon import and continue on from there. Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8. If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.
- kccqzy 2y ago
  > For instance, if it were the case that it did a bad job of encoding Mandarin I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.
  - PeterisP 2y ago
    A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.
jcranmer 2y ago
I haven't seen discussion of this point yet, but the post completely fails to provide any data to back up its assertion that charset detection heuristics works, because the feedback I've seen from people who actually work with charsets is that it largely doesn't (especially if you're based on naive one-byte frequency analysis). Okay, sure, it works if you want to distinguish between KOI8-R and Windows-1252, but what about Windows-1252 and Windows-1257? See for example this effort in building a universal charset detector in Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1551276 https://bugzilla.mozilla.org/show_bug.cgi?id=1551276
- toast0 2y ago
  I've done some charset detection, although it's been a while. Heuristics kind of work for somethings --- I'm a big fan of if it's decodable as utf-8, it's probably utf-8, unless there's zero bytes (in most text). If there's a lot of zero bytes, maybe it's UCS-2 or UTF-16, and you can try to figure out the byte order and if it decodes as utf-16. If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.
- renhanxue 2y ago
  Yeah. The fantastic python library ftfy ("fixes text for you", https://ftfy.readthedocs.io/en/latest/index.html https://ftfy.readthedocs.io/en/latest/index.html), designed to fix mangled Unicode (mojibake, of many different varieties), mentions in its docs that heuristic encoding guessers are the cause of many of the problems ftfy is designed to fix. It's magical, by the way. It has an entire section of its documentation dedicated to why you should not use heuristic charset detection: https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-chardet https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-c...
  - torstenvl 2y ago
    That section explains why not to use a specific naive charset detection library that doesn't have a strong prior for UTF-8. There's no basis for extrapolating that further.
mgaunard 2y ago
There is a pretty successful world language standard: English.
- bawolff 2y ago
  Kids these days. Latin was a much more elegant standard.
  - CoastalCoder 2y ago
    Yeah, but: Quis legentibus scribit?
deleted 2y ago
[deleted]
stalfosknight 2y ago
You can on all of Apple's platforms.
deleted 2y ago
[deleted]
hchak 2y ago
Hoping that LLMs can solve our “Tower of Babel” problem… :)
Dwedit 2y ago
But you can assume non-UTF-8 upon seeing an invalid UTF-8 byte sequence. From there, it can be application-specific depending on what encoding you expect to see. (And of course UTF-16 if there's a BOM)
klysm 2y ago
Meh I'd rather start making those assumptions and blame everything that doesn't use UTF-8 as broken.
lifthrasiir 2y ago
You can't just assume UTF-8, but you can verify that it is almost surely encoded in UTF-8 unlike other legacy encodings. Which makes UTF-8 the first and foremost consideration.
Havoc 2y ago
You underestimate my willingness to happy path code...
hnick 2y ago
Based on my past role, you can't even assume UTF-8 when the file says it's UTF-8. Clients would constantly send CSV or other files with an explicit BOM or other marking indicating UTF-8 but the parser would choke since they just output native Windows-1252 or similar into it. I think some programs just spit it out since it's standard.
kazinator 2y ago
Indeed, you can't assume UTF-8. What you do, rather, is drop support for non-UTF-8. Work with tech-stacks whose text handling is based strictly around Unicode and UTF-8, and find enough opportunities that way that you don't have to care about anything else. Let the customers who cling to data in weird encodings go to someone who makes it a nice to support that.
djha-skin 2y ago
Joel spolsky spoke against this exact statistics-based approach when he wrote about unicode[1]: > What do web browsers do if they don’t find any Content-Type, either in the http headers or the meta tag? Internet Explorer actually does something quite interesting: it tries to guess, based on the frequency in which various bytes appear in typical text in typical encodings of various languages, what language and encoding was used. Because the various old 8 bit code pages tended to put their national letters in different ranges between 128 and 255, and because every human language has a different characteristic histogram of letter usage, this actually has a chance of working. It’s truly weird, but it does seem to work often enough that naïve web-page writers who never knew they needed a Content-Type header look at their page in a web browser and it looks ok, until one day, they write something that doesn’t exactly conform to the letter-frequency-distribution of their native language, and Internet Explorer decides it’s Korean and displays it thusly, proving, I think, the point that Postel’s Law about being “conservative in what you emit and liberal in what you accept” is quite frankly not a good engineering principle. 1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- bastawhiz 2y ago
  Exactly. I used this technique at Mozilla in 2010 when processing Firefox add-ons, and it misidentified scripts as having the wrong encoding pretty frequently. There's far less weird encoding out there than there are false positives from statistics-based approaches.
- mark-r 2y ago
  20 years old, but still true.
- kelnos 2y ago
  I don't think he was speaking against the statistics-based approach itself, just against Postel's Law in general. Ideally people would see gibberish (or an error message) immediately if they don't provide an encoding; then they'll know something is wrong, figure it out, fix it, and never have the issue again. But if we're in a situation where we already have lots and lots of text documents that don't have an encoding specified, and we believe it's not feasible to require everyone to fix that, then it's actually pretty amazing that we can often correctly guess the encoding.
deleted 2y ago
[deleted]
rch 2y ago
Don't assume; force UTF-8.
ahi 2y ago
Why utf-8 when we have MARC-8? https://en.wikipedia.org/wiki/MARC-8 https://en.wikipedia.org/wiki/MARC-8
- moefh 2y ago
  I had never heard of MARC-8 before, but after a quick google search: "There are no defined MARC-8 character sets for Armenian, Bengali, Devanagari, Ethiopic, Syriac, Tamil or Thai."[0] "If a character is not representable in MARC-8 of a MARC-21 record, then UTF-8 must be used instead."[1] [0] https://help.oclc.org/Metadata_Services/Connexion/Connexion_client/International_cataloging/Get_started/10Valid_character_sets_for_supported_scripts https://help.oclc.org/Metadata_Services/Connexion/Connexion_... [1] https://en.wikipedia.org/wiki/MARC-8 https://en.wikipedia.org/wiki/MARC-8
hot_gril 2y ago
Java and Javascript both use UTF-16 for internal string representation, even though JSON specifies UTF-8. Windows APIs too. I'm still not sure why, but it means that one char uses at least 2 bytes even if it's in the ASCII range.
- mark-r 2y ago
  Early adopters of Unicode used the first available encoding, UCS-2. UTF-16 is an extension of that to handle the increased range of code points that came later.
AzzyHN 2y ago
I'm not a programmer, but how hard is it to write something that just checks what encoding the file uses. Or at least does its best to guess?
- bastawhiz 2y ago
  It's not easy! Go check out the source for chardet (Python). It's not pretty.
- torstenvl 2y ago
  Pretty hard. There's no true way of knowing when you're right. You can make educated guesses based on statistical likelihood of certain patterns, but nothing stops people from constructing text that happens to look more "normal" when interpreted as another character set. If I see the bytes 0xE5 0xD1 0xCD 0xC8 0xC7, then it is a decent bet that it's intended to be مرحبا in DOS 708. But there's no definitive reason why it couldn't be σ╤═╪╫ (DOS 437) or åÑÍÈÇ (Windows-1252) or Еямих (KOI8-RU). Especially if you don't know for sure that the data is intended to be natural language. I can be pretty reasonably certain that it isn't, say, Japanese. 0xE5 in ShiftJIS is the first of a double-byte sequence, and it's odd, so the next byte would have to come from the range 0x40-0x9E, and 0xD1 does not fall within that range. In other words, it simply isn't valid ShiftJIS (it also isn't valid UTF-8). So you can narrow down the possibilities... but you're still making an educated guess. And you have to figure out a way to program that fuzzy, judgment-y analysis into software.
- hgs3 2y ago
  I've written code to do this. If you're lucky there will be a BOM (byte order mark) or MIME type to indicate the encoding form. In this case you know the encoding. If you don't have this information, then you must guess the encoding. The issue is guessing may not produce accurate results, especially if you don't have enough bytes to go by. The program I wrote to guess the encoding would scan the bytes in multiple passes. Each pass would check if the bytes encoded valid characters in some specific encoding form. After a pass completed I would assign it a score based on how many successfully encoded characters were (or were not) found. After all passes completed I'd pick the highest score and assume that was the encoding. This approach ended up being reasonably reliable assuming there were enough bytes to go by.
jheriko 2y ago
Fuck me it's 2024 2014 me shaking his head in ways 2004 me saw coming in 1994. Im only 40. FUCK
pylua 2y ago
Stupid question: how are the headers passed for http? What encoding describes the encoding ?
- a57721 2y ago
  The RFCs for HTTP require headers to be in US-ASCII.
- mark-r 2y ago
  I think the headers are pure ASCII, which as noted is a subset of almost all encodings.
- deleted 2y ago
  [deleted]
missblit 2y ago
Encodings I have used or been exposed to in my career: ASCII, Latin1, Windows-1252, UTF-8, UTF-16, UTF-32, GBK, Zawgyi, Shift-JIS
timoteostewart 2y ago
Fascinating topic. There are two ways the user/client/browser receives reports about the character encoding of content. And there are hefty caveats about how reliable those reports are. (1) First, the Web server usually reports a character encoding, a.k.a. charset, in the HTTP headers that come with the content. Of course, the HTTP headers are not part of the HTML document but are rather part of the overhead of what the Web server sends to the user/client/browser. (The HTTP headers and the `head` element of an HTML document are entirely different.) One of these HTTP headers is called Content-Type, and conventionally this header often reports a character encoding, e.g., "Content-Type: text/html; charset=UTF-8". So this is one place a character encoding is reported. If the actual content is not an (X)HTML file, the HTTP header might be the only report the user/client/browser receives about the character encoding. Consider accessing a plain text file via HTTP. The text file isn't likely to itself contain information about what character encoding it uses. The HTTP header of "Content-Type: text/plain; charset=UTF-8" might be the only character encoding information that is reported. (2) Now, if the content is an (X)HTML page, a charset encoding is often also reported in the content itself, generally in the HTML document's head section in a meta tag such as '<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>' or '<meta charset="utf-8">'. Now just because an HTML document self-reports that it uses a UTF-8 (or whatever) character encoding, that's hardly a guarantee that the document does in fact use said character encoding. Consider the case of a program that generates web pages using a boilerplate template still using an ancient default of ISO-8859-1 in the meta charset tag of its head element, even though the body content that goes into the template is being pulled from a database that spits out a default of utf-8. Boom. Mismatch. Janky code is spitting out mismatched and inaccurate character encoding information every day. Or to consider web servers. Consider a web server whose config file contains the typo "uft-8" because somebody fat-fingered while updating the config (I've seen this in random web pages.). Or consider a web server that uses a global default of "utf-8" in its outgoing HTTP headers even when the content being served is a hodge-podge of UTF-8, WINDOWS-1251, WINDOWS-1252, and ISO-8859-1. This too happens all the time. I think the most important takeaway is that with both HTTP headers and meta tags, there's no intrinsic link between the character encoding being reported and the actual character encoding of the content. What a Web server tells me and what's in the meta tag in the markup just count as two reports. They might be accurate, they might not be. If it really matters to me what the character encoding is, there's nothing for it but to determine the character encoding myself. I have a Hacker News reader, https://www.thnr.net https://www.thnr.net, and my program downloads the URL for every HN story with an outgoing link. I have seen binary files sent with a "UTF-8" Content-Type header. I have seen UTF-8 files sent with a "inode/x-empty" Content-Type header. My logs have literally hundreds of goofy inaccurate reports of content types and character encodings. Because I'm fastidious and I want to know what a file actually is, I have a function `get_textual_mimetype` that analyzes the content of what the URL's web server sends me. My program downloads the content and uses tools such as `iconv` and `isutf8` to get some information about what encoding it might be. It uses `xmlwf` to check if it's well-formed XML. It uses `jq` to check whether it's valid JSON. It uses `libmagic`. There's a lot of fun stuff the program does to pin down with a high degree of certainty what the content is. I want my program to know whether the content is an application/pdf, an iamge/webp, a text/html, an application/xhtml+xml, a text/x-csrc, or whatever. Only a rigorous analysis will tell you the truth. (If anyone is curious, the source for `get_textual_mimetype` is in the repo for my HN reader project: https://github.com/timoteostewart/timbos-hn-reader/blob/main/utils_mimetypes_magic.py https://github.com/timoteostewart/timbos-hn-reader/blob/main... )
neonsunset 2y ago
I will assume UTF-8 and if it's not, it will be your fault :)
flohofwoe 2y ago
The post seems to assume that only UTF-16 has Byte Order Marks, but as pointless as it sounds, UTF-8 has a BOM too (EF BB BF). It seems to be a Windows thing though, haven't seen it in the wild anywhere else (and also rarely on Windows - since text editors typically allow to save UTF-8 files with or without BOM. I guess it depends on the text editor which of those is the default).
- calpaterson 2y ago
  Yes you're right, UTF-8 technically does as well. I've never seen them in real life either. UTF-16 BOMs do have a useful function as I recall: they really help Excel detect your character encoding (Excel is awful at detecting character encoding).
- LinAGKar 2y ago
  That's not really a byte order mark though, it's just the UTF-8 encoding of U+FEFF, which corresponds to the byte order mark in UTF-16. Honestly, emitting that into UTF-8 was probably the result of a bug originally, caused by Windows Unicode APIs being designed for UTF-16.
  - flohofwoe 2y ago
    Wikipedia begs to differ though (check under "UTF-8": https://en.wikipedia.org/wiki/Byte_order_mark https://en.wikipedia.org/wiki/Byte_order_mark) Also, at least the Rider IDE and VSCode lets you select to save UTF-8 with or without BOM. My guess is that it's used mainly to differentiate between UTF-8 and codepage encoded 8-bit ASCII files (which are still very common on Windows).
mseepgood 2y ago
Yes, I can.
matheusmoreira 2y ago
Yeah but I'm gonna do it anyway. If it's not UTF-8 it's terrible and broken and not worth supporting unless some serious cash is on the table.
mschuster91 2y ago
> CSV files, in particular, have no way to signal, in-band, which encoding is used. That's actually wrong. Add an UTF-8 BOM, that's enough for Excel (and some other libraries) to know what is going on [1]. [1] https://csv.thephpleague.com/8.0/bom/ https://csv.thephpleague.com/8.0/bom/
vkaku 2y ago
The probability of web content not in UTF-8 is increasingly getting lower and lower. Last I tracked, as of this month, 0.3% of surveyed web pages used Shift JIS. It has been declining steadily. I really hope people move to UTF-8. While it is important to understand how the code pages and encodings helped, I think it's a good time to actually start moving a lot of applications to use UTF-8. I am perfectly okay if people want to use UTF-16 (the OG Unicode) and it's extensions alternatively, especially for Asian applications. Yes, historic data preservation requires a different strategy than designing stuff for the future. It is okay to however migrate to these encodings and keep giving old data and software new life.
- layer8 2y ago
  Just to be pedantic, the OG Unicode is UCS-2, not UTF-16, the main difference being that surrogate characters didn’t exist originally.
  - vkaku 2y ago
    Good point. Ah, we peel onions every few years.
eqvinox 2y ago
The solution, obviously, is to train an LLM to recognize the character set.
- layer8 2y ago
  That sounds like an efficient use of computing resources.
tanin 2y ago
I'm actually having this issue where users import CSV files that don't seem to be valid. DuckDB would throw out an error like: Invalid Closing Quote: found non trimable byte after quote at line 34, Invalid Input Error: Invalid unicode (byte sequence mismatch) detected in value construction, Value with unterminated quote found One example: pgAdmin can export a database table into a CSV... but the CSV isn't valid for DuckDB to consume. Because, for some odd reason, pgAdmin uses a single quote to escape a double quote. This blog is pretty timely. Thank you for writing it!
mihaaly 2y ago
Excellent article, good content, good length, enlightened subtexts and references, joy to read.
chungy 2y ago
The article's pretty weird for presenting little-endian UTF-16 as normal and barely even mentioning that big-endian is an option (in fact, seems to refer to it as "backwards"), even though big-endian is a much more human readable format.
rob74 2y ago
30 years ago: "you can't just assume ASCII" Today: "you can't just assume UTF-8" The more things change, the more they stay the same...
camgunz 2y ago
Dear lazyweb: I think I read something about Postel's Law being essential to the internet's success -- maybe this was also IPv6 related? Does anyone else remember this?
- camgunz 2y ago
  (I found it: https://apenwarr.ca/log/20200708 https://apenwarr.ca/log/20200708)
planede 2y ago
A bag of bytes is a bag of bytes. Any encoding should be either assumed by the protocol being used or otherwise specified.
TypicalHog 2y ago
IMO humanity should just do ASCII and call it a day. I'm not talking about viability of this, just my wishes.
mikhailfranco 2y ago
Developers should assume UTF-8 for text files going forward. UTF-8 should have no BOM. It is the default. And there are no undefined Byte sequences that need an Order. Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work? Others variants of Unicode have BOMs, e.g. UTF-16BE. We know CJK languages need UTF-16 for compression. The BOM is only a couple more bytes. No problem, so far so good. But there are old files, that are in 'platform encoding'. Fine, let there be an OS 'locale', that has a default encoding. That default can be overridden with another OS 'encoding' variable. And that can be overridden by an application arg. And there may be a final default for a specific application, that is only ever used with one language encoding. Then individual files can override all of the above ... Text file (serialization) formats should define handling of optional BOM followed by an ASCII header that defines the encoding of the body that follows. One can also imagine .3 filetypes that have a unique or default encoding, with values maintained by, say, IANA (like MIME types). XML got this right. JSON and CSV are woefully neglectful, almost to the point of criminal liability. But in the absence of all of the above, the default-default-default-default-default is UTF-8. We are talking about the future, not the past. Design for UTF-8 default, and BOMs for other Unicodes. Microsoft should have defined BOMs for Windows-Blah encodings, not for UTF-8! When the whole world looks to the future, Microsoft will follow. Eventually. Reluctantly.
- calpaterson 2y ago
  If csv files bring criminal liability then I am guilty. Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G https://discord.gg/uqu4BkNP5G Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching. I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap
  - mikhailfranco 2y ago
    I explained how old files could be read. Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem. You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible. But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now? [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)
kdklol 2y ago
>You can't just assume UTF-8 But I will, because in this day and age, I should be perfectly able to do so. Non-use of UTF-8 should be simply considered a bug and not treating text as UTF-8 should frankly be a you problem. At least for anything reasonably modern, by which I mean made in the last 15 years at least.
teliskr 2y ago
I maintain a system I created in 2004 (crazy right). Not sure how we lived; but at the time, emojis were not as much of a thing. This has come to bite me several times.
GnarfGnarf 2y ago
Why would anyone use anything other than UTF-8 in this day and age? Windows took a gamble years ago when the winner was not obvious, we’re stuck with UCS-2, but you can circumvent that with a good string library like QT’s Qstring.
iamcreasy 2y ago
By heuristics, is the author referring to the rules and policies published by Unicode? [1] Link[1] was referring as a solution to this problem in this article: Absolute Minimum Every Software Developer Must Know About Unicode in 2023 [2]. [1] https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Bound... [2] https://tonsky.me/blog/unicode/ https://tonsky.me/blog/unicode/
teddyh 2y ago
If you’re actually in a position where you need to guess the encoding, something like “ftfy” <https://github.com/rspeer/python-ftfy https://github.com/rspeer/python-ftfy> (webapp: <https://ftfy.vercel.app/ https://ftfy.vercel.app/>) is a perfectly reasonable choice. But, you should always do your absolute utmost not to be put in a situation where guessing is your only choice.
otikik 2y ago
However you can check for invalid utf-8 sequences, throw an error with "invalid encoding on byte x, please use valid utf-8" if encountered, and from that point on assume utf-8.
teknopaul 2y ago
I suspect the author does not about the first bit being 1 thing. utf8 is magic. You can assume US ASCII for lots of very useful text protocols , like http and Stomp and not care what the variable string bytes mean. Soooo many software architects don't grok the magic of it. You can define 8bit parser that check for "a:"(some string)\n and work with a shit load of human languages. The company I work for does not realise that most of f the 50 year old legacy C it has is fine with utf8 for all the arbitrary fixed length or \0 terminated strings it stores.
lolc 2y ago
Just the most recent episode: A statistician is using PHP, on Windows, to analyze text for character frequency. He's rather confused by the UTF-16LE encoding and thinks the character "A" is numbered 4100 because that's what is shown in a hex-editor. I tried explaining about the little-endian part, and mb-string functions in PHP. And that PHP is not a good fit for his projects. Then I realized that this is hilarious and I won't be able to kick him from his local minimum there. Everything he could learn about encodings would first complicate his work.
jheriko 2y ago
no mention of the BOM... tragic.

195 points by calpaterson 2y ago

bhaney 2y ago

I'm just gonna assume UTF-8

calpaterson 2y ago
榥\ue0af侬펭懃䒥亷
- bhaney 2y ago
  And a good day to you too, my friend whose input I'm going to discard
  - eqvinox 2y ago
    It's garbage anyway, which you can (non-reliably) guess by there being a Korean character in the middle of C/J kanji. (Kanji are not completely gone from Korean, but mostly.)
    - calpaterson 2y ago
      It's mojibake for "probably a bad idea" [in Chinese]
duskwuff 2y ago
I'm disappointed that the article doesn't discuss this in more detail. Most byte sequences are not valid UTF-8. If you can decode a message as UTF-8 with no errors, that is almost certainly the correct encoding to use; it's extremely unlikely that some text in another encoding just happened to be perfectly valid as UTF-8. (The converse is not true; most 8-bit text encodings will happily decode UTF-8 sequences to nonsense strings like ðŸš©.) If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding. But that should be a fallback, not the first thing you try.
- actionfromafar 2y ago
  Not extremely unlikely. Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII.
  - PaulDavisThe1st 2y ago
    such encodings are also UTF-8 then, are they not?
    - tsimionescu 2y ago
      I think there are a bunch of encodings that just repurposed a few ASCII characters as different characters - someone on this page was giving the example of some Swedish encoding where {}| were replaced with three accented Swedish letters. There are probably a bunch of others. In those cases, the text will decode fine as UTF-8, but it will display the wrong thing.
  - srj 2y ago
    A distinction without a difference?
    - actionfromafar 2y ago
      I meant, some messages will decode fine as UTF-8, but in some other messages there may be letters which don't fit in 7 bits. So some simple testing, especially with English words, will show it to work fine. But as soon as a non-7-bit characters creeps, it will stop working fine.
  - duskwuff 2y ago
    > Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII. At which point the message is effectively ASCII. UTF-8 is a superset of ASCII, so "decoding" ASCII as UTF-8 is fine. (Yes, I know there are some Japanese text encodings where 0x5c is decoded as "¥" instead of "\". But they're sometimes treated as backslashes even though they look like ¥ symbols so handling them "correctly" is complicated.)
klysm 2y ago
You can’t just do that! /s

jujube3 2y ago

Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is no need to have dozens of incompatible systems.

qwertox 2y ago
You can't assume it if you're handed a random sample of text files.
- duskwuff 2y ago
  You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements. (Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
  - ranger_danger 2y ago
    Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.
    - duskwuff 2y ago
      Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
      - ranger_danger 2y ago
        I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.

pronoiac 2y ago

Archive copy: https://web.archive.org/web/20240429061925/https://csvbase.com/blog/9 https://web.archive.org/web/20240429061925/https://csvbase.c...

o11c 2y ago

Default UTF-8 is better than the linked suggestion of using a heuristic, but failing catastrophic when old data is encountered is unacceptable. There must be a fallback. (Note that the heuristic for "is this intended to be UTF-8" is pretty reliable, but most other encoding-detection heuristics are very bad quality)

dandigangi 2y ago

Except I can

Veserv 2y ago

Off-topic, but the bit numbering convention is deliciously confusing. Little-endian bytes (lowest byte is leftmost) and big-endian bits (bits contributing less numerical value are rightmost) are normal, but the bits are referenced/numbered little-endian (first bit is leftmost even though it contributes the most numerical value). When I first read the numbering convention I thought it was going to be a breath of fresh air of someone using the much more sane, but non-standard, little-endian bits with little-endian bytes, but it was actually another layered twist. Hopefully someday English can write numbers little-endian, which is objectively superior, and do away with this whole mess.

kstrauser 2y ago
> Hopefully someday English can write numbers little-endian, which is objectively superior Upon reading this, I threw my laptop out the window.
- Veserv 2y ago
  It actually is if we did not need to consider historical baggage. Especially in programming where we already use in-band encoding like 0x to denote a hex string or 0b to denote a binary string. I like using 1{s}, e.g. 1x to denote a little-endian hex string and 1b to denote a little-endian binary string, to denote little-endian encoding. But, even ignoring programming, it is still better in normal use. The Arabic language got it right writing little-endian (Arabic numbers are written the same, but Arabic is a right-to-left language so it is actually little-endian), and the European languages just stole it stupid by copying the form instead of the function.
  - filmor 2y ago
    Arabian numerals are derived from Indian scripts, which are written left to right, yet the order of digits is already the same we have today.
    - pezezin 2y ago
      Not to mention that in most (all?) spoken languages numbers are big-endian.

bandyaboot 2y ago

> In the most popular character encoding, UTF-8, character number 65 ("A") is written: > 01000001 > Only the second and final bits are 1, or "on". Isn’t it more accurate to say that the first and penultimate bits are 1, or “on”?

fl7305 2y ago
It depends on whether your bit numbering is like x86 (your description), or PowerPC (left most bit is 0).
- duskwuff 2y ago
  Basically everyone uses x86 bit numbering. It has the pleasant property that the place value of every bit is always 2^n (or -2^n for a sign bit), and zero-extending a value doesn't change the numbering of its bits.
  - fl7305 2y ago
    Sure, it is by far the industry standard. It works much better for handling discrete integers. Once you get into bitfield instructions it is nice to have bit 0 be the "left most bit".
  - mark-r 2y ago
    It's a mistake to call it x86 bit numbering, it's way older than that.
- bandyaboot 2y ago
  The more I thought it through, even assuming x86, I guess there’s just no “correct” way to casually reference bit positions when we read them in the opposite order from the machine. Are they being referenced from the perspective of a human consumer of text, or the machine’s perspective as a consumer of bits? If I were writing that content, I’d have a difficult time deciding on which to use. If I were writing for a lay person, referencing left-to-right seems obvious, but in this case where the audience is primarily developers, it becomes much less obvious.

vitaut 2y ago

This is so spectacularly outdated. KOI-8 has been dead for ages.

JonChesterfield 2y ago

How about assume utf-8, and if someone has some binary file they'd rather a program interpret as some other format, they turn it into utf-8 using a standalone program first. Instead of burning this guess-what-bytes-they-might-like nonsense into all the software. We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

zarzavat 2y ago
Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players. It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.
- fl7305 2y ago
  > It’s much easier to tell the people with old cassette tapes to rip them I assume you mean "rip them", as in transcode to a different format? In that case, you need a tool that takes the old input format(s) and convert them to the new format. For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8. Isn't the point of the article to describe how an engineer would create such a tool?
  - 0xffff2 2y ago
    A tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
    - fl7305 2y ago
      > should be asking the user to select explicit input and output formats It depends on the requirements. If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.

dublin 2y ago

Make your life easy. Assume 7-bit ASCII. No one needs all those other characters, anyway...

AlienRobot 2y ago
Do we really need 128 permutations just to express an alphabet of 26 letters? I think we should use a 4 bit encoding. 0 - NUL 1-7 - aeiouwy 8 - space 9-12 - rst 13-15 - modifiers When modifier bits are set, the values of the next half-byte change to represent the rest of the alphabet, numbers, symbols, etc. depending on the bits set.
- apantel 2y ago
  R S T L N E
- Dylan16807 2y ago
  I like it, but what if we use 2 to pick symbols, 3 to pick numbers, and 4 and 5 to pick capital letters.
jonathankoren 2y ago
As someone who has repeatedly had to deal with Unicode nonsense; I wholeheartedly agree. Also, you don’t need accents. You just need to know how to read and have context. See: live and live, read and read, etc.

drdaeman 2y ago

Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024" then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun. Or, you know, just say "nah, I can, those ancient stuff don't matter (outside of obligatory exceptions, like software archeology) anymore." If someone wants to feed me a KOI8-R or JIS X 0201 CSV heirloom, they should convert it into something modern first.

deleted 2y ago
[deleted]
skissane 2y ago
> Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024" I have a hobby interest in IBM mainframes and IBM i, so yes to EBCDIC for me. (I have encountered them professionally too, but only to a very limited extent.) In practice, I find looking for 0x40 (EBCDIC space) a useful heuristic. Even in binary files, since many mainframe data structures are fixed length space padded. > then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun. Actual use of UTF-EBCDIC, while not nonexistent, has always been extremely rare. A person could spend an entire career dedicated to IBM mainframes and never encounter it EBCDIK, at first I wondered if that was a joke, now I realise it is a name used for non-IBM Japanese EBCDIC code pages. Again, something one can spend a whole career in mainframes and never encounter – if one never works in Japan, if one works for a vendor whose products aren't sold in Japan, probably even if you work for a vendor whose products are sold in Japan but only to IBM sites (as opposed to Fujitsu/Hitachi sites)

smeagull 2y ago

I absolutely can. If it's not UTF-8, I assume it's worthless.

AlienRobot 2y ago

"You can't assume a 32 bit integer starts from 0"

Karellen 2y ago

Don't worry, I never assume UTF-8. I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, and I get non-UTF-8 input, I'm fine bailing on that with a "malformed input - please correct" error.

lmm 2y ago
[flagged]
- pavel_lishin 2y ago
  Does the entirety of Japan shun utf8?
  - koito17 2y ago
    Many Japanese websites have migrated from Shift-JIS to UTF-8, but this still ignores the fact that e.g. television captioning uses special characters[1] that are not found in UTF-8 or Shift-JIS. Windows itself has a habit of using its own Windows-932 encoding, which frequently causes problems in the Unix software I use. (e.g. Emacs fails at auto-detecting this format, and instructing Emacs to use Shift-JIS will result in decoding issues) [1] section 1.2 in https://www.fontucom.com/pdf/AFSARIBR-Map.pdf https://www.fontucom.com/pdf/AFSARIBR-Map.pdf
    - cheeze 2y ago
      If Emacs fails to do it correctly, end developers are screwed.
      - koito17 2y ago
        Windows-932 is the text encoding I dread the most. I wish literally everything could use Unicode, but we're not quite there yet. There's a reason encoding-japanese[1] on NPM has nearly 1 million weekly downloads. Unfortunately, I have to use it in one of my React Native applications since the one of servers I speak to returns text encoded as Shift-JIS and Hermes does not implement JavaScript's TextDecoder API. Shift-JIS is marginally better than the Windows code page, but I'd really prefer UTF-8. [1] https://www.npmjs.com/package/encoding-japanese https://www.npmjs.com/package/encoding-japanese

groestl 2y ago

I will assume it, I will enforce it where I can, and I will fight tooth and nail should push come to shove. I got 99 problems, but charsets aint one of them.

zadokshi 2y ago

Better to assume UTF8 and fail with a clear message/warning. Sure you can offer to guess to help the end user if it fails, but as other people have pointed out, it’s been standard for a long time now. Even python caved and accepted it as the default: https://peps.python.org/pep-0686/ https://peps.python.org/pep-0686/

koito17 2y ago

The comments in this thread are a bit amusing. I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world". Many Japanese documents and sites still use Shift JIS. Windows has this strange Windows-932 format that you will frequently encounter in CUE files outputted by some CD ripping software. ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. These special characters are mostly icons used in traffic and weather reports, but transcoding to UTF-8 still causes trouble with these icons.

bongodongobob 2y ago
Most people on this site probably live in the world where everything is done in English. That's the norm for the vast majority of businesses and people in the US.
- fl7305 2y ago
  Even for those people, there is still a ton of old text files in Windows-1252 etc floating around. You can choose to never work on projects where you have to deal with files like that. But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.
  - pixl97 2y ago
    Why would you write a tool that does that instead of just digging up one that's already written? This sounds like the folly of writing ones own encryption library.
    - fl7305 2y ago
      Which library do you pick? Which ones are good? Bad? What are the common pitfalls that the library must deal with? How do you know how to evaluate them if all you know is "not UTF-8? Abort with error" ?

norir 2y ago

If it's turtles all the way down and at every level you use utf-8, it's hard to see how any input with a different encoding (for the same underlying text) will not be detected before any unintended side effects were invoked. At this point, I don't see any sufficiently good reason to not use utf-8 exclusively in any new system. Conversions to and from other encodings would only be done at well defined boundaries when I'm calling into dependencies that require non utf-8 input for whatever reason.

kstrauser 2y ago

If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, and the sender doesn't like the results, that's on them. I'm willing to spend approximately zero effort trying to guess what nonstandard thing they're trying to send me unless they're paying me or my company a whole lot of money, in which case I'll convert it to UTC upon import and continue on from there. Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8. If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.

kccqzy 2y ago
> For instance, if it were the case that it did a bad job of encoding Mandarin I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.
- PeterisP 2y ago
  A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.

jcranmer 2y ago

I haven't seen discussion of this point yet, but the post completely fails to provide any data to back up its assertion that charset detection heuristics works, because the feedback I've seen from people who actually work with charsets is that it largely doesn't (especially if you're based on naive one-byte frequency analysis). Okay, sure, it works if you want to distinguish between KOI8-R and Windows-1252, but what about Windows-1252 and Windows-1257? See for example this effort in building a universal charset detector in Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1551276 https://bugzilla.mozilla.org/show_bug.cgi?id=1551276

toast0 2y ago
I've done some charset detection, although it's been a while. Heuristics kind of work for somethings --- I'm a big fan of if it's decodable as utf-8, it's probably utf-8, unless there's zero bytes (in most text). If there's a lot of zero bytes, maybe it's UCS-2 or UTF-16, and you can try to figure out the byte order and if it decodes as utf-16. If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.
renhanxue 2y ago
Yeah. The fantastic python library ftfy ("fixes text for you", https://ftfy.readthedocs.io/en/latest/index.html https://ftfy.readthedocs.io/en/latest/index.html), designed to fix mangled Unicode (mojibake, of many different varieties), mentions in its docs that heuristic encoding guessers are the cause of many of the problems ftfy is designed to fix. It's magical, by the way. It has an entire section of its documentation dedicated to why you should not use heuristic charset detection: https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-chardet https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-c...
- torstenvl 2y ago
  That section explains why not to use a specific naive charset detection library that doesn't have a strong prior for UTF-8. There's no basis for extrapolating that further.

mgaunard 2y ago

There is a pretty successful world language standard: English.

bawolff 2y ago
Kids these days. Latin was a much more elegant standard.
- CoastalCoder 2y ago
  Yeah, but: Quis legentibus scribit?

deleted 2y ago

[deleted]

stalfosknight 2y ago

You can on all of Apple's platforms.

deleted 2y ago

[deleted]

hchak 2y ago

Hoping that LLMs can solve our “Tower of Babel” problem… :)

Dwedit 2y ago

But you can assume non-UTF-8 upon seeing an invalid UTF-8 byte sequence. From there, it can be application-specific depending on what encoding you expect to see. (And of course UTF-16 if there's a BOM)

klysm 2y ago

Meh I'd rather start making those assumptions and blame everything that doesn't use UTF-8 as broken.

lifthrasiir 2y ago

You can't just assume UTF-8, but you can verify that it is almost surely encoded in UTF-8 unlike other legacy encodings. Which makes UTF-8 the first and foremost consideration.

Havoc 2y ago

You underestimate my willingness to happy path code...

hnick 2y ago

Based on my past role, you can't even assume UTF-8 when the file says it's UTF-8. Clients would constantly send CSV or other files with an explicit BOM or other marking indicating UTF-8 but the parser would choke since they just output native Windows-1252 or similar into it. I think some programs just spit it out since it's standard.

kazinator 2y ago

Indeed, you can't assume UTF-8. What you do, rather, is drop support for non-UTF-8. Work with tech-stacks whose text handling is based strictly around Unicode and UTF-8, and find enough opportunities that way that you don't have to care about anything else. Let the customers who cling to data in weird encodings go to someone who makes it a nice to support that.

djha-skin 2y ago

Joel spolsky spoke against this exact statistics-based approach when he wrote about unicode[1]: > What do web browsers do if they don’t find any Content-Type, either in the http headers or the meta tag? Internet Explorer actually does something quite interesting: it tries to guess, based on the frequency in which various bytes appear in typical text in typical encodings of various languages, what language and encoding was used. Because the various old 8 bit code pages tended to put their national letters in different ranges between 128 and 255, and because every human language has a different characteristic histogram of letter usage, this actually has a chance of working. It’s truly weird, but it does seem to work often enough that naïve web-page writers who never knew they needed a Content-Type header look at their page in a web browser and it looks ok, until one day, they write something that doesn’t exactly conform to the letter-frequency-distribution of their native language, and Internet Explorer decides it’s Korean and displays it thusly, proving, I think, the point that Postel’s Law about being “conservative in what you emit and liberal in what you accept” is quite frankly not a good engineering principle. 1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

bastawhiz 2y ago
Exactly. I used this technique at Mozilla in 2010 when processing Firefox add-ons, and it misidentified scripts as having the wrong encoding pretty frequently. There's far less weird encoding out there than there are false positives from statistics-based approaches.
mark-r 2y ago
20 years old, but still true.
kelnos 2y ago
I don't think he was speaking against the statistics-based approach itself, just against Postel's Law in general. Ideally people would see gibberish (or an error message) immediately if they don't provide an encoding; then they'll know something is wrong, figure it out, fix it, and never have the issue again. But if we're in a situation where we already have lots and lots of text documents that don't have an encoding specified, and we believe it's not feasible to require everyone to fix that, then it's actually pretty amazing that we can often correctly guess the encoding.

deleted 2y ago

[deleted]

rch 2y ago

Don't assume; force UTF-8.

ahi 2y ago

Why utf-8 when we have MARC-8? https://en.wikipedia.org/wiki/MARC-8 https://en.wikipedia.org/wiki/MARC-8

moefh 2y ago
I had never heard of MARC-8 before, but after a quick google search: "There are no defined MARC-8 character sets for Armenian, Bengali, Devanagari, Ethiopic, Syriac, Tamil or Thai."[0] "If a character is not representable in MARC-8 of a MARC-21 record, then UTF-8 must be used instead."[1] [0] https://help.oclc.org/Metadata_Services/Connexion/Connexion_client/International_cataloging/Get_started/10Valid_character_sets_for_supported_scripts https://help.oclc.org/Metadata_Services/Connexion/Connexion_... [1] https://en.wikipedia.org/wiki/MARC-8 https://en.wikipedia.org/wiki/MARC-8

hot_gril 2y ago

Java and Javascript both use UTF-16 for internal string representation, even though JSON specifies UTF-8. Windows APIs too. I'm still not sure why, but it means that one char uses at least 2 bytes even if it's in the ASCII range.

mark-r 2y ago
Early adopters of Unicode used the first available encoding, UCS-2. UTF-16 is an extension of that to handle the increased range of code points that came later.

AzzyHN 2y ago

I'm not a programmer, but how hard is it to write something that just checks what encoding the file uses. Or at least does its best to guess?

bastawhiz 2y ago
It's not easy! Go check out the source for chardet (Python). It's not pretty.
torstenvl 2y ago
Pretty hard. There's no true way of knowing when you're right. You can make educated guesses based on statistical likelihood of certain patterns, but nothing stops people from constructing text that happens to look more "normal" when interpreted as another character set. If I see the bytes 0xE5 0xD1 0xCD 0xC8 0xC7, then it is a decent bet that it's intended to be مرحبا in DOS 708. But there's no definitive reason why it couldn't be σ╤═╪╫ (DOS 437) or åÑÍÈÇ (Windows-1252) or Еямих (KOI8-RU). Especially if you don't know for sure that the data is intended to be natural language. I can be pretty reasonably certain that it isn't, say, Japanese. 0xE5 in ShiftJIS is the first of a double-byte sequence, and it's odd, so the next byte would have to come from the range 0x40-0x9E, and 0xD1 does not fall within that range. In other words, it simply isn't valid ShiftJIS (it also isn't valid UTF-8). So you can narrow down the possibilities... but you're still making an educated guess. And you have to figure out a way to program that fuzzy, judgment-y analysis into software.
hgs3 2y ago
I've written code to do this. If you're lucky there will be a BOM (byte order mark) or MIME type to indicate the encoding form. In this case you know the encoding. If you don't have this information, then you must guess the encoding. The issue is guessing may not produce accurate results, especially if you don't have enough bytes to go by. The program I wrote to guess the encoding would scan the bytes in multiple passes. Each pass would check if the bytes encoded valid characters in some specific encoding form. After a pass completed I would assign it a score based on how many successfully encoded characters were (or were not) found. After all passes completed I'd pick the highest score and assume that was the encoding. This approach ended up being reasonably reliable assuming there were enough bytes to go by.

jheriko 2y ago

Fuck me it's 2024 2014 me shaking his head in ways 2004 me saw coming in 1994. Im only 40. FUCK

pylua 2y ago

Stupid question: how are the headers passed for http? What encoding describes the encoding ?

a57721 2y ago
The RFCs for HTTP require headers to be in US-ASCII.
mark-r 2y ago
I think the headers are pure ASCII, which as noted is a subset of almost all encodings.
deleted 2y ago
[deleted]

missblit 2y ago

Encodings I have used or been exposed to in my career: ASCII, Latin1, Windows-1252, UTF-8, UTF-16, UTF-32, GBK, Zawgyi, Shift-JIS

timoteostewart 2y ago

Fascinating topic. There are two ways the user/client/browser receives reports about the character encoding of content. And there are hefty caveats about how reliable those reports are. (1) First, the Web server usually reports a character encoding, a.k.a. charset, in the HTTP headers that come with the content. Of course, the HTTP headers are not part of the HTML document but are rather part of the overhead of what the Web server sends to the user/client/browser. (The HTTP headers and the `head` element of an HTML document are entirely different.) One of these HTTP headers is called Content-Type, and conventionally this header often reports a character encoding, e.g., "Content-Type: text/html; charset=UTF-8". So this is one place a character encoding is reported. If the actual content is not an (X)HTML file, the HTTP header might be the only report the user/client/browser receives about the character encoding. Consider accessing a plain text file via HTTP. The text file isn't likely to itself contain information about what character encoding it uses. The HTTP header of "Content-Type: text/plain; charset=UTF-8" might be the only character encoding information that is reported. (2) Now, if the content is an (X)HTML page, a charset encoding is often also reported in the content itself, generally in the HTML document's head section in a meta tag such as '<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>' or '<meta charset="utf-8">'. Now just because an HTML document self-reports that it uses a UTF-8 (or whatever) character encoding, that's hardly a guarantee that the document does in fact use said character encoding. Consider the case of a program that generates web pages using a boilerplate template still using an ancient default of ISO-8859-1 in the meta charset tag of its head element, even though the body content that goes into the template is being pulled from a database that spits out a default of utf-8. Boom. Mismatch. Janky code is spitting out mismatched and inaccurate character encoding information every day. Or to consider web servers. Consider a web server whose config file contains the typo "uft-8" because somebody fat-fingered while updating the config (I've seen this in random web pages.). Or consider a web server that uses a global default of "utf-8" in its outgoing HTTP headers even when the content being served is a hodge-podge of UTF-8, WINDOWS-1251, WINDOWS-1252, and ISO-8859-1. This too happens all the time. I think the most important takeaway is that with both HTTP headers and meta tags, there's no intrinsic link between the character encoding being reported and the actual character encoding of the content. What a Web server tells me and what's in the meta tag in the markup just count as two reports. They might be accurate, they might not be. If it really matters to me what the character encoding is, there's nothing for it but to determine the character encoding myself. I have a Hacker News reader, https://www.thnr.net https://www.thnr.net, and my program downloads the URL for every HN story with an outgoing link. I have seen binary files sent with a "UTF-8" Content-Type header. I have seen UTF-8 files sent with a "inode/x-empty" Content-Type header. My logs have literally hundreds of goofy inaccurate reports of content types and character encodings. Because I'm fastidious and I want to know what a file actually is, I have a function `get_textual_mimetype` that analyzes the content of what the URL's web server sends me. My program downloads the content and uses tools such as `iconv` and `isutf8` to get some information about what encoding it might be. It uses `xmlwf` to check if it's well-formed XML. It uses `jq` to check whether it's valid JSON. It uses `libmagic`. There's a lot of fun stuff the program does to pin down with a high degree of certainty what the content is. I want my program to know whether the content is an application/pdf, an iamge/webp, a text/html, an application/xhtml+xml, a text/x-csrc, or whatever. Only a rigorous analysis will tell you the truth. (If anyone is curious, the source for `get_textual_mimetype` is in the repo for my HN reader project: https://github.com/timoteostewart/timbos-hn-reader/blob/main/utils_mimetypes_magic.py https://github.com/timoteostewart/timbos-hn-reader/blob/main... )

neonsunset 2y ago

I will assume UTF-8 and if it's not, it will be your fault :)

flohofwoe 2y ago

The post seems to assume that only UTF-16 has Byte Order Marks, but as pointless as it sounds, UTF-8 has a BOM too (EF BB BF). It seems to be a Windows thing though, haven't seen it in the wild anywhere else (and also rarely on Windows - since text editors typically allow to save UTF-8 files with or without BOM. I guess it depends on the text editor which of those is the default).

calpaterson 2y ago
Yes you're right, UTF-8 technically does as well. I've never seen them in real life either. UTF-16 BOMs do have a useful function as I recall: they really help Excel detect your character encoding (Excel is awful at detecting character encoding).
LinAGKar 2y ago
That's not really a byte order mark though, it's just the UTF-8 encoding of U+FEFF, which corresponds to the byte order mark in UTF-16. Honestly, emitting that into UTF-8 was probably the result of a bug originally, caused by Windows Unicode APIs being designed for UTF-16.
- flohofwoe 2y ago
  Wikipedia begs to differ though (check under "UTF-8": https://en.wikipedia.org/wiki/Byte_order_mark https://en.wikipedia.org/wiki/Byte_order_mark) Also, at least the Rider IDE and VSCode lets you select to save UTF-8 with or without BOM. My guess is that it's used mainly to differentiate between UTF-8 and codepage encoded 8-bit ASCII files (which are still very common on Windows).

mseepgood 2y ago

Yes, I can.

matheusmoreira 2y ago

Yeah but I'm gonna do it anyway. If it's not UTF-8 it's terrible and broken and not worth supporting unless some serious cash is on the table.

mschuster91 2y ago

> CSV files, in particular, have no way to signal, in-band, which encoding is used. That's actually wrong. Add an UTF-8 BOM, that's enough for Excel (and some other libraries) to know what is going on [1]. [1] https://csv.thephpleague.com/8.0/bom/ https://csv.thephpleague.com/8.0/bom/

vkaku 2y ago

The probability of web content not in UTF-8 is increasingly getting lower and lower. Last I tracked, as of this month, 0.3% of surveyed web pages used Shift JIS. It has been declining steadily. I really hope people move to UTF-8. While it is important to understand how the code pages and encodings helped, I think it's a good time to actually start moving a lot of applications to use UTF-8. I am perfectly okay if people want to use UTF-16 (the OG Unicode) and it's extensions alternatively, especially for Asian applications. Yes, historic data preservation requires a different strategy than designing stuff for the future. It is okay to however migrate to these encodings and keep giving old data and software new life.

layer8 2y ago
Just to be pedantic, the OG Unicode is UCS-2, not UTF-16, the main difference being that surrogate characters didn’t exist originally.
- vkaku 2y ago
  Good point. Ah, we peel onions every few years.

eqvinox 2y ago

The solution, obviously, is to train an LLM to recognize the character set.

layer8 2y ago
That sounds like an efficient use of computing resources.

tanin 2y ago

I'm actually having this issue where users import CSV files that don't seem to be valid. DuckDB would throw out an error like: Invalid Closing Quote: found non trimable byte after quote at line 34, Invalid Input Error: Invalid unicode (byte sequence mismatch) detected in value construction, Value with unterminated quote found One example: pgAdmin can export a database table into a CSV... but the CSV isn't valid for DuckDB to consume. Because, for some odd reason, pgAdmin uses a single quote to escape a double quote. This blog is pretty timely. Thank you for writing it!

mihaaly 2y ago

Excellent article, good content, good length, enlightened subtexts and references, joy to read.

chungy 2y ago

The article's pretty weird for presenting little-endian UTF-16 as normal and barely even mentioning that big-endian is an option (in fact, seems to refer to it as "backwards"), even though big-endian is a much more human readable format.

rob74 2y ago

30 years ago: "you can't just assume ASCII" Today: "you can't just assume UTF-8" The more things change, the more they stay the same...

camgunz 2y ago

Dear lazyweb: I think I read something about Postel's Law being essential to the internet's success -- maybe this was also IPv6 related? Does anyone else remember this?

camgunz 2y ago
(I found it: https://apenwarr.ca/log/20200708 https://apenwarr.ca/log/20200708)

planede 2y ago

A bag of bytes is a bag of bytes. Any encoding should be either assumed by the protocol being used or otherwise specified.

TypicalHog 2y ago

IMO humanity should just do ASCII and call it a day. I'm not talking about viability of this, just my wishes.

mikhailfranco 2y ago

Developers should assume UTF-8 for text files going forward. UTF-8 should have no BOM. It is the default. And there are no undefined Byte sequences that need an Order. Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work? Others variants of Unicode have BOMs, e.g. UTF-16BE. We know CJK languages need UTF-16 for compression. The BOM is only a couple more bytes. No problem, so far so good. But there are old files, that are in 'platform encoding'. Fine, let there be an OS 'locale', that has a default encoding. That default can be overridden with another OS 'encoding' variable. And that can be overridden by an application arg. And there may be a final default for a specific application, that is only ever used with one language encoding. Then individual files can override all of the above ... Text file (serialization) formats should define handling of optional BOM followed by an ASCII header that defines the encoding of the body that follows. One can also imagine .3 filetypes that have a unique or default encoding, with values maintained by, say, IANA (like MIME types). XML got this right. JSON and CSV are woefully neglectful, almost to the point of criminal liability. But in the absence of all of the above, the default-default-default-default-default is UTF-8. We are talking about the future, not the past. Design for UTF-8 default, and BOMs for other Unicodes. Microsoft should have defined BOMs for Windows-Blah encodings, not for UTF-8! When the whole world looks to the future, Microsoft will follow. Eventually. Reluctantly.

calpaterson 2y ago
If csv files bring criminal liability then I am guilty. Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G https://discord.gg/uqu4BkNP5G Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching. I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap
- mikhailfranco 2y ago
  I explained how old files could be read. Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem. You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible. But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now? [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)

kdklol 2y ago

>You can't just assume UTF-8 But I will, because in this day and age, I should be perfectly able to do so. Non-use of UTF-8 should be simply considered a bug and not treating text as UTF-8 should frankly be a you problem. At least for anything reasonably modern, by which I mean made in the last 15 years at least.

teliskr 2y ago

I maintain a system I created in 2004 (crazy right). Not sure how we lived; but at the time, emojis were not as much of a thing. This has come to bite me several times.

GnarfGnarf 2y ago

Why would anyone use anything other than UTF-8 in this day and age? Windows took a gamble years ago when the winner was not obvious, we’re stuck with UCS-2, but you can circumvent that with a good string library like QT’s Qstring.

iamcreasy 2y ago

By heuristics, is the author referring to the rules and policies published by Unicode? [1] Link[1] was referring as a solution to this problem in this article: Absolute Minimum Every Software Developer Must Know About Unicode in 2023 [2]. [1] https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Bound... [2] https://tonsky.me/blog/unicode/ https://tonsky.me/blog/unicode/

teddyh 2y ago

If you’re actually in a position where you need to guess the encoding, something like “ftfy” <https://github.com/rspeer/python-ftfy https://github.com/rspeer/python-ftfy> (webapp: <https://ftfy.vercel.app/ https://ftfy.vercel.app/>) is a perfectly reasonable choice. But, you should always do your absolute utmost not to be put in a situation where guessing is your only choice.

otikik 2y ago

However you can check for invalid utf-8 sequences, throw an error with "invalid encoding on byte x, please use valid utf-8" if encountered, and from that point on assume utf-8.

teknopaul 2y ago

I suspect the author does not about the first bit being 1 thing. utf8 is magic. You can assume US ASCII for lots of very useful text protocols , like http and Stomp and not care what the variable string bytes mean. Soooo many software architects don't grok the magic of it. You can define 8bit parser that check for "a:"(some string)\n and work with a shit load of human languages. The company I work for does not realise that most of f the 50 year old legacy C it has is fine with utf8 for all the arbitrary fixed length or \0 terminated strings it stores.

lolc 2y ago

Just the most recent episode: A statistician is using PHP, on Windows, to analyze text for character frequency. He's rather confused by the UTF-16LE encoding and thinks the character "A" is numbered 4100 because that's what is shown in a hex-editor. I tried explaining about the little-endian part, and mb-string functions in PHP. And that PHP is not a good fit for his projects. Then I realized that this is hilarious and I won't be able to kick him from his local minimum there. Everything he could learn about encodings would first complicate his work.

jheriko 2y ago

no mention of the BOM... tragic.