7 ms·

It is amazing that big endian is almost dead. It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC. Main-core computing is vastly mor

It is amazing that big endian is almost dead. It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC. Main-core computing is vastly more homogenous than when I was born almost 50 years ago. I guess that's a natural progression for technology.

Aardwolf 1y ago
Now just UTF-16 and non '\n' newline types remaining to go
- hypeatei 1y ago
  UTF-16 will be quite the mountain as Windows APIs and web specifications/engines default to it for historical reasons.
  - int_19h 1y ago
    It's not just Windows and JavaScript. On Apple platforms, NSString is UTF-16. On Linux, Qt uses UTF-16 strings. Looking at languages, we have Java (which is where JS got this bug from) and C# both enshrining it in their respective language specs. So it's far more pervasive than people think, and will likely be in the picture for decades to come.
    - electroly 1y ago
      ICU (International Components for Unicode, the library published by the Unicode folks) itself uses UTF-16 internally, and most of these things are built on ICU. I agree strongly with your conclusion--UTF-16 isn't going anywhere. I don't think the ICU people are even talking about changing the internals to UTF-8.
    - account42 1y ago
      Qt could really change if they wanted to and really should have by now - it's not like they keep long-term backwards compatibility anyway unlike the others that you mentioned. Of course they chose to integrate JavaScript so that's less likely now.
- syncsynchalt 1y ago
  Of the two UTF-16 is much less of a problem, it's trivially[1] and losslessly convertible. [1] Ok I admit, not trivially when it comes to unpaired surrogates, BOMs, endian detection, and probably a dozen other edge and corner cases I don't even know about. But you can offload the work to pretty well-understood and trouble-free library calls.
  - Aardwolf 1y ago
    It causes such issues as opening of files by filename not being cross platform with standard libc functions since in Windows w-string versions are required since UTF-16 chars can have 0-bytes that aren't zero terminators
    - codedokode 1y ago
      It makes sense because UTF-16 is a sequence of uint16_t values, so both bytes must be zero, not just one.
      - syncsynchalt 1y ago
        Yes, it makes sense, but it still resulted in a lot of work. Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose. Even some multi-byte encodings would work if they chose to avoid using 0-value bytes for this reason. UTF-16LE/BE (and UTF-32 for that matter) chose not to allow for this, and the result is that if you want UTF-16 support in your existing C-string-based syscalls you need to make a second copy of every syscall which supports strings in your UTF-16 type of choice.
        codedokode 1y ago
        > Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose That's completely wrong. If a syscall (or a function) expects text in encoding A, you should not be sending it in encoding B because it would be interpreted incorrectly, or even worse, this would become a vulnerability. For every function, encoding must be specified as are specified the types of arguments, constraints and ownership rules. Sadly many open source libraries do not do it. How are you supposed to call a function when you don't know the expected encoding? Also, it is better to send a pointer and a length of the string rather than potentially infinitely search for a zero byte. > and the result is that if you want UTF-16 support in your existing C-string-based syscalls There is no need to support multiple encodings, it only makes things complicated. The simplest solution would be to use UTF-8 for all kernel facilities as a standard. For example, it would be better if open() syscall required valid UTF-8 string for a file name. This would leave no possibility for displaying file names as question marks.
- jeberle 1y ago
  UTF-16 arguably is Unicode 2.0+. It's how the code point address space is defined. Code points are either 1 or 2 16-bit code units. Easy. Compare w/ UTF-8 where a code point may be 1, 2, 3, or 4 8-bit code units. UTF-16 is annoying, but it's far from the biggest design failure in Unicode.
  - adgjlsfhk1 1y ago
    UTF-16 is the worst of all worlds. Either use UTF32 where code-points are fixed, or if you care about space efficiency use UTF8
    - mort96 1y ago
      UTF-32 is arguably even more worst of all worlds. You don't get fixed-size units in any meaningful way. Yes you have fixed sized code points, but those aren't the "units" you care about; you still have variable size grapheme clusters, so you still can't do things like reversing a string or splitting a string at an arbitrary index or anything else like that. Yet it consumes twice the space of UTF-16 for almost everything, and four times the space of UTF-8 for many things. UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything. As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.
      - kbolino 1y ago
        In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format. There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.
        1y ago
- augustk 1y ago
  > Now just UTF-16 and non '\n' newline types remaining to go Also ISO 8601 (YYYY-MM-DD) should be the default date format.
dgshsg 1y ago
We'll have to deal with it forever in network protocols. Thankfully that's rather walled off from most software.
- newpavlov 1y ago
  As well as in a number of widely spread cryptographic algorithms (e.g. SHA-2), which use BE for historic reasons.
  - NooneAtAll3 1y ago
    just call it 2-AHS and you're done :)
mococa 1y ago
Good call out, I have just removed some #ifdef about endianness from my engine.
- mort96 1y ago
  I have some places in some software where I assume little endian for simplicity, and I just leave in a static_assert(std::endian::native == std::endian::little) to let future me (or future someone else) know that a particular piece of code must be modified if it is ever to run on a not-little-endian machine.
  - mococa 1y ago
    Another good another call. Thank you.
  - account42 1y ago
    In an ideal world you could just write endian-independent code (i.e. read byte by byte) and leave the compiler optimizer to sort it out. This has the benefit of also not tripping up any alignment restrictions.
    - mort96 1y ago
      Here's my most recent use case: I have a relatively large array of uint16_t with highly repetitive (low entropy) data. I want to serialize that to disk, without wasting a lot of space. I run compress2 from zlib on the data when serializinsg it, and decompress it when deserializing. However, these files make sense to use between machines, so I have defined the file format to use compressed little endian 16-bit unsigned ints. Therefore, if you ever want to run this code on a big-endian machine, you need to add some code to first flip the bytes around before compressing, then flipping them back after decompressing. You're right that when your code is iterating through data byte for byte, you can write it in an endian-agnostic way and let the optimizer take care of recognizing that your shifts and ORs can be replaced with a memcpy on little-endian systems. But it's not always that simple.
goku12 1y ago
> It is amazing that big endian is almost dead. I wish the same applied to written numbers in LTR scripts. Arithmetic operations would be a lot easier to do that way on paper or even mentally. I also wish that the world would settle on a sane date-time format like the ISO 8601 or RFC 3339 (both of which would reverse if my first wish is also granted). > It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC. I never really understood those non-8-bit bytes, especially the 7 bit byte. If you consider the multiplexer and demux/decoder circuits that are used heavily in CPUs, FPGAs and custom digital circuits, the only number that really makes sense is 8. It's what you get for a 3 bit selector code. The other nearby values being 4 and 16. Why did they go for 7 bits instead of 8? I assume that it was a design choice made long before I was even born. Does anybody know the rationale?
- idoubtit 1y ago
  > I also wish that the world would settle on a sane date-time format like the ISO 8601 IIRC, in most countries the native format is D-M-Y (with varying separators), but some Asian countries use Y-M-D. Since those formats are easy to distinguish, that's no problem. That's why Y-M-D is spreading in Europe for official or technical documents. There's mainly one country which messes things up...
  - tavavex 1y ago
    YYYY-MM-DD is also the official date format in Canada, though it's not officially enforced, so outside of government documents you end up seeing a bit of all three formats all over the place. I've always used ISO 8601 and no one bats an eye, and it's convenient since YYYY-DD-MM isn't really a thing, so it can't be confused for anything else, unlike the other two formats.
  - zahlman 1y ago
    YMD has caught on, I think, because it allows for the numbers to be "in order" (not mixed-endian) while still having the month before the day which matches the practice for speaking dates in (at least) the US and Canada.
ndiddy 1y ago
Big endian will stay around as long as IBM continues to put in the resources to provide first-class Linux support on s390x. Of course if you don’t expect your software to ever be run on s390x you can just assume little-endian, but that’s already been the case for the vast majority of software developers ever since Apple stopped supporting PowerPC.
- metaphor 1y ago
  > ...that’s already been the case for the vast majority of software developers ever since Apple stopped supporting PowerPC. For better or worse, PowerPC is still quite entrenched in the industrial embedded space.
  - ndiddy 1y ago
    Right, and the vast majority of software developers aren't writing software intended to run on PowerPC industrial control boards.
chasil 1y ago
"...with the CPU running in big endian mode." Hey, you! You're supposed to be dead! https://wiki.netbsd.org/ports/evbarm/ https://wiki.netbsd.org/ports/evbarm/

Aardwolf 1y ago

Now just UTF-16 and non '\n' newline types remaining to go

hypeatei 1y ago
UTF-16 will be quite the mountain as Windows APIs and web specifications/engines default to it for historical reasons.
- int_19h 1y ago
  It's not just Windows and JavaScript. On Apple platforms, NSString is UTF-16. On Linux, Qt uses UTF-16 strings. Looking at languages, we have Java (which is where JS got this bug from) and C# both enshrining it in their respective language specs. So it's far more pervasive than people think, and will likely be in the picture for decades to come.
  - electroly 1y ago
    ICU (International Components for Unicode, the library published by the Unicode folks) itself uses UTF-16 internally, and most of these things are built on ICU. I agree strongly with your conclusion--UTF-16 isn't going anywhere. I don't think the ICU people are even talking about changing the internals to UTF-8.
  - account42 1y ago
    Qt could really change if they wanted to and really should have by now - it's not like they keep long-term backwards compatibility anyway unlike the others that you mentioned. Of course they chose to integrate JavaScript so that's less likely now.
syncsynchalt 1y ago
Of the two UTF-16 is much less of a problem, it's trivially[1] and losslessly convertible. [1] Ok I admit, not trivially when it comes to unpaired surrogates, BOMs, endian detection, and probably a dozen other edge and corner cases I don't even know about. But you can offload the work to pretty well-understood and trouble-free library calls.
- Aardwolf 1y ago
  It causes such issues as opening of files by filename not being cross platform with standard libc functions since in Windows w-string versions are required since UTF-16 chars can have 0-bytes that aren't zero terminators
  - codedokode 1y ago
    It makes sense because UTF-16 is a sequence of uint16_t values, so both bytes must be zero, not just one.
    - syncsynchalt 1y ago
      Yes, it makes sense, but it still resulted in a lot of work. Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose. Even some multi-byte encodings would work if they chose to avoid using 0-value bytes for this reason. UTF-16LE/BE (and UTF-32 for that matter) chose not to allow for this, and the result is that if you want UTF-16 support in your existing C-string-based syscalls you need to make a second copy of every syscall which supports strings in your UTF-16 type of choice.
      - codedokode 1y ago
        > Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose That's completely wrong. If a syscall (or a function) expects text in encoding A, you should not be sending it in encoding B because it would be interpreted incorrectly, or even worse, this would become a vulnerability. For every function, encoding must be specified as are specified the types of arguments, constraints and ownership rules. Sadly many open source libraries do not do it. How are you supposed to call a function when you don't know the expected encoding? Also, it is better to send a pointer and a length of the string rather than potentially infinitely search for a zero byte. > and the result is that if you want UTF-16 support in your existing C-string-based syscalls There is no need to support multiple encodings, it only makes things complicated. The simplest solution would be to use UTF-8 for all kernel facilities as a standard. For example, it would be better if open() syscall required valid UTF-8 string for a file name. This would leave no possibility for displaying file names as question marks.
jeberle 1y ago
UTF-16 arguably is Unicode 2.0+. It's how the code point address space is defined. Code points are either 1 or 2 16-bit code units. Easy. Compare w/ UTF-8 where a code point may be 1, 2, 3, or 4 8-bit code units. UTF-16 is annoying, but it's far from the biggest design failure in Unicode.
- adgjlsfhk1 1y ago
  UTF-16 is the worst of all worlds. Either use UTF32 where code-points are fixed, or if you care about space efficiency use UTF8
  - mort96 1y ago
    UTF-32 is arguably even more worst of all worlds. You don't get fixed-size units in any meaningful way. Yes you have fixed sized code points, but those aren't the "units" you care about; you still have variable size grapheme clusters, so you still can't do things like reversing a string or splitting a string at an arbitrary index or anything else like that. Yet it consumes twice the space of UTF-16 for almost everything, and four times the space of UTF-8 for many things. UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything. As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.
    - kbolino 1y ago
      In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format. There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.
      - 1y ago
augustk 1y ago
> Now just UTF-16 and non '\n' newline types remaining to go Also ISO 8601 (YYYY-MM-DD) should be the default date format.

dgshsg 1y ago

We'll have to deal with it forever in network protocols. Thankfully that's rather walled off from most software.

newpavlov 1y ago
As well as in a number of widely spread cryptographic algorithms (e.g. SHA-2), which use BE for historic reasons.
- NooneAtAll3 1y ago
  just call it 2-AHS and you're done :)

mococa 1y ago

Good call out, I have just removed some #ifdef about endianness from my engine.

mort96 1y ago
I have some places in some software where I assume little endian for simplicity, and I just leave in a static_assert(std::endian::native == std::endian::little) to let future me (or future someone else) know that a particular piece of code must be modified if it is ever to run on a not-little-endian machine.
- mococa 1y ago
  Another good another call. Thank you.
- account42 1y ago
  In an ideal world you could just write endian-independent code (i.e. read byte by byte) and leave the compiler optimizer to sort it out. This has the benefit of also not tripping up any alignment restrictions.
  - mort96 1y ago
    Here's my most recent use case: I have a relatively large array of uint16_t with highly repetitive (low entropy) data. I want to serialize that to disk, without wasting a lot of space. I run compress2 from zlib on the data when serializinsg it, and decompress it when deserializing. However, these files make sense to use between machines, so I have defined the file format to use compressed little endian 16-bit unsigned ints. Therefore, if you ever want to run this code on a big-endian machine, you need to add some code to first flip the bytes around before compressing, then flipping them back after decompressing. You're right that when your code is iterating through data byte for byte, you can write it in an endian-agnostic way and let the optimizer take care of recognizing that your shifts and ORs can be replaced with a memcpy on little-endian systems. But it's not always that simple.

goku12 1y ago

> It is amazing that big endian is almost dead. I wish the same applied to written numbers in LTR scripts. Arithmetic operations would be a lot easier to do that way on paper or even mentally. I also wish that the world would settle on a sane date-time format like the ISO 8601 or RFC 3339 (both of which would reverse if my first wish is also granted). > It will be relegated to the computing dustbin like non-8-bit bytes and EBCDIC. I never really understood those non-8-bit bytes, especially the 7 bit byte. If you consider the multiplexer and demux/decoder circuits that are used heavily in CPUs, FPGAs and custom digital circuits, the only number that really makes sense is 8. It's what you get for a 3 bit selector code. The other nearby values being 4 and 16. Why did they go for 7 bits instead of 8? I assume that it was a design choice made long before I was even born. Does anybody know the rationale?

idoubtit 1y ago
> I also wish that the world would settle on a sane date-time format like the ISO 8601 IIRC, in most countries the native format is D-M-Y (with varying separators), but some Asian countries use Y-M-D. Since those formats are easy to distinguish, that's no problem. That's why Y-M-D is spreading in Europe for official or technical documents. There's mainly one country which messes things up...
- tavavex 1y ago
  YYYY-MM-DD is also the official date format in Canada, though it's not officially enforced, so outside of government documents you end up seeing a bit of all three formats all over the place. I've always used ISO 8601 and no one bats an eye, and it's convenient since YYYY-DD-MM isn't really a thing, so it can't be confused for anything else, unlike the other two formats.
- zahlman 1y ago
  YMD has caught on, I think, because it allows for the numbers to be "in order" (not mixed-endian) while still having the month before the day which matches the practice for speaking dates in (at least) the US and Canada.

ndiddy 1y ago

Big endian will stay around as long as IBM continues to put in the resources to provide first-class Linux support on s390x. Of course if you don’t expect your software to ever be run on s390x you can just assume little-endian, but that’s already been the case for the vast majority of software developers ever since Apple stopped supporting PowerPC.

metaphor 1y ago
> ...that’s already been the case for the vast majority of software developers ever since Apple stopped supporting PowerPC. For better or worse, PowerPC is still quite entrenched in the industrial embedded space.
- ndiddy 1y ago
  Right, and the vast majority of software developers aren't writing software intended to run on PowerPC industrial control boards.

chasil 1y ago

"...with the CPU running in big endian mode." Hey, you! You're supposed to be dead! https://wiki.netbsd.org/ports/evbarm/ https://wiki.netbsd.org/ports/evbarm/