4 ms·
One thing to realize is that UTF-8 is here and required only because we're all stuck with limitations based on the low-power, low-limitations of early systems.
by pierrebai 6y ago
One thing to realize is that UTF-8 is here and required only because we're all stuck with limitations based on the low-power, low-limitations of early systems.
There is not a single reason, other than historical, backward compatibility and the overall ecosystem putting incommensurate pressure on all future systems, why the basic memory unit is 8 bits.
We're stuck on 8 bits because all API, OS and specifications assume the basic units is 8 bits. If computer memory had evolved faster, or computer usage spread slower, the basic unit could have been 32 bits, or even 64 bits.
Think how much easier things would be. UTF could be encoded as a single unit, without ambiguous order.
What we've lost due to byte and UTF-8 is for example the text editor as the universal editor. Due to the need of UTF-8 decoding, we're no longer able to open a any (possibly binary) file as text and save it reliably after editing one character.
There are other consequences, for example at the file-system level for file names, corruption, how to handle invalid UTF-8 sequences, etc.
(And before you talk about the byte order problem, that is the point: there is no order problem for the basic unit you choose. No one (outside of some low-level 1-bit serial interfaces) cars about bit order or nibble order of bytes. Because the bit order is hidden within all interface accepting byte as the unit.)
- cesarb 6y agoIt's hard to argue against a counterfactual, but I believe there is a "sweet spot" for the size of the basic memory unit, and that "sweet spot" is probably in the range from 4 to 16 bits (power of two sizes are desirable, so the possible sizes would be 4 bits, 8 bits, and 16 bits), so even if computers evolved from scratch today already starting with large memories, the basic unit of memory addressing still wouldn't be that big. My guess is that the chosen size would be 16 bits.