3 ms·

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you ha

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

evincarofautumn 15y ago
No, that’s not endianness; endianness refers to the ordering of bytes within a multi-byte value—least significant byte first or most significant byte first, generally. The order of octets in a UTF-8 code point is fixed, and because a bit is not an addressable unit of memory, the storage order of bits within an octet is immaterial.
kijin 15y ago
It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16. If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings. http://en.wikipedia.org/wiki/Endianness http://en.wikipedia.org/wiki/Endianness
- sopooneo 15y ago
  Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?
  - ori_b 15y ago
    utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.
    - kijin 15y ago
      No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.
      - Tloewald 15y ago
        Yes but those two bytes will be in the same order regardless of the endianness of the system.
      - ori_b 15y ago
        Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit. This is not the case for UTF-16 and UTF-32.
        ybungalobill 15y ago
        AFAIK the correct term is "byte oriented".
  - lambda 15y ago
    Yes, basically. UTF-8 doesn't encode a code point as a single integer; it encodes it as a sequence of bytes, with a particular order, where some of the bits are used to represent the code point, and some of them are just used to represent whether you are looking at an initial byte or a continuation byte. I'd recommend checking out the description of UTF-8 in Wikipdia. The tables make it fairly clear how the encoding works: http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description
lambda 15y ago
No. You never need to know how to interpret the bits in each byte; you cannot address individual bits. Endianness refers to how different bytes within a multi-byte value are addressed; little endian means that the smaller addresses refer to the lower order bytes, big endian mean that smaller addresses refer to the higher order bytes. Think of it as which order you write the digits in a number. Each digit (byte) means the same thing regardless of whether you are big-endian or little-endian. But in big endian, you would write one thousand two hundred thirty four as 1234, while in little endian you would write it 4321.

evincarofautumn 15y ago

No, that’s not endianness; endianness refers to the ordering of bytes within a multi-byte value—least significant byte first or most significant byte first, generally. The order of octets in a UTF-8 code point is fixed, and because a bit is not an addressable unit of memory, the storage order of bits within an octet is immaterial.

kijin 15y ago

It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16. If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings. http://en.wikipedia.org/wiki/Endianness http://en.wikipedia.org/wiki/Endianness

sopooneo 15y ago
Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?
- ori_b 15y ago
  utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.
  - kijin 15y ago
    No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.
    - Tloewald 15y ago
      Yes but those two bytes will be in the same order regardless of the endianness of the system.
    - ori_b 15y ago
      Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit. This is not the case for UTF-16 and UTF-32.
      - ybungalobill 15y ago
        AFAIK the correct term is "byte oriented".
- lambda 15y ago
  Yes, basically. UTF-8 doesn't encode a code point as a single integer; it encodes it as a sequence of bytes, with a particular order, where some of the bits are used to represent the code point, and some of them are just used to represent whether you are looking at an initial byte or a continuation byte. I'd recommend checking out the description of UTF-8 in Wikipdia. The tables make it fairly clear how the encoding works: http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description

lambda 15y ago

No. You never need to know how to interpret the bits in each byte; you cannot address individual bits. Endianness refers to how different bytes within a multi-byte value are addressed; little endian means that the smaller addresses refer to the lower order bytes, big endian mean that smaller addresses refer to the higher order bytes. Think of it as which order you write the digits in a number. Each digit (byte) means the same thing regardless of whether you are big-endian or little-endian. But in big endian, you would write one thousand two hundred thirty four as 1234, while in little endian you would write it 4321.