6 ms·

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply wri

by wtbob 10y ago

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data. sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why? As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague. It's all so depressing.

morecoffee 10y ago
> Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. The thing is, as close to free as possible is surprisingly expensive. Protobuf's varint encoding is extremely branchy, and hurts performance in a datacenter environment (where bandwidth is free, and CPU is expensive). > As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague. Did they though? Arguably there are far more ARM CPUs (like the one in your pocket) than there are server CPUs. Since cellphones and other low power devices are almost all big endian, it seems like network byte order would have been better to use. High powered servers can pay the cost of coding them, but battery powered devices cannot afford to do so.
- niftich 10y ago
  ARM (since v3) is bi-endian for data accesses and defaults to little. You can confirm this by searching the ARM Information Center [1] for 'support for mixed-endian data', but I can't get a working URL for that exact page. [1] http://infocenter.arm.com/help/index.jsp http://infocenter.arm.com/help/index.jsp
- cmrdporcupine 10y ago
  ARM is bi-endian but my understanding is almost everybody is running it little-endian these days.
  - DonHopkins 10y ago
    The MIPS is bi-endian, and it's referred to as SPIM when driving backwards. For some reason, Sun never got around to making a bi-endian version of SPARC called CRAPS. Speaking of Sun, I have a confession to make: I'm bi-stellar. I love both Star Wars and Star Trek. [1] https://chortle.ccsu.edu/AssemblyTutorial/Chapter-15/ass15_4.html https://chortle.ccsu.edu/AssemblyTutorial/Chapter-15/ass15_4...
    - deleted 10y ago
      [deleted]
kentonv 10y ago
Big-endian vs. little-endian is an ancient flamewar that isn't going to go away any time soon, but sure, let's argue. Once you've spent as much time twiddling bits as I have (as the author of proto2 and Cap'n Proto), you start to realize that little endian is much easier to work with than big-endian. For example: - To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the pointer doesn't change. - Just about any arithmetic operation on integers (e.g. adding) starts from the least-significant bits and moves up. It's nice if that can mean iterating forward from the start instead of backwards from the end, e.g. when implementing a "bignum" library. - Which of the following is simpler? // Extract nth bit from byte array, assuming LE order. (bytes[n/8] >> (n%8)) & 1 // Extract nth bit from byte array, assuming BE order. (bytes[n/8] >> (7 - n%8)) & 1 There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing. I think the correct answer won here.
- teddyh 10y ago
  I thought the best summary of the big-endian/little-endian question was written in 1980 in Internet Experiment Note (IEN) 137: https://www.ietf.org/rfc/ien/ien137.txt https://www.ietf.org/rfc/ien/ien137.txt Also more reader-friendly here: https://www.computer.org/csdl/mags/co/1981/10/01667115.pdf https://www.computer.org/csdl/mags/co/1981/10/01667115.pdf
  - kentonv 10y ago
    I think figure 1 illustrates a common misunderstanding. People who object to little-endian are often imagining it in their heads as order (3): they imagine that the bits of each byte are ordered most-significant to least-significant (big-endian), but then for some reason the bytes are ordered the opposite, least-significant to most-significant (little-endian). That indeed would make no sense. But because most architectures don't provide any way to address individual bits, only bytes, it's entirely up to the observer to decide in which order they want to imagine the bits. When using little-endian, you imagine that the bits are in little-endian order, to be consistent with the bytes, and then everything is nice and consistent.
cyphar 10y ago
> > Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data. > sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why? Little endian means that a CPU designer can make buses shorter, which makes the CPU more efficient and smaller. There are also several benefits from the programming side. So it is actually better than big endian in /some/ cases, at the cost of being less intuitive to humans. So while Intel did choose little endian, they had very good reason to (and it's probably why everything except SystemZ and POWER use it).
- gpderetta 10y ago
  AFAIK, POWER has supported LE for a while and at least linux on Power has moved to LE by default.
  - deleted 10y ago
    [deleted]
- makomk 10y ago
  Hah. According to Wikipedia: "The Datapoint 2200 used simple bit-serial logic with little-endian to facilitate carry propagation. When Intel developed the 8008 microprocessor for Datapoint, they used little-endian for compatibility. However, as Intel was unable to deliver the 8008 in time, Datapoint used a medium scale integration equivalent, but the little-endianness was retained in most Intel designs." Which I guess is a reason to use little-endian, though not one that's really relevant to anyone that's ever actually used an Intel CPU. (ARM, incidentally, was little-endian by default because the 6502 was.)

by wtbob 10y ago

morecoffee 10y ago

> Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. The thing is, as close to free as possible is surprisingly expensive. Protobuf's varint encoding is extremely branchy, and hurts performance in a datacenter environment (where bandwidth is free, and CPU is expensive). > As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague. Did they though? Arguably there are far more ARM CPUs (like the one in your pocket) than there are server CPUs. Since cellphones and other low power devices are almost all big endian, it seems like network byte order would have been better to use. High powered servers can pay the cost of coding them, but battery powered devices cannot afford to do so.

niftich 10y ago
ARM (since v3) is bi-endian for data accesses and defaults to little. You can confirm this by searching the ARM Information Center [1] for 'support for mixed-endian data', but I can't get a working URL for that exact page. [1] http://infocenter.arm.com/help/index.jsp http://infocenter.arm.com/help/index.jsp
cmrdporcupine 10y ago
ARM is bi-endian but my understanding is almost everybody is running it little-endian these days.
- DonHopkins 10y ago
  The MIPS is bi-endian, and it's referred to as SPIM when driving backwards. For some reason, Sun never got around to making a bi-endian version of SPARC called CRAPS. Speaking of Sun, I have a confession to make: I'm bi-stellar. I love both Star Wars and Star Trek. [1] https://chortle.ccsu.edu/AssemblyTutorial/Chapter-15/ass15_4.html https://chortle.ccsu.edu/AssemblyTutorial/Chapter-15/ass15_4...
  - deleted 10y ago
    [deleted]

kentonv 10y ago

Big-endian vs. little-endian is an ancient flamewar that isn't going to go away any time soon, but sure, let's argue. Once you've spent as much time twiddling bits as I have (as the author of proto2 and Cap'n Proto), you start to realize that little endian is much easier to work with than big-endian. For example: - To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the pointer doesn't change. - Just about any arithmetic operation on integers (e.g. adding) starts from the least-significant bits and moves up. It's nice if that can mean iterating forward from the start instead of backwards from the end, e.g. when implementing a "bignum" library. - Which of the following is simpler? // Extract nth bit from byte array, assuming LE order. (bytes[n/8] >> (n%8)) & 1 // Extract nth bit from byte array, assuming BE order. (bytes[n/8] >> (7 - n%8)) & 1 There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing. I think the correct answer won here.

teddyh 10y ago
I thought the best summary of the big-endian/little-endian question was written in 1980 in Internet Experiment Note (IEN) 137: https://www.ietf.org/rfc/ien/ien137.txt https://www.ietf.org/rfc/ien/ien137.txt Also more reader-friendly here: https://www.computer.org/csdl/mags/co/1981/10/01667115.pdf https://www.computer.org/csdl/mags/co/1981/10/01667115.pdf
- kentonv 10y ago
  I think figure 1 illustrates a common misunderstanding. People who object to little-endian are often imagining it in their heads as order (3): they imagine that the bits of each byte are ordered most-significant to least-significant (big-endian), but then for some reason the bytes are ordered the opposite, least-significant to most-significant (little-endian). That indeed would make no sense. But because most architectures don't provide any way to address individual bits, only bytes, it's entirely up to the observer to decide in which order they want to imagine the bits. When using little-endian, you imagine that the bits are in little-endian order, to be consistent with the bytes, and then everything is nice and consistent.

cyphar 10y ago

> > Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data. > sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why? Little endian means that a CPU designer can make buses shorter, which makes the CPU more efficient and smaller. There are also several benefits from the programming side. So it is actually better than big endian in /some/ cases, at the cost of being less intuitive to humans. So while Intel did choose little endian, they had very good reason to (and it's probably why everything except SystemZ and POWER use it).

gpderetta 10y ago
AFAIK, POWER has supported LE for a while and at least linux on Power has moved to LE by default.
- deleted 10y ago
  [deleted]
makomk 10y ago
Hah. According to Wikipedia: "The Datapoint 2200 used simple bit-serial logic with little-endian to facilitate carry propagation. When Intel developed the 8008 microprocessor for Datapoint, they used little-endian for compatibility. However, as Intel was unable to deliver the 8008 in time, Datapoint used a medium scale integration equivalent, but the little-endianness was retained in most Intel designs." Which I guess is a reason to use little-endian, though not one that's really relevant to anyone that's ever actually used an Intel CPU. (ARM, incidentally, was little-endian by default because the 6502 was.)