8 ms·
Endianness, a constant source of conflict for decades
- bitwize 5y agoLittle endian won. You almost don't have to worry about a new piece of software running on a big-endian machine.
- minusf 5y agowhich is a loss, as according to openbsd developers developing for both caught a lot of bugs otherwise not found easily..
- edflsafoiewq 5y agoGive us an example.
- saurik 5y ago(I mean it just seems obvious you would catch more bugs given the format of big-endian, right?... it comes directly from the dual of one of the not-really-a-benefits of little-endian: how you can cast between pointers of different integer types without moving the pointer.)
- agent327 5y agoI'm totally onboard with casting integers, and I can understand casting between pointers-to-char and pointers-to-other, but why on earth would you want to cast between pointers to (different) integers? That seems like asking for trouble, for no benefit I can discern...
- masklinn 5y agohttps://news.ycombinator.com/item?id=28218704 https://news.ycombinator.com/item?id=28218704
- iainmerrick 5y agoCouldn’t you keep that benefit by testing on emulated big-endian hardware, though? (edit to fix interesting autocorrect glitch... “bug-endian” indeed!)
- iainmerrick 5y agoApart from most file formats and internet standards being big-endian, you mean? Although to borrow from minusf’s point, it’s good for software robustness that file formats and hardware use different endianness, as it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32.
- walki 5y ago> it’s good for software robustness that file formats and hardware use different endianness, as it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32. Except that it is very bad for performance. As far as CPUs are concerned little-endian has definitely won, most CPU architectures that have been big endian in the past (e.g. PowerPC) are now little endian by default. If all new CPU architectures are little endian this means that within a decade or two there won't be any operating systems that support big endian anymore.
- iainmerrick 5y agoFor performance, can’t you “just” have a swap-endianness instruction in your CPU, and have the compiler use it when it detects byte-shuffling code? (That may even happen already on some architectures for all I know)
- walki 5y ago> For performance, can’t you “just” have a swap-endianness instruction in your CPU Yes, most CPUs have special instructions for swapping between little and big endian byte arrangement. The GCC compiler has the __builtin_bswap64(x) for accessing this instruction. However this is an additional instruction that needs to be executed for each read of a 64-bit word that needs to be converted, in some workloads this can double the number of executed instructions and hence add significant overhead. Supporting big endian CPUs in systems programming sucks beyond imagination. There are virtually no big endian users anymore and making sure your software works fine on big endian requires testing it on a big endian CPU. However it is not possible to buy a big endian CPU anymore as there exist no more consumer big endian CPUs. For this reason I still have a Mac PowerPC from 2003 at home running an ancient version of Mac OS X. But over the last 2 years I have stopped testing my software on big endian, I just don't care about big endian anymore...
- Miiko 5y agoProbably because English is not my native language, but it always looked to me that the names are backward: * "big-endian" should have "big" (most significant) part on the end * and "little-endian" should have "little" (least significant) bits at the end Is there different mnemonics to remember what is what?
- yetihehe 5y agoTry with original source [0]. Big-endian, because some Lilliputians eat eggs starting from big end, little-endian, because some start from little end. Works for me. [0] https://en.wikipedia.org/wiki/Endianness#Etymology https://en.wikipedia.org/wiki/Endianness#Etymology
- Miiko 5y agoThanks! now that makes sense - indeed, "endian" refer not to what's on the end, but from which end we start writing the number.
- kevin_thibedeau 5y agoThe little end is at the lower address for LE.
- yetihehe 5y agoUnless you show your memory layout on picture, where adress 0 is on top.
- kevin_thibedeau 5y agoStill numerically lower.
- anyfoo 5y agoIt's not, a 16bit LE value at address 0 will have the low byte at address 0 and the high byte at 1. Unless I misunderstood what you meant with "little end"? (But then there is still room for confusion in that term, apparently.)
- kazinator 5y agoUnder little endian, the same power digits of differently sized operands are at the same offsets. For instance, whether we have a 16 bit operand or 128 bit operand, the least signficant 8 bits of either one are at the lowest address and so on. This is important if we want to, say, add them together. However, this effect can hide bugs under little endian, which will instantly reproduce on big endian. Suppose that, say, a function expects a 32 bit parameter, but the caller thinks it is passing a byte, whose value is XX. Suppose that by fluke the memory is all zeros. Under little endian, the caller puts XX at the right memory location in the stack, resulting in XX 00 00 00. And, by golly, the callee gets the corect 32 bit value XX. Under big endian, even if by fluke the memory is all zeros, the caller will put the XX byte resulting in the same XX 00 00 00. But this now looks like a huge 32 bit value to the callee, hopefully caught in testing. The apparently correct value will not be caught in testing. Little endian would need nonzero values in the extra bytes instead of the fluky zeros in order to see a bad value.
- swiley 5y agoYes, we should have stuck with Big-endian.
- dataflow 5y agoHexadecimal data looks very confusing with big endian. 01234567 somehow becomes equal to 4567 0123 and 67 45 23 01... pretty darn out-of-order and unnatural.
- Lvl999Noob 5y agoIsn't that the little endian representation? I thought big endian starts with the most significant bit. Here, I think 0 would be the most significant nibble and would be written at the left most point.
- dataflow 5y agoSorry, I should clarify. I meant what you would see in a hex editor would turn out like that. i.e. if you had the byte sequence 67 45 23 01, and decided to display it in 2-byte words, LE could just display 6745 2301; you'd know the first byte is still the least significant regardless of the grouping, and you can just regard the spacing as a visual grouping aid (pretty natural). If you tried e.g. "Go To -> Offset 3", you'd still land on the 01, just as if you went right by 3 positions visually... pretty intuitive. Compare that with BE, where it'd have to show you 4567 0123, and suddenly if you Go To -> Offset 3 ('01'), you'd land to the left of offset 2 ('23'), which to me seems super confusing.
- vardump 5y agoYou're getting your endians confused. Bytes 67 45 23 01 interpreted as 16-bit words in: Little endian, LE, or what x86 uses: 4567 0123 Big endian, BE, or what 68k uses: 6745 2301
- dataflow 5y agoI'm not saying the same thing you think I'm saying, but I do realize I'm definitely explaining what I mean poorly... likely because hex editors themselves have multiple behaviors in this regard. Some merely use spaces to separate bytes visually, others actually group the bytes and parse them as integers, swapping them as needed to read like human writing. Let me try illustrating a different situation, hopefully without that confusion. Let's say your data starts with the byte sequence 67 45 23 01... If you assume these represents some LE numbers, and want to multiply by 256 (decimal), you end up with 00 67 45 23 01... it really doesn't matter (and you don't need to know) what the word sizes were. That's the only sane result, and the byte at offset 2 would end up being 45h... end of story. Even if your number was only supposed to be N bytes and now it's N+1 bytes, you can just chop it back to N bytes and your result will still be correct (modulo 256^N) and as intuitive as it could be. But if you start working in BE, suddenly things get confusing fast. Imagine what this operation would be for 2-byte BE words. The first word in BE is 6745 and now becomes 674500, and you overflowed by 1 byte. So which part do you keep and which part do you overflow to the next word? If you keep the 6745, then the 00 ends up affecting the second word rather than the first one, which is just completely nonsensical. The other option is to keep 4500 and and shove the 67 into the next word, turning it from 2301 into 230167. Now you have to repeat the same procedure with the 23, etc. until you reach the end of the data. Now look at what just happened in the BE case. You have the bizarre situation where your words are internally BE (i.e. "go to offset 0" would now land on the 00 byte, which are not the first 2 characters in the editor!). And across words, they're still treated like LE—the inter-word overflows are still moving bytes to higher-order words on the right, not the left! There's just no sane way to do math with N-byte words and avoid LE entirely; even if you treat each word as BE, you're absolutely forced to treat the word sequence as LE. The only way to actually avoid all LE is to interpret the whole thing as 1 gigantic bignum, where "multiply by 256" ends up being translated into "append 00 to the end of the stream". That's great if your data really was 1 gigantic bignum, but not so much if your data was just typical ints or longs. If this is still confusing (I realize it might be) then I'm not sure how else to put my thoughts into words unfortunately (no pun intended). Hopefully you can kind of see what I'm getting at though, even if I'm explaining some portions of it poorly (sorry).
- KingOfCoders 5y ago"there is a distinct advantage to writing numbers in little endian order." Only if you write left to right, and not right to left.
- deleted 5y ago[deleted]
- nybble41 5y agoIf you write right-to-left and use little-endian writing order then you still write the least significant digit first, so the one's places line up along the right edge. The advantage is the same.
- dandanua 5y agoNice explanation and examples. I also wrote about this problem in a more broader context of mathematics and quantum computing [1] (it's not displayed nicely in Firefox for some reason). [1] https://github.com/dandanua/little-endian-vs-big-endian-in-quantum-computing/blob/master/txt.pdf https://github.com/dandanua/little-endian-vs-big-endian-in-q...
- nly 5y agoDoes it matter? Most good static languages have a type system strong enough to express endianness as part of the data type, making conversion transparent
- BiteCode_dev 5y agoIt matters for protocols and data formats.
- a_t48 5y agoPutting aside “good” vs “popular”, static typing doesn’t save you from accidentally tagging the wrong endianness for your external data sources. And it matters if you’re doing math with your CPUs native type.
- nly 5y agoIf you only have one definition of your data atructure and that definition encodes the endianness you're done, and you cannot misuse it.
- PennRobotics 5y agoFor those unfamiliar with German, they read numbers aloud in a unique way. 65536 is "sixty five thousand five hundred six and thirty" (Edit: this is inaccurate). I present German-endian as a common enemy! ----- At least for me, reading 16- and/or 32-bit multi-channel sensor data and then transceiving via 8-bit radio (atmega32u4) is at least a little easier when all the endians align. No byte swapping. No jumping a pointer ahead for each byte in a packet. Most importantly, no calling swapbytes (and its relevant data casts) in Matlab reading the serial data directly off the microcontroller. While this is all relatively straightforward to code, each extra line is another tiny chance of error, and the failure modes in streaming data are not always obvious e.g. FIFO size is a power of two but channel data is an odd number of bytes/words. Do you drop data or pause until FIFO empties? Do circular buffers increment or decrement? Luckily, modern sensors (e.g. IMUs) I've used usually have registers to byte swap, drop LSBs when not needed, choose right or left zero bit padding, change channel order, alter FIFO behavior, interrupt at a FIFO threshold, and so on.
- TacticalMalice 5y ago"five and sixty thousand five hundred six and thirty", right? Dutch is similar and this is a source of mistakes when writing down (phone) numbers. I've resorted to calling out the digits in LTR order.
- junon 5y agoCorrect. The grouping matters, and double-digits in groupings are also reversed - "fünf und sechzig tausend fünf hundert sechs und dreißig". Another thing, years in German aren't spoken as "twenty twenty-one" as we commonly do in English, but instead the number is spoken out fully - "two-thousand one and twenty" ("zwei tausend eins und zwanzig").
- twic 5y agoIn England, the little voice in Google Maps told me to take the "B one thousand, one hundred and thirteen", when every human i know would call that road the "B one one one three".
- baybal2 5y agoBig endian, little endian, all pale in comparison to the horror of mixed endian people have in embedded.
- scratcheee 5y ago>[big endian] Binary dumps look more in line with how humans with left-to-right scripts expect to read numbers. Maybe I'm just dumb, but surely the actual language order is entirely irrelevant? If hypothetically we wrote in English right-to-left instead, then we'd write our numbers right-to-left, and our memory dumps right-to-left, so then we'd find that little-endian caused the data to start with the smallest byte first (on the right). Mirroring the language doesn't undo a mirroring within the language, and that's what little-endian is, so a more accurate statement would be: >Binary dumps look more in line with how humans with single-direction scripts expect to read numbers.
- cies 5y ago> If hypothetically we wrote in English right-to-left instead, then we'd write our numbers right-to-left If you read the article the author shows you that in RTL langs (where our current number system originated from) the numbers were also RTL. We just stuck to the convention. Interesting how this little bit of RTL snuck into Europes otherwise LTR languages to the extend that when i type a number in a spreadsheet it changes the allignment to with this... Interesting/insightful article!
- thaumasiotes 5y ago> If you read the article the author shows you that in RTL langs (where our current number system originated from) the numbers were also RTL. We just stuck to the convention. Well, no, the author says this: > Our modern numbering system has its roots in the Hindu numbering system, which was invented somewhere between the 1st and 4th century. Like the dominant writing system of the time, numbers were written right-to-left This is not obvious - it appears that there was a right-to-left Indic script centered around Pakistan ( https://en.wikipedia.org/wiki/Kharosthi https://en.wikipedia.org/wiki/Kharosthi ) and a left-to-right one ( https://en.wikipedia.org/wiki/Brahmi_script https://en.wikipedia.org/wiki/Brahmi_script ) farther south / east. Before that, Sanskrit was written from left to right. It seems far more likely to me in any event that the order in which numbers are written, when the system is innovated, will reflect the order in which they are spoken in whatever language, not directly the order in which the language is written down. Over time they will always develop a big-endian order, because that allows sorting them.
- rkangel 5y ago> Most network protocols and formats until recently have been big endian (Network Byte Order). This misses the point of the original reason for this - big endian is more convenient if you're reading the value into a shift register. You don't need to know in advance how big the value is because you just shift the contents left each time you get a new byte and you end up with the appropriate 0 padded value. Basically for a period of history, big endian was easier to implement for comms and little endian for processors. The big endian comms reasons have generally died out while the processor ones remain. Network byte order (big endian in communication) is a very strong convention though. If you define a network protocol and use little endian then I will be very sad.
- AnIdiotOnTheNet 5y agoWhy? If I'm designing my own protocol I could use middle-endian EBCDIC encoded octal and it shouldn't make any difference to anything else.
- rkangel 5y agoAre you building every implementation of the protocol? Or are you publishing a communications standard for people to implement? If I'm building a network stack and at every layer I'm pulling values out of the header in network byte order, except in one special case where I have to pull them the other way (for no good reason) then that's the protocol author's fault. If there is a good reason for using middle-Endian EBCDIC (compatibility or technical) then fine. If not, then please use the thing that's the strong convention and therefore simplest to work with.
- dahfizz 5y agoAs long as you don't want anyone to adopt the protocol, then go ahead.
- ghusbands 5y agoBig-endian is how we write each byte in a hex editor, and so big-endian should be more intuitive. If we have the number 0xDEADBEEF and view it in a memory dump, a big-endian system gives: DEADBEEF And a little-endian system gives: FEBEADDE Only one of those reads naturally. As long as your increasing memory order goes from left to right, as is the norm.
- AnIdiotOnTheNet 5y ago> FEBEADDE 0xDEADBEFE?
- recursive 5y agoAny significance to `BE`? It's the only byte that remains untouched.
- api 5y agoLittle-endian is superior at the system level largely because casts between integer sizes are free. Casting a 32-bit LE value to an 8-bit value is just reinterpreting what the pointer means. This is a fairly common operation. Big-endian is pretty much dead. I am not aware of any BE systems on sale right now outside small MIPS routers and embedded. If the Internet were designed today network byte order would probably be little-endian. New cryptographic algorithms are being designed little-endian-first.
- swiley 5y agoIt also hides errors that come from casting between different pointers types.
- api 5y agoThat's the job of the language/compiler. Rust does a fine job of this.
- classichasclass 5y agoOpenPOWER systems (POWER8 and up) swing both ways. That said, my POWER9 Raptor Talos II runs little endian, though it is completely capable of running big (and at the low OPAL level in fact does), and some operating systems on it support that.
- ithkuil 5y ago> Binary dumps look more in line with how humans with left-to-right scripts expect to read numbers. I remember VMS EXAMINE command but also printed DEC manuals showing hex dumps where the columns where numbered RTL. The ascii view on the right half of the hex dump was following the LTR order, so basically the two views were mirrored. (see example at http://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/4556/4556p004.htm http://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/4556/4556p004....) With such a rendering, little endian does indeed natural. We're doing it all the time when rendering bit positions: bit pos: 3210 bit val: 1100 extending this layout for byte indices is quite natural indeed.
- rwmj 5y agoWhen POWER 7 moved to LE, little endian essentially won [edit: for CPUs, not for network protocols]. The only CPU architecture we support that is still big endian is s/390.
- classichasclass 5y agoThe little endian mode in POWER7 was not frequently used and I believe has limitations. It's officially a thing for POWER8 and up. That said, my POWER9 runs little, though it's perfectly capable of running big.
- renox 5y agoThe thing to remember is that things starts by 'oral/spoken' numbers then by written numbers. When we speak we don't really use big endian nor little endian, we say 'six thousand one hundred' not 'six one zero zero' and obviously we prefer to start with the 'big' part because it's the most important for the listener: I don't really care if your price is '6 thousand and one' or '6 thousand and two', the listener hear '6 thousand' and then switch off.. And then written language followed oral language of course..