3 ms·
Big-endian is the canonical form for storage and the wire because back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old
by api 9y ago
Big-endian is the canonical form for storage and the wire because back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc. All these are dead or dying now.
It's easy enough to deal with BE files and protocols by just swapping bytes. I'm referring to hardware architectures. Are there still any big-endian chips out there? Does it still make sense to support big-endian chips in new code as long as that code is never intended for embedded or very old systems?
- tlb 9y agoSwapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a tiny bit of care it can be.) For new code running in-core in 2018, I think little-endian is quite safe.
- api 9y agoYes, and I think it follows that new protocols and new file formats should be little-endian if possible.
- bitwize 9y ago> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you define a struct in C(++), there are not just endianness issues, but implementation-dependent issues of padding and alignment you have to consider. Recently, most compilers on most architectures standardized on self-alignment rules for all primitive types except char, but this is not guaranteed by the standard, it will bite you in the ass when you least expect it, and it will be decades yet before all the C code in the world is displaced by a single-implementation language like Rust. The best way to write portable code that will work as intended is to treat all data coming from disk or over the wire as a bag of bytes, and not attempt to alias it to a struct.
- api 9y agoMost compilers have directives for packing structs. What the OP suggests is something that can be done for high performance, but don't expect it to be portable outside of common architectures and compilers. I'd consider it something that shouldn't be done unless you really need the simplicity or performance.
- IshKebab 9y agoNo, dumping internal representations to disk is orthogonal to zero-static data formats. You can have a well defined format that doesn't require byte-by-byte parsing. Look up FlatBuffers and CapnProto.
- bitwize 9y agoBoth of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.
- kentonv 9y agoThat's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct. To call that "deferring the parsing to access time" does not make sense. (Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.) Your statement earlier: > The correct way to handle binary data is to unpack it into the struct byte by byte. This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof. On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works". (Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)
- drewg123 9y agoSmall pedantic correction from a former Alpha kernel hacker: DEC Alpha was Big Endian only for some rare Cray systems. Everything else was Little Endian.
- mattst88 9y agoI'm interested to know what you worked on on Alpha.
- drewg123 9y agoI first did a Myrinet driver for DEC OSF/1. I then worked on the FreeBSD port to DEC alpha. I helped with initial bringup and all the various issues around that. Then did a lot of the platform support, and the alpha-specific part of the Linux ABI compat layer, as well as the DEC OSF/1 binary compat layer. I ran a box on my desktop running FreeBSD/alpha as my primary workstation for years (API UP1000).
- deleted 9y ago[deleted]
- macintux 9y agoMany, many years ago I worked for Progeny, and we contracted with HP to help bring Debian to Itanium. We quickly discovered that a fair bit of Linux software (or at least Debian packaging of same) only worked properly on 64-bit DEC Alpha because the architecture was Little Endian; 32-bit pointers would typically "just work(™)" because the code was grabbing the first 32 bits, and the other half of the pointer addresses was typically 0x0. Since we were porting to Big Endian hardware (just now discovered Itanium was selectable, not sure why it was Big Endian in our case) we were getting a fair bit of null pointer exceptions. Anyway, ancient history, but that was my only direct exposure to endian-madness, so it has stuck with me.
- zlynx 9y agoHP wanted to replace PA-RISC and that was big-endian. Intel of course wanted little-endian. That's why Itanium was selectable.
- gumby 9y agoFWIW every one of the machines you cited (“Sparc, old MIPS, old PPC, DEC Alpha, etc”) post dated the formation of the Internet (the arpanet transitioning to TCP); the Internet protocols just followed existing arpanet practice. Which was due to big-endian processors being common, but the dominant networked machine of that era was the PDP-10.
- astrodust 9y agoThe PDP successor, the VAX, was strangely little-endian.
- watmough 9y agoCould Dave Cutler have foreseen ... nahhh!
- dasmoth 9y agoNot that strange — it was really the successor of the PDP-11, which was a quite different architecture.
- gumby 9y agoPDP-11 had a little endian architecture while big endian machines like the PDP-6/PDP-10 predominated on the net. To make life more interesting, the later PDP-10/20 mainframes used PDP-11 minicomputers as front end processors and often as network processors so byte-swapping was the norm. Luckily the PDP-10 allowed bytes ranging from 1-36 bits wide - “byte” had not yet standardized on 8 bits
- kazinator 9y agoBig-endian means that am ultra-high-speed routing fabric can receive the most significant part of an address address and possibly start making a routing decision before the other bytes of the frame have arrived. Hands down no brainer. Even the fact that headers come before payloads is a kind of "big endian", as is the fact that important information tends to occur earlier in headers. Look at a basic Ethernet frame. The destination address is first, so before even knowing the source address, or anything else, the switch can know where that frame will be sent.
- astrodust 9y agoMaybe that mattered on serial connections, but on a 10Gbit switch you're talking nonsense.
- dogecoinbase 9y agoNo, cut-through mode for packet forwarding (including on layer 3 prefix matches!) is a feature in current use on L2/3 10/40/100Gb switches, primarily in HFT/HPC environments.
- astrodust 9y agoIt's my understanding that this works with the complete address, not just the first byte, where it begins processing before the whole frame is received. On 100Gbit you're talking 0.01ns per bit, so 0.32ns for a full IPv4 address. This compared to, potentially, 15.24ns for a complete IPv4 frame.
- apaprocki 9y agoJust a small nit: the CPUs are all still actively developed and produced. For how much longer is anyone's guess. You shouldn't think of them as "very old systems". I run modern code happily on 64 CPU 4.3ghz POWER8 machines :)
- microtherion 9y ago> back when the Internet was designed most "pro" machines were big-endian: Sparc, old MIPS, old PPC, DEC Alpha, etc You kids sure know how to make someone feel really old... Much of the internet design predates the 68000, which in turn predates the RISC architectures you cite by half a decade and more.