4 ms·

Apart from most file formats and internet standards being big-endian, you mean? Although to borrow from minusf’s point, it’s good for software robustness that

Apart from most file formats and internet standards being big-endian, you mean? Although to borrow from minusf’s point, it’s good for software robustness that file formats and hardware use different endianness, as it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32.

walki 5y ago
> it’s good for software robustness that file formats and hardware use different endianness, as it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32. Except that it is very bad for performance. As far as CPUs are concerned little-endian has definitely won, most CPU architectures that have been big endian in the past (e.g. PowerPC) are now little endian by default. If all new CPU architectures are little endian this means that within a decade or two there won't be any operating systems that support big endian anymore.
- iainmerrick 5y ago
  For performance, can’t you “just” have a swap-endianness instruction in your CPU, and have the compiler use it when it detects byte-shuffling code? (That may even happen already on some architectures for all I know)
  - walki 5y ago
    > For performance, can’t you “just” have a swap-endianness instruction in your CPU Yes, most CPUs have special instructions for swapping between little and big endian byte arrangement. The GCC compiler has the __builtin_bswap64(x) for accessing this instruction. However this is an additional instruction that needs to be executed for each read of a 64-bit word that needs to be converted, in some workloads this can double the number of executed instructions and hence add significant overhead. Supporting big endian CPUs in systems programming sucks beyond imagination. There are virtually no big endian users anymore and making sure your software works fine on big endian requires testing it on a big endian CPU. However it is not possible to buy a big endian CPU anymore as there exist no more consumer big endian CPUs. For this reason I still have a Mac PowerPC from 2003 at home running an ancient version of Mac OS X. But over the last 2 years I have stopped testing my software on big endian, I just don't care about big endian anymore...
- classichasclass 5y ago
  Power ISA is actually still big by default, and even on little-endian capable systems starts big endian until changed by the OS. Low-level OPAL calls on OpenPOWER systems are made big endian, even if the OS is little (the OS has to switch the processor mode).
flohofwoe 5y ago
> as it forces you to read things byte-by-byte Indeed, reading file headers byte by byte also avoids alignment issues on some CPUs. At least older ARM CPUs trapped misaligned reads (not sure if this is still the case though).
- walki 5y ago
  > not sure if this is still the case though No this is not the case anymore. Nowadays support for unaligned memory accesses is very good on ARM and most other CPU architectures. On x86 aligned memory used to be very important for SIMD but now there are even special SIMD instructions for unaligned data and the performance overhead of unaligned memory accesses is generally very small in my experience.
foxfluff 5y ago
> it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32 Why is this good? How does the extra work make software more robust?

walki 5y ago

> it’s good for software robustness that file formats and hardware use different endianness, as it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32. Except that it is very bad for performance. As far as CPUs are concerned little-endian has definitely won, most CPU architectures that have been big endian in the past (e.g. PowerPC) are now little endian by default. If all new CPU architectures are little endian this means that within a decade or two there won't be any operating systems that support big endian anymore.

iainmerrick 5y ago
For performance, can’t you “just” have a swap-endianness instruction in your CPU, and have the compiler use it when it detects byte-shuffling code? (That may even happen already on some architectures for all I know)
- walki 5y ago
  > For performance, can’t you “just” have a swap-endianness instruction in your CPU Yes, most CPUs have special instructions for swapping between little and big endian byte arrangement. The GCC compiler has the __builtin_bswap64(x) for accessing this instruction. However this is an additional instruction that needs to be executed for each read of a 64-bit word that needs to be converted, in some workloads this can double the number of executed instructions and hence add significant overhead. Supporting big endian CPUs in systems programming sucks beyond imagination. There are virtually no big endian users anymore and making sure your software works fine on big endian requires testing it on a big endian CPU. However it is not possible to buy a big endian CPU anymore as there exist no more consumer big endian CPUs. For this reason I still have a Mac PowerPC from 2003 at home running an ancient version of Mac OS X. But over the last 2 years I have stopped testing my software on big endian, I just don't care about big endian anymore...
classichasclass 5y ago
Power ISA is actually still big by default, and even on little-endian capable systems starts big endian until changed by the OS. Low-level OPAL calls on OpenPOWER systems are made big endian, even if the OS is little (the OS has to switch the processor mode).

flohofwoe 5y ago

> as it forces you to read things byte-by-byte Indeed, reading file headers byte by byte also avoids alignment issues on some CPUs. At least older ARM CPUs trapped misaligned reads (not sure if this is still the case though).

walki 5y ago
> not sure if this is still the case though No this is not the case anymore. Nowadays support for unaligned memory accesses is very good on ARM and most other CPU architectures. On x86 aligned memory used to be very important for SIMD but now there are even special SIMD instructions for unaligned data and the performance overhead of unaligned memory accesses is generally very small in my experience.

foxfluff 5y ago

> it forces you to read things byte-by-byte rather than lazily assuming you can just read 4 bytes and cast them directly to an int32 Why is this good? How does the extra work make software more robust?