4 ms·

Isn't that the little endian representation? I thought big endian starts with the most significant bit. Here, I think 0 would be the most significant nibble an

Isn't that the little endian representation? I thought big endian starts with the most significant bit. Here, I think 0 would be the most significant nibble and would be written at the left most point.

dataflow 5y ago
Sorry, I should clarify. I meant what you would see in a hex editor would turn out like that. i.e. if you had the byte sequence 67 45 23 01, and decided to display it in 2-byte words, LE could just display 6745 2301; you'd know the first byte is still the least significant regardless of the grouping, and you can just regard the spacing as a visual grouping aid (pretty natural). If you tried e.g. "Go To -> Offset 3", you'd still land on the 01, just as if you went right by 3 positions visually... pretty intuitive. Compare that with BE, where it'd have to show you 4567 0123, and suddenly if you Go To -> Offset 3 ('01'), you'd land to the left of offset 2 ('23'), which to me seems super confusing.
- vardump 5y ago
  You're getting your endians confused. Bytes 67 45 23 01 interpreted as 16-bit words in: Little endian, LE, or what x86 uses: 4567 0123 Big endian, BE, or what 68k uses: 6745 2301
  - dataflow 5y ago
    I'm not saying the same thing you think I'm saying, but I do realize I'm definitely explaining what I mean poorly... likely because hex editors themselves have multiple behaviors in this regard. Some merely use spaces to separate bytes visually, others actually group the bytes and parse them as integers, swapping them as needed to read like human writing. Let me try illustrating a different situation, hopefully without that confusion. Let's say your data starts with the byte sequence 67 45 23 01... If you assume these represents some LE numbers, and want to multiply by 256 (decimal), you end up with 00 67 45 23 01... it really doesn't matter (and you don't need to know) what the word sizes were. That's the only sane result, and the byte at offset 2 would end up being 45h... end of story. Even if your number was only supposed to be N bytes and now it's N+1 bytes, you can just chop it back to N bytes and your result will still be correct (modulo 256^N) and as intuitive as it could be. But if you start working in BE, suddenly things get confusing fast. Imagine what this operation would be for 2-byte BE words. The first word in BE is 6745 and now becomes 674500, and you overflowed by 1 byte. So which part do you keep and which part do you overflow to the next word? If you keep the 6745, then the 00 ends up affecting the second word rather than the first one, which is just completely nonsensical. The other option is to keep 4500 and and shove the 67 into the next word, turning it from 2301 into 230167. Now you have to repeat the same procedure with the 23, etc. until you reach the end of the data. Now look at what just happened in the BE case. You have the bizarre situation where your words are internally BE (i.e. "go to offset 0" would now land on the 00 byte, which are not the first 2 characters in the editor!). And across words, they're still treated like LE—the inter-word overflows are still moving bytes to higher-order words on the right, not the left! There's just no sane way to do math with N-byte words and avoid LE entirely; even if you treat each word as BE, you're absolutely forced to treat the word sequence as LE. The only way to actually avoid all LE is to interpret the whole thing as 1 gigantic bignum, where "multiply by 256" ends up being translated into "append 00 to the end of the stream". That's great if your data really was 1 gigantic bignum, but not so much if your data was just typical ints or longs. If this is still confusing (I realize it might be) then I'm not sure how else to put my thoughts into words unfortunately (no pun intended). Hopefully you can kind of see what I'm getting at though, even if I'm explaining some portions of it poorly (sorry).
    - nybble41 5y ago
      > Some merely use spaces to separate bytes visually, others actually group the bytes and parse them as integers, swapping them as needed to read like human writing. In other words, your problem is that your hex editor is incorrectly assuming that the numbers are little-endian. If it interpreted them as big-endian then nothing would be swapped since the order of the bytes matches the standard conventions for numbers in most European writing systems. This is not a big-endian problem, it's a little-endian problem. A decent hex editor will allow you to set the byte order to match your file. (And as the article points out, the issue would be reversed if the bytes were displayed right-to-left following the same conventions that we use for numbers… but then all your strings would be reversed.) Both representations have strengths and weaknesses depending on what you want to do. Most arbitrary-precision math works better with LE. On the other hand, hexadecimal string formatting works better with the BE encoding, where LE would require either the input or output to be reversed.
      - dataflow 5y ago
        No, this has nothing to do with the editor or bignums. I'm just using those to illustrate the underlying inconsistency. The most natural order/sequence out there is that of natural numbers with zero (aka whole numbers), i.e. N0 = 0, 1, 2, 3, ... So I'm basically arguing that means the most digit places are in the same order, i.e. the coefficients would be ordered as 256^N i.e. 256^0, 256^1, 256^2, 256^3, ...
        nybble41 5y ago
        I was addressing your first point about how the data is displayed in hex editors. But the rest IMHO is just fixating on one particular operation ("multiplying by 256" by prepending bytes without regard for the data format) which does not seem particularly useful or prudent. Sure, in LE you can prepend a 00 byte to multiply by 256. In BE you do the same by appending a 00 byte at the end, and a 00 at the beginning would just be extra padding, exactly like adding a 0 at the start or end of a decimal number (which are also typically written in big-endian notation). In either case you need to allow for the change in the overall length or risk losing data and probably corrupting the rest of the data file in the process. If your data is actually groups of 16-bit or 32-bit elements rather than bytes then you need to add a whole number of elements for this to make any sense at all, regardless of byte order. You can't have an arbitrary-precision integer consisting of 5½ elements. Removing a byte from the end to preserve the length corresponds to reducing module 256^N in LE notation or dividing by 256 in BE; which is "more natural" would depend on the context. For most arbitrary-precision math operations LE is easier to work with. The only thing making BE more "natural" in some situations (in particular, reading numbers out of a byte-oriented hex listing) is the historical accident that we borrowed our number system from right-to-left languages, where they were written little-endian with the one's place on the right, without reversing the order to match the surrounding text. Which leads to the alignment issues highlighted in the article.

dataflow 5y ago

Sorry, I should clarify. I meant what you would see in a hex editor would turn out like that. i.e. if you had the byte sequence 67 45 23 01, and decided to display it in 2-byte words, LE could just display 6745 2301; you'd know the first byte is still the least significant regardless of the grouping, and you can just regard the spacing as a visual grouping aid (pretty natural). If you tried e.g. "Go To -> Offset 3", you'd still land on the 01, just as if you went right by 3 positions visually... pretty intuitive. Compare that with BE, where it'd have to show you 4567 0123, and suddenly if you Go To -> Offset 3 ('01'), you'd land to the left of offset 2 ('23'), which to me seems super confusing.

vardump 5y ago
You're getting your endians confused. Bytes 67 45 23 01 interpreted as 16-bit words in: Little endian, LE, or what x86 uses: 4567 0123 Big endian, BE, or what 68k uses: 6745 2301
- dataflow 5y ago
  I'm not saying the same thing you think I'm saying, but I do realize I'm definitely explaining what I mean poorly... likely because hex editors themselves have multiple behaviors in this regard. Some merely use spaces to separate bytes visually, others actually group the bytes and parse them as integers, swapping them as needed to read like human writing. Let me try illustrating a different situation, hopefully without that confusion. Let's say your data starts with the byte sequence 67 45 23 01... If you assume these represents some LE numbers, and want to multiply by 256 (decimal), you end up with 00 67 45 23 01... it really doesn't matter (and you don't need to know) what the word sizes were. That's the only sane result, and the byte at offset 2 would end up being 45h... end of story. Even if your number was only supposed to be N bytes and now it's N+1 bytes, you can just chop it back to N bytes and your result will still be correct (modulo 256^N) and as intuitive as it could be. But if you start working in BE, suddenly things get confusing fast. Imagine what this operation would be for 2-byte BE words. The first word in BE is 6745 and now becomes 674500, and you overflowed by 1 byte. So which part do you keep and which part do you overflow to the next word? If you keep the 6745, then the 00 ends up affecting the second word rather than the first one, which is just completely nonsensical. The other option is to keep 4500 and and shove the 67 into the next word, turning it from 2301 into 230167. Now you have to repeat the same procedure with the 23, etc. until you reach the end of the data. Now look at what just happened in the BE case. You have the bizarre situation where your words are internally BE (i.e. "go to offset 0" would now land on the 00 byte, which are not the first 2 characters in the editor!). And across words, they're still treated like LE—the inter-word overflows are still moving bytes to higher-order words on the right, not the left! There's just no sane way to do math with N-byte words and avoid LE entirely; even if you treat each word as BE, you're absolutely forced to treat the word sequence as LE. The only way to actually avoid all LE is to interpret the whole thing as 1 gigantic bignum, where "multiply by 256" ends up being translated into "append 00 to the end of the stream". That's great if your data really was 1 gigantic bignum, but not so much if your data was just typical ints or longs. If this is still confusing (I realize it might be) then I'm not sure how else to put my thoughts into words unfortunately (no pun intended). Hopefully you can kind of see what I'm getting at though, even if I'm explaining some portions of it poorly (sorry).
  - nybble41 5y ago
    > Some merely use spaces to separate bytes visually, others actually group the bytes and parse them as integers, swapping them as needed to read like human writing. In other words, your problem is that your hex editor is incorrectly assuming that the numbers are little-endian. If it interpreted them as big-endian then nothing would be swapped since the order of the bytes matches the standard conventions for numbers in most European writing systems. This is not a big-endian problem, it's a little-endian problem. A decent hex editor will allow you to set the byte order to match your file. (And as the article points out, the issue would be reversed if the bytes were displayed right-to-left following the same conventions that we use for numbers… but then all your strings would be reversed.) Both representations have strengths and weaknesses depending on what you want to do. Most arbitrary-precision math works better with LE. On the other hand, hexadecimal string formatting works better with the BE encoding, where LE would require either the input or output to be reversed.
    - dataflow 5y ago
      No, this has nothing to do with the editor or bignums. I'm just using those to illustrate the underlying inconsistency. The most natural order/sequence out there is that of natural numbers with zero (aka whole numbers), i.e. N0 = 0, 1, 2, 3, ... So I'm basically arguing that means the most digit places are in the same order, i.e. the coefficients would be ordered as 256^N i.e. 256^0, 256^1, 256^2, 256^3, ...
      - nybble41 5y ago
        I was addressing your first point about how the data is displayed in hex editors. But the rest IMHO is just fixating on one particular operation ("multiplying by 256" by prepending bytes without regard for the data format) which does not seem particularly useful or prudent. Sure, in LE you can prepend a 00 byte to multiply by 256. In BE you do the same by appending a 00 byte at the end, and a 00 at the beginning would just be extra padding, exactly like adding a 0 at the start or end of a decimal number (which are also typically written in big-endian notation). In either case you need to allow for the change in the overall length or risk losing data and probably corrupting the rest of the data file in the process. If your data is actually groups of 16-bit or 32-bit elements rather than bytes then you need to add a whole number of elements for this to make any sense at all, regardless of byte order. You can't have an arbitrary-precision integer consisting of 5½ elements. Removing a byte from the end to preserve the length corresponds to reducing module 256^N in LE notation or dividing by 256 in BE; which is "more natural" would depend on the context. For most arbitrary-precision math operations LE is easier to work with. The only thing making BE more "natural" in some situations (in particular, reading numbers out of a byte-oriented hex listing) is the historical accident that we borrowed our number system from right-to-left languages, where they were written little-endian with the one's place on the right, without reversing the order to match the surrounding text. Which leads to the alignment issues highlighted in the article.