5 ms·

Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say y

by Asooka 7y ago

Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] + someListOfNumbers[3]; If headerOffset is not a multiple of 4, bad things will happen. On x86 you'll get away with it... Unless you were summing many more ints and the compiler decides to use aligned SSE loads for speed[1]. On ARM you'll get a SIGBUS just for trying to read unaligned ints (IIRC). You can fix that by wrapping your ints or whatever you're trying to read in a packed struct, but it is yet another thing to keep in mind. [1] http://pzemtsov.github.io/2016/11/06/bug-story-alignment-on-x86.html http://pzemtsov.github.io/2016/11/06/bug-story-alignment-on-...

saagarjha 7y ago
Why not use a memcpy here and not rely on undefined behavior? I’m sure the compiler will optimize out the call anyways, especially if you enforce alignment.
ben-schaaf 7y ago
> Oh I see you didn't get to caveat 5 > On x86 you'll get away with it... Guess we'll have to wait for the proliferation of a different architecture to encounter this one :)
- 0xffff2 7y ago
  Different architectures like ARM? Plenty of code will only ever run on an x86 chip, but a non-trivial amount will run on x86 and ARM at some point. I've certainly been bitten by this before.
  - ben-schaaf 7y ago
    Sure, but my point is that both Sublime products only support x86 currently. There's likely a fair amount of other stuff that would break if we ported to ARM.
    - 0xffff2 7y ago
      Didn't realize who I was replying to. Oops. If you're confident that the code is going to stay x86 only then sure. Personally I've found my code has an annoying habit of growing legs and moving to other projects, often without my knowledge until much later. Of course when this happens the person doing it usually has little regard for any assumptions I made when writing the original code.
    - deleted 7y ago
      [deleted]
planteen 7y ago
You should not directly use structs for data serialization. If you end up having to support a big endian platform, your structure orders will change, not to mention potential packing issues as you said. https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html?m=1 https://commandcenter.blogspot.com/2012/04/byte-order-fallac...
- hyc_symas 7y ago
  Wasting a good performance optimization on the rare chance that you might one day have to support a different endian architecture is IMO a poor tradeoff. The number of big-endian machines in use today is continually shrinking, and the number that are active on a heterogeneous network is even smaller. In LMDB we simply document "don't use this with remote filesystems" and avoid the issue - if you're never sharing files with other machines, there's never a question of mixed endian accessors.
  - planteen 7y ago
    I still think you shouldn't be directly sending structs over the wire or to disk. The alternatives are so much better - SQLite or Cap’n Proto. I'm a bit shell shocked from supporting both big and little endian in structs from previous jobs. I've had nightmare situations with it twice. I do embedded systems and while little endian is winning there too, you still have legacy things like the LEON (SPARC) that is big endian. I've heard lots of network embedded hardware is big endian too for obvious reasons.
    - cerebellum42 7y ago
      To disk is probably a bad idea but over the wire has some legitimate applications. Being reliant on same endian systems can be alright if for example you're building a distributed computing system where the same binary will be executed on a bunch of systems and all you're doing is sharing computation results between those instances. You do get unrivaled serialization speed that way. Timely Dataflow [1] works that way by default, but it also has an option for "real" serialization if that is required. Admittedly that's a fairly specific application but it's real and sometimes it's a good tradeoff. [1] https://github.com/TimelyDataflow/timely-dataflow https://github.com/TimelyDataflow/timely-dataflow
loeg 7y ago
It's invalid for the compiler to vectorize scalar operations with unknown alignment if those vector operations require alignment. (That said, on x86 many AVX operations work fine unaligned, and are just as fast as the aligned versions on recent microarchitectures.)
legulere 7y ago
If you dereference pointers with arbitrary offsets read from a file, alignment issues are the least of your issues.

by Asooka 7y ago

saagarjha 7y ago

Why not use a memcpy here and not rely on undefined behavior? I’m sure the compiler will optimize out the call anyways, especially if you enforce alignment.

ben-schaaf 7y ago

> Oh I see you didn't get to caveat 5 > On x86 you'll get away with it... Guess we'll have to wait for the proliferation of a different architecture to encounter this one :)

0xffff2 7y ago
Different architectures like ARM? Plenty of code will only ever run on an x86 chip, but a non-trivial amount will run on x86 and ARM at some point. I've certainly been bitten by this before.
- ben-schaaf 7y ago
  Sure, but my point is that both Sublime products only support x86 currently. There's likely a fair amount of other stuff that would break if we ported to ARM.
  - 0xffff2 7y ago
    Didn't realize who I was replying to. Oops. If you're confident that the code is going to stay x86 only then sure. Personally I've found my code has an annoying habit of growing legs and moving to other projects, often without my knowledge until much later. Of course when this happens the person doing it usually has little regard for any assumptions I made when writing the original code.
  - deleted 7y ago
    [deleted]

planteen 7y ago

You should not directly use structs for data serialization. If you end up having to support a big endian platform, your structure orders will change, not to mention potential packing issues as you said. https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html?m=1 https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

hyc_symas 7y ago
Wasting a good performance optimization on the rare chance that you might one day have to support a different endian architecture is IMO a poor tradeoff. The number of big-endian machines in use today is continually shrinking, and the number that are active on a heterogeneous network is even smaller. In LMDB we simply document "don't use this with remote filesystems" and avoid the issue - if you're never sharing files with other machines, there's never a question of mixed endian accessors.
- planteen 7y ago
  I still think you shouldn't be directly sending structs over the wire or to disk. The alternatives are so much better - SQLite or Cap’n Proto. I'm a bit shell shocked from supporting both big and little endian in structs from previous jobs. I've had nightmare situations with it twice. I do embedded systems and while little endian is winning there too, you still have legacy things like the LEON (SPARC) that is big endian. I've heard lots of network embedded hardware is big endian too for obvious reasons.
  - cerebellum42 7y ago
    To disk is probably a bad idea but over the wire has some legitimate applications. Being reliant on same endian systems can be alright if for example you're building a distributed computing system where the same binary will be executed on a bunch of systems and all you're doing is sharing computation results between those instances. You do get unrivaled serialization speed that way. Timely Dataflow [1] works that way by default, but it also has an option for "real" serialization if that is required. Admittedly that's a fairly specific application but it's real and sometimes it's a good tradeoff. [1] https://github.com/TimelyDataflow/timely-dataflow https://github.com/TimelyDataflow/timely-dataflow

loeg 7y ago

It's invalid for the compiler to vectorize scalar operations with unknown alignment if those vector operations require alignment. (That said, on x86 many AVX operations work fine unaligned, and are just as fast as the aligned versions on recent microarchitectures.)

legulere 7y ago

If you dereference pointers with arbitrary offsets read from a file, alignment issues are the least of your issues.