3 ms·

Why is it such a terrible idea? No need to add complexity, dependancies and reduced performance by using these libraries.

lifthrasiir 7mo ago
No defined binary encoding, no guarantee about concurrent modifications, performance trade-offs (mmap is NOT always faster than sequential reads!) and more.
- josefx 7mo ago
  Doesn't that just describe low level file IO in general?
amluto 7mo ago
Lots of reasons: The code is not portable between architectures. You can’t actually define your data structure. You can pretend with your compiler’s version of “pack” with regrettable results. You probably have multiple kinds of undefined behavior. Dealing with compatibility between versions of your software is awkward at best. You might not even get amazing performance. mmap is not a panacea. Page faults and TLB flushing are not free. You can’t use any sort of advanced data types — you get exactly what C gives you. Forget about enforcing any sort of invariant at the language level.
- Negitivefrags 7mo ago
  I've written a lot of code using that method, and never had any portability issues. You use types with number of bits in them. Hell, I've slung C structs across the network between 3 CPU architectures. And I didn't even use htons! Maybe it's not portable to some ancient architecture, but none that I have experienced. If there is undefined behavior, it's certainly never been a problem either. And I've seen a lot of talk about TLB shootdown, so I tried to reproduce those problems but even with over 32 threads, mmap was still faster than fread into memory in the tests I ran. Look, obviously there are use cases for libraries like that, but a lot of the time you just need something simple, and writing some structs to disk can go a long way.
  - ddtaylor 7mo ago
    C allows most of this, whereas C++ doesn't allow pointer aliasing without a compiler flag, tricks and problems. I agree you can certainly just use bytes of the correct sizes, but often to get the coverage you need for the data structure you end up writing some form of wrapper or fixup code, which is still easier and gives you the control versus most of the protobuf like stuff that introduces a lot of complexity and tons of code.
    - nly 7mo ago
      __attribute__((may_alias, packed)) right on the struct.
      - johannes1234321 7mo ago
        That is not C, but a non-standard extension and thus not portable.
        josefx 7mo ago
        > non-standard extension and thus not portable Modern versions of standard C aren't very portable either, unless you plan to stick to the original version of K&R C you have to pick and choose which implementations you plan to support.
        ddtaylor 7mo ago
        I disagree. Modern C with C17 and C23 make this less of an issue. Sure, some vendors suck and some people take shortcuts with embedded systems, but the standard is there and adopted by GCC, Clang and even MSVC has shaped up a bit.
        josefx 7mo ago
        > GCC, Clang and even MSVC Well, if that is the standard for portability then may_alias might as well be standard. GCC and Clang support it and MSVC doesn't implement the affected optimization as far as I can find.
- direwolf20 7mo ago
  "Portable" has originally meant "able to be ported" and not "is already ported"
jraph 7mo ago
Because a struct might not serialize the same way from a CPU architecture to another. The sizes of ints, the byte order and the padding can be different for instance.
- bloppe 7mo ago
  C has had fixed size int types since C99. And you've always been able to define struct layouts with perfect precision (struct padding is well defined and deterministic, and you can always use __attribute__(packed) and bit fields for manual padding). Endianness might kill your portability in theory. but in practice, nobody uses big endian anymore. Unless you're shipping software for an IBM mainframe, little endian is portable.
  - fc417fc802 7mo ago
    It can be made to work (as you point out), and the core idea is great, but the implementation is terrible. You have to stop and think about struct layout rules rather than declaring your intent and having the compiler check for errors. As usual C is a giant pile of exquisitely crafted footguns. A "sane" version of the feature would provide for marking a struct as intended for ser/des at which point you'd be required to spell out every last alignment, endianness, and bit width detail. (You'd still have to remember to mark any structs used in conjunction with mmap but C wouldn't be any fun if it was safe.)
  - P-Nuts 7mo ago
    You just define the structures in terms of some e.g. uint32_le etc types for which you provide conversion functions to native endianness. On a little endian platform the conversion is a no-op.

lifthrasiir 7mo ago

No defined binary encoding, no guarantee about concurrent modifications, performance trade-offs (mmap is NOT always faster than sequential reads!) and more.

josefx 7mo ago
Doesn't that just describe low level file IO in general?

amluto 7mo ago

Lots of reasons: The code is not portable between architectures. You can’t actually define your data structure. You can pretend with your compiler’s version of “pack” with regrettable results. You probably have multiple kinds of undefined behavior. Dealing with compatibility between versions of your software is awkward at best. You might not even get amazing performance. mmap is not a panacea. Page faults and TLB flushing are not free. You can’t use any sort of advanced data types — you get exactly what C gives you. Forget about enforcing any sort of invariant at the language level.

Negitivefrags 7mo ago
I've written a lot of code using that method, and never had any portability issues. You use types with number of bits in them. Hell, I've slung C structs across the network between 3 CPU architectures. And I didn't even use htons! Maybe it's not portable to some ancient architecture, but none that I have experienced. If there is undefined behavior, it's certainly never been a problem either. And I've seen a lot of talk about TLB shootdown, so I tried to reproduce those problems but even with over 32 threads, mmap was still faster than fread into memory in the tests I ran. Look, obviously there are use cases for libraries like that, but a lot of the time you just need something simple, and writing some structs to disk can go a long way.
- ddtaylor 7mo ago
  C allows most of this, whereas C++ doesn't allow pointer aliasing without a compiler flag, tricks and problems. I agree you can certainly just use bytes of the correct sizes, but often to get the coverage you need for the data structure you end up writing some form of wrapper or fixup code, which is still easier and gives you the control versus most of the protobuf like stuff that introduces a lot of complexity and tons of code.
  - nly 7mo ago
    __attribute__((may_alias, packed)) right on the struct.
    - johannes1234321 7mo ago
      That is not C, but a non-standard extension and thus not portable.
      - josefx 7mo ago
        > non-standard extension and thus not portable Modern versions of standard C aren't very portable either, unless you plan to stick to the original version of K&R C you have to pick and choose which implementations you plan to support.
        ddtaylor 7mo ago
        I disagree. Modern C with C17 and C23 make this less of an issue. Sure, some vendors suck and some people take shortcuts with embedded systems, but the standard is there and adopted by GCC, Clang and even MSVC has shaped up a bit.
        josefx 7mo ago
        > GCC, Clang and even MSVC Well, if that is the standard for portability then may_alias might as well be standard. GCC and Clang support it and MSVC doesn't implement the affected optimization as far as I can find.
direwolf20 7mo ago
"Portable" has originally meant "able to be ported" and not "is already ported"

jraph 7mo ago

Because a struct might not serialize the same way from a CPU architecture to another. The sizes of ints, the byte order and the padding can be different for instance.

bloppe 7mo ago
C has had fixed size int types since C99. And you've always been able to define struct layouts with perfect precision (struct padding is well defined and deterministic, and you can always use __attribute__(packed) and bit fields for manual padding). Endianness might kill your portability in theory. but in practice, nobody uses big endian anymore. Unless you're shipping software for an IBM mainframe, little endian is portable.
- fc417fc802 7mo ago
  It can be made to work (as you point out), and the core idea is great, but the implementation is terrible. You have to stop and think about struct layout rules rather than declaring your intent and having the compiler check for errors. As usual C is a giant pile of exquisitely crafted footguns. A "sane" version of the feature would provide for marking a struct as intended for ser/des at which point you'd be required to spell out every last alignment, endianness, and bit width detail. (You'd still have to remember to mark any structs used in conjunction with mmap but C wouldn't be any fun if it was safe.)
- P-Nuts 7mo ago
  You just define the structures in terms of some e.g. uint32_le etc types for which you provide conversion functions to native endianness. On a little endian platform the conversion is a no-op.