5 ms·

I'd say that in general any format of variable size that does not state its length before the variable part is provoking buffer-overflow bugs. Thus such formats

by nine_k 2y ago

I'd say that in general any format of variable size that does not state its length before the variable part is provoking buffer-overflow bugs. Thus such formats should be avoided in any kind of binary interop, This holds even if the total length of the data in an interaction is not known ahead of time. E.g. an audio stream can be of indeterminate length, not known when the first byte is sent over the network, but each UDP packet has a well-determined length given in the header. The length field can be made variable-size in a rather fool-proof way [1], allowing to economically represent both tiny and huge sizes. (Zip files, WAD files, etc have that info at the very end, but this is because a file has a well-defined end before you start appending to it; fseek(fp, 0, SEEK_END) can't miss.) [1]: http://personal.kent.edu/~sbirch/Music_Production/MP-II/MIDI/midi_file_format.htm#vlq http://personal.kent.edu/~sbirch/Music_Production/MP-II/MIDI...

queuebert 2y ago
Niklaus Wirth has entered the chat.
- Tuna-Fish 2y ago
  There are a whole bunch of other things I dislike about Pascal, but in this one he was just undeniably correct. Worrying about 1-3 extra bytes (on the 16 and 32 bit platforms where it potentially mattered) was just not worth all the issues that null-terminated strings brought with them. My current favorite string implementations are the various compact string crates for Rust. Generally, you want a string to be able to do at least three things: - Pointer, length, capacity tuple, for heap-allocated strings, 24B on x64. - String inlined into the 24B buffer. - Pointer, length tuple where pointer points to rodata. You can do any of that and the discriminator in 24B, given the healthy assumption that all strings are shorter than 2^63. Sadly, switching costs are massive and every programming language is pretty much struck with the string they started with. Hopefully, whatever comes next can crib from smartstring or compact_str or the like.
  - gumby 2y ago
    There was a paper on this in the late 70s from the Cedar group at PARC. This was back when computer science papers were actual scientific papers, so full of analysis of different alogorithms' performance with counted vs delimited strings. Counted strings won hands down on anything but strings so short the length was a large percentage of overall size. Yet...since nobody reads the literature, we have all continued to suffer.
    - ninkendo 2y ago
      The late 70s was way too late for this… null terminated strings were already adopted by C and UNIX by then, and the rest is history.
      - eru 2y ago
        And sadly, we know that the C folks don't read papers; otherwise they wouldn't have come up with Go later.
  - IgorPartola 2y ago
    Yes except when no. Imagine you are writing performance sensitive code. You want to get a substring from a string, one that is not going to live outside your hot loop. In standard C you can just reference a part of a string with a pointer offset. All standard functions will continue working and you didn’t have to make any calls outside of your loop, not to the allocator, not to memcopy, nothing. With strings being objects that are prefixed by a header cannot do this. At a minimum you need to allocate a new header, if not the whole string. Yes that’s the safer route but also a lot less performant. Most crucially, you can build the header string implementation on top of C strings. You cannot do the opposite. Realistically though C strings (aka null terminated strings) are just not a great thing because of the null termination. For my money, I would prefer to just use untermianted arrays and a separate size variable, as well as wide character strings for actual display stuff. This way all the interop must include string lengths (or some other way to determine length), and all internal stuff may be just ASCII but must not leave your internal logic and never be shown to the user.
- ThreatSystems 2y ago
  Up to
- msla 2y ago
  Yes, his arrays which have length as an immutable part of their type certainly prevent certain kinds of bugs. Too bad about making it impossible to write generic array-handling subroutines, even if you accept the generally inexpressive type system as a given.
EasyMark 2y ago
one of the main reasons I use c++ as a "better c" and use the string library in embedded (or an embedded optimized version) all time, along with basic containers and algos, and "simple" classes and inheritance. Yeah I know there are c string libraries out there that also solve the problem, and even wrote a couple of my own over the years for "minimal size to functionality" ratio.
- kevin_thibedeau 2y ago
  The problem with C++ strings for embedded is the heavy dependence on the heap for many operations and the constant copying of ROM literals into RAM objects. If your target platform is "embedded" with 1GiB+ RAM then you can work in systems programming mode and not care. If it's 32KiB, a heap may be too much of a liability.
  - Thorrez 2y ago
    I think you can probably avoid unnecessary copies by using std::string_view where possible instead of std::string .
interroboink 2y ago
I'm sure you know this, but just another point for people to keep in mind: using a length+contents representation makes it harder to modify the payload, if needed (more bookkeeping). And using a variable-size length makes that even harder, since you might have to shift or re-copy the full payload to make room for the new "length" header. Of course, once you're done processing and are sending it along (as in serialization, that you mention), it's not an issue.

by nine_k 2y ago

queuebert 2y ago

Niklaus Wirth has entered the chat.

Tuna-Fish 2y ago
There are a whole bunch of other things I dislike about Pascal, but in this one he was just undeniably correct. Worrying about 1-3 extra bytes (on the 16 and 32 bit platforms where it potentially mattered) was just not worth all the issues that null-terminated strings brought with them. My current favorite string implementations are the various compact string crates for Rust. Generally, you want a string to be able to do at least three things: - Pointer, length, capacity tuple, for heap-allocated strings, 24B on x64. - String inlined into the 24B buffer. - Pointer, length tuple where pointer points to rodata. You can do any of that and the discriminator in 24B, given the healthy assumption that all strings are shorter than 2^63. Sadly, switching costs are massive and every programming language is pretty much struck with the string they started with. Hopefully, whatever comes next can crib from smartstring or compact_str or the like.
- gumby 2y ago
  There was a paper on this in the late 70s from the Cedar group at PARC. This was back when computer science papers were actual scientific papers, so full of analysis of different alogorithms' performance with counted vs delimited strings. Counted strings won hands down on anything but strings so short the length was a large percentage of overall size. Yet...since nobody reads the literature, we have all continued to suffer.
  - ninkendo 2y ago
    The late 70s was way too late for this… null terminated strings were already adopted by C and UNIX by then, and the rest is history.
    - eru 2y ago
      And sadly, we know that the C folks don't read papers; otherwise they wouldn't have come up with Go later.
- IgorPartola 2y ago
  Yes except when no. Imagine you are writing performance sensitive code. You want to get a substring from a string, one that is not going to live outside your hot loop. In standard C you can just reference a part of a string with a pointer offset. All standard functions will continue working and you didn’t have to make any calls outside of your loop, not to the allocator, not to memcopy, nothing. With strings being objects that are prefixed by a header cannot do this. At a minimum you need to allocate a new header, if not the whole string. Yes that’s the safer route but also a lot less performant. Most crucially, you can build the header string implementation on top of C strings. You cannot do the opposite. Realistically though C strings (aka null terminated strings) are just not a great thing because of the null termination. For my money, I would prefer to just use untermianted arrays and a separate size variable, as well as wide character strings for actual display stuff. This way all the interop must include string lengths (or some other way to determine length), and all internal stuff may be just ASCII but must not leave your internal logic and never be shown to the user.
ThreatSystems 2y ago
Up to
msla 2y ago
Yes, his arrays which have length as an immutable part of their type certainly prevent certain kinds of bugs. Too bad about making it impossible to write generic array-handling subroutines, even if you accept the generally inexpressive type system as a given.

EasyMark 2y ago

one of the main reasons I use c++ as a "better c" and use the string library in embedded (or an embedded optimized version) all time, along with basic containers and algos, and "simple" classes and inheritance. Yeah I know there are c string libraries out there that also solve the problem, and even wrote a couple of my own over the years for "minimal size to functionality" ratio.

kevin_thibedeau 2y ago
The problem with C++ strings for embedded is the heavy dependence on the heap for many operations and the constant copying of ROM literals into RAM objects. If your target platform is "embedded" with 1GiB+ RAM then you can work in systems programming mode and not care. If it's 32KiB, a heap may be too much of a liability.
- Thorrez 2y ago
  I think you can probably avoid unnecessary copies by using std::string_view where possible instead of std::string .

interroboink 2y ago

I'm sure you know this, but just another point for people to keep in mind: using a length+contents representation makes it harder to modify the payload, if needed (more bookkeeping). And using a variable-size length makes that even harder, since you might have to shift or re-copy the full payload to make room for the new "length" header. Of course, once you're done processing and are sending it along (as in serialization, that you mention), it's not an issue.