3 ms·

First common 32 bit system was Win 95, which required 4MB of RAM (not GB!). The 4-byte prefix would be considered extremely wasteful in those times - maybe not

by theamk 7mo ago

First common 32 bit system was Win 95, which required 4MB of RAM (not GB!). The 4-byte prefix would be considered extremely wasteful in those times - maybe not for a single string, but anytime when there is a list of strings involved, such as constants list. (As a point of reference, Turbo Pascal's default strings still had 1-byte length field). Plus, C-style strings allow a lot of optimizations - if you have a mutable buffer with data, you can make a string out of them with zero copy and zero allocations. strtok(3) is an example of such approach, but I've implemented plenty of similar parsers back in the day. INI, CSV, JSON, XML - query file size, allocate buffer once, read it into the buffer, drop some NULL's into strategic positions, maybe shuffle some bytes around for that rare escape case, and you have a whole bunch of C strings, ready to use, and with no length limits. Compared to this, Pascal strings would be incredibly painful to use... So you query file size, allocate, read it, and then what? 1-byte length is too short, and for 2+ byte length, you need a secondary buffer to copy string to. And how big should this buffer be? Are you going to be dynamically resizing it or wasting some space? And sure, _today_ I no longer write code like that, I don't mind dropping std::string into my code, it'd just a meg or so of libraries and 3x overhead for short strings - but that's nothing those days. But back when those conventions were established, it was really really important.

priceishere 7mo ago
How do you drop nulls in the middle of a string without requiring O(N) extra space to restore the original characters?
amluto 7mo ago
> query file size, allocate buffer once, read it into the buffer, drop some NULL's into strategic positions, maybe shuffle some bytes around for that rare escape case, and you have a whole bunch of C strings, ready to use, and with no length limits. I have also done this, but I would argue that, even at the time, the design was very poor. A much much better solution would have been wise pointers — pass around the length of the string separately from the pointer, much like string_view or Rust’s &str. Then you could skip the NULL-writing part. Maybe C strings made sense on even older machines which had severely limited registers —- if you have an accumulator and one resister usable as a pointer, you want to minimize the number of variables involved in a computation.
delta_p_delta_x 7mo ago
> zero copy and zero allocations This is a red herring, because when you actually read the strings out, you still need to iterate through the length for each string—zero copy, zero allocation, but linear complexity. > query file size, allocate buffer once, read it into the buffer, drop some NULL's into strategic positions, maybe shuffle some bytes around for that rare escape case, and you have a whole bunch of C strings, ready to use, and with no length limits. I write parsers in a very different way—I keep the file buffer around as read-only until the end of the pipeline, prepare string views into the buffer, and pipe those along to the next step.
- dh2022 7mo ago
  I think the concern was conserving memory ( which was scarce back then) and not iterating through each substring.
  - delta_p_delta_x 7mo ago
    I am very sceptical about that. Much safer and cleaner languages like ML and Lisp were contemporary to C, and were equally developed on memory-scarce hardware.
    - kelnos 7mo ago
      They were also comparatively slow, no? And their runtimes used up much more of that scarce memory than a C program did.
    - theamk 7mo ago
      Maybe on the high-end machines in some fancy lab somewhere? All I saw were 386's and 486's, and I am pretty sure every piece of software I ever used was either C or Turbo Pascal or direct assembly. In the mid-90s, Java appeared and I remember how horribly slow those Java apps were compared to C/Pascal code.
  - priceishere 7mo ago
    But does it even conserve memory? Copying a string when you have the length is 2 bytes of machine code on x86 (rep movsb). Remember, code takes up memory too.
kstrauser 7mo ago
> First common 32 bit system was Win 95 We're just going to ignore Amigas, and any Unix workstations?
dev-ns8 7mo ago
Besides my DA/Algo classes in College, I've never used C seriously. And you know, it's semantics like this that really make me go WTF lol.... From strtok man page... "The first time that strtok() is called, str should be specified; subsequent calls, wishing to obtain further tokens from the same string, should pass a null pointer instead." Really?? a null pointer.. This is valid code: char str[] = "C is fucking weird, ok? I said it, sue me."; char *result = strtok(str, ","); char *res = strtok(NULL, ","); Why is that ok?
- kelnos 7mo ago
  You have to understand the context, and the time period. Memory and CPU cycles were precious. All computers being 24/7 networked wasn't a thing, so security wasn't much of a concern. API design tended to reflect that.
  - dev-ns8 7mo ago
    Not mentioned in my initial comment, but yeah, I'm viscerally aware of the affect the time period and resources at the time have on API design in C and other languages from that time period. The null pointer in place of the operand here just seemed like a really good quirk to point out
- tialaramex 7mo ago
  It's like this because the 1970s C programmer, typically a single individual, is expected to maintain absolute knowledge of the full context of everything at all times. So these functions (the old non-re-entrant C functions) just assume you - that solo programmer - will definitely know which string you're currently tokenising and would never have say, a sub-routine which also needs to tokenize strings. All of this is designed before C11, which means that hilariously it's actually always Undefined Behaviour to write multi-threaded code in C. There's no memory ordering rules yet in the language, and if you write a data race (how could you not in multi-threaded code) then the Sequentially Consistent if Data Race Free proof, SC/DRF does not apply and in C all bets are off if you lose Sequential Consistency† So in this world that's enough, absolute mastery and a single individual keeping track of everything. Does it work? Not very well but hey, it was cheap. † This is common and you should assume you're fucked in any concurrent language which doesn't say otherwise. In safe Rust you can't write a data race so you're always SC, in Java losing SC is actually guaranteed safe (you probably no longer understand your program, but it does have a meaning and you could reason about it) but in many languages which say nothing it's game over because it was game over in C and they can't do better.

by theamk 7mo ago

priceishere 7mo ago

How do you drop nulls in the middle of a string without requiring O(N) extra space to restore the original characters?

amluto 7mo ago

> query file size, allocate buffer once, read it into the buffer, drop some NULL's into strategic positions, maybe shuffle some bytes around for that rare escape case, and you have a whole bunch of C strings, ready to use, and with no length limits. I have also done this, but I would argue that, even at the time, the design was very poor. A much much better solution would have been wise pointers — pass around the length of the string separately from the pointer, much like string_view or Rust’s &str. Then you could skip the NULL-writing part. Maybe C strings made sense on even older machines which had severely limited registers —- if you have an accumulator and one resister usable as a pointer, you want to minimize the number of variables involved in a computation.

delta_p_delta_x 7mo ago

> zero copy and zero allocations This is a red herring, because when you actually read the strings out, you still need to iterate through the length for each string—zero copy, zero allocation, but linear complexity. > query file size, allocate buffer once, read it into the buffer, drop some NULL's into strategic positions, maybe shuffle some bytes around for that rare escape case, and you have a whole bunch of C strings, ready to use, and with no length limits. I write parsers in a very different way—I keep the file buffer around as read-only until the end of the pipeline, prepare string views into the buffer, and pipe those along to the next step.

dh2022 7mo ago
I think the concern was conserving memory ( which was scarce back then) and not iterating through each substring.
- delta_p_delta_x 7mo ago
  I am very sceptical about that. Much safer and cleaner languages like ML and Lisp were contemporary to C, and were equally developed on memory-scarce hardware.
  - kelnos 7mo ago
    They were also comparatively slow, no? And their runtimes used up much more of that scarce memory than a C program did.
  - theamk 7mo ago
    Maybe on the high-end machines in some fancy lab somewhere? All I saw were 386's and 486's, and I am pretty sure every piece of software I ever used was either C or Turbo Pascal or direct assembly. In the mid-90s, Java appeared and I remember how horribly slow those Java apps were compared to C/Pascal code.
- priceishere 7mo ago
  But does it even conserve memory? Copying a string when you have the length is 2 bytes of machine code on x86 (rep movsb). Remember, code takes up memory too.

kstrauser 7mo ago

> First common 32 bit system was Win 95 We're just going to ignore Amigas, and any Unix workstations?

dev-ns8 7mo ago

Besides my DA/Algo classes in College, I've never used C seriously. And you know, it's semantics like this that really make me go WTF lol.... From strtok man page... "The first time that strtok() is called, str should be specified; subsequent calls, wishing to obtain further tokens from the same string, should pass a null pointer instead." Really?? a null pointer.. This is valid code: char str[] = "C is fucking weird, ok? I said it, sue me."; char *result = strtok(str, ","); char *res = strtok(NULL, ","); Why is that ok?

kelnos 7mo ago
You have to understand the context, and the time period. Memory and CPU cycles were precious. All computers being 24/7 networked wasn't a thing, so security wasn't much of a concern. API design tended to reflect that.
- dev-ns8 7mo ago
  Not mentioned in my initial comment, but yeah, I'm viscerally aware of the affect the time period and resources at the time have on API design in C and other languages from that time period. The null pointer in place of the operand here just seemed like a really good quirk to point out
tialaramex 7mo ago
It's like this because the 1970s C programmer, typically a single individual, is expected to maintain absolute knowledge of the full context of everything at all times. So these functions (the old non-re-entrant C functions) just assume you - that solo programmer - will definitely know which string you're currently tokenising and would never have say, a sub-routine which also needs to tokenize strings. All of this is designed before C11, which means that hilariously it's actually always Undefined Behaviour to write multi-threaded code in C. There's no memory ordering rules yet in the language, and if you write a data race (how could you not in multi-threaded code) then the Sequentially Consistent if Data Race Free proof, SC/DRF does not apply and in C all bets are off if you lose Sequential Consistency† So in this world that's enough, absolute mastery and a single individual keeping track of everything. Does it work? Not very well but hey, it was cheap. † This is common and you should assume you're fucked in any concurrent language which doesn't say otherwise. In safe Rust you can't write a data race so you're always SC, in Java losing SC is actually guaranteed safe (you probably no longer understand your program, but it does have a meaning and you could reason about it) but in many languages which say nothing it's game over because it was game over in C and they can't do better.