8 ms·

Fixing C Strings

32 points by ushakov 2y ago

deleted 2y ago
[deleted]
simscitizen 2y ago
There are quite a few of these "better C string" idioms floating around. Another one to consider is e.g. https://github.com/antirez/sds https://github.com/antirez/sds (used by Redis), which instead stores the string contents in-line with the metadata.
cozzyd 2y ago
Why not have the null terminator so you can pass to normal printf? You could even do something crazy with packing a null byte with sz on 64-bit systems (since you will never have a string that long anyway...)
- D4ckard 2y ago
  Yes, there are really cool packing techniques. See this talk for example: https://www.youtube.com/watch?v=kPR8h4-qZdk https://www.youtube.com/watch?v=kPR8h4-qZdk I don't include the null-terminator because I use this type in my own environment where I never use null-terminated strings so there is no need for it.
- bigpingo 2y ago
  printf("%.*s", len, str); lets you pass the length as an int argument.
ropejumper 2y ago
Two people have already mentioned things like storing the length inline or including a null-terminator to be backwards-compatible. What's described there is basically the same as std::string_view or &str, and to me one of the biggest reasons to use these structures is that your particular view of the string doesn't interfere with someone else's. You can slice your string in the middle and just look at it piecewise without bothering anyone else. Choosing between these trade-offs just depends on what you're doing. I'd definitely choose this pattern if I were to write a parser for instance.
- Dwedit 2y ago
  The problem with string views is that they are borrowing the parent string, so you'd need to hold a strong reference to the parent string. This is easy to do in a garbage collected language, because you don't have to do anything. But it's a lot more complicated if you need to do this with reference counting. Do you make every single string view update the reference counter? Do you make a special lighter string view that doesn't keep a counted reference, and is subject to memory safety issues?
  - wruza 2y ago
    These are regular questions in languages with (and without) reference counting, what’s so special about string views?
    - Dwedit 2y ago
      There's basically no distinction between a string view and an array slice. It's borrowing an array, and the view is nothing but a reference to the parent, start position, and length. But views are also implemented as a plain pointer and a length, and that's where the memory safety issues from borrowing begin.
      - wruza 2y ago
        I understand the concern, but can’t you just maintain an actual reference field to parent_str in a string view? Unless I missed some no-extra-fields constraint itt, then sorry for the noise.
        Dwedit 2y ago
        Let's say you took a document as a string, and split it up into words using a lot of string views. Every string view created would affect the reference count of the parent string. Then every time you work with the string views, saving temporary instances, passing them to a function, assigning them, whatever, you're affecting the parent string's reference count. And reference counts are often atomic integer operations, so it might not be a regular memory increment, instead it would be an interlocked increment. And if there's multiple threads, the CPU cores will be competing over who gets to keep the reference counter in their L1 cache line. (There is a way around this where you can give threads an their own reference counter)
zabzonk 2y ago
I have been using null terminated strings since the mid 1970s - before using C, and have never had any problems with them.I have never seen an explanation from someone that has that makes any sense.
- atiedebee 2y ago
  In my experience, the standard library is inconsistent with its 0 terminator handling. fgets will treat the length passed as the capacity of the buffer, and terminate the last byte with a 0. scanf however treats the length as the number of characters to read, meaning that you need a capacity of n+1 to make sure the 0 terminator is stored properly as well. Its quite easy to mess up placing the 0 terminator yourself too. It's an overall unnecessary burden that could've been fixed quite easily.
  - zabzonk 2y ago
    the standard library certainly has a few problems, if you haven't read the docs, but that does not mean that i do.
    - kelnos 2y ago
      I'm sorry that not every programmer in the world has achieved your level of supreme perfection. We shouldn't design our languages or stdlib with the assumption that everyone will have read every line of documentation about them, and (even if they have) remember everything every time they sit down in front of their text editor. That's unrealistic. I don't actually believe you that you've been programming for 50 years and never misused a string or a string API in C or a language with similar string handling. But even if I did believe you, it wouldn't matter. Many people make mistakes, and those mistakes have cost people a lot of time, money, and stress. If you've not read about any of these instances, then I suggest you've been living under a rock and are incredibly out of touch. Or you're just trolling.
      - Quis_sum 2y ago
        C was designed in the 1970s with the goal of giving you minimal overhead, bar using a macro-assembler. The way C handles "strings" provides exactly that. The OP clearly stated that he did not mind the overhead (in terms of executable size, memory consumption, execution speed) in his particular use case.
kevin_thibedeau 2y ago
> Current compilers warn you if the format string doesn’t match its arguments. But this only works on functions that have the same signature as printf so it doesn’t work on my implementation. GCC has the format attribute that lets you have printf type checking on your own variadic functions: https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/Common-Function-Attributes.html#index-Wformat-3 https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/Common-Functio...
- WalterBright 2y ago
  So does D: https://dlang.org/spec/pragma.html#printf https://dlang.org/spec/pragma.html#printf It was a huge win, at least for me. I implemented it because I was really sick of mismatches. (Although I was careful to use the right formats, when refactoring I'd change a type and then the printf's would go awry. Having the compiler flag them made for quick fixing.)
- norir 2y ago
  Format strings are a technique that I think largely should be left behind anyway. String interpolation is in my experience usually shorter, easier to read and will always be checked by the compiler.
  - jonhohle 2y ago
    It’s shorter until you need any type of formatting. It’s hard to get more terse than `%8.2f`.
    - gizmo686 2y ago
      You can do that with string interpolation too. In python it would be: print(f"{foo:8.2f}") compared with C-style printf: printf("%8.2f", foo)
    - rowanG077 2y ago
      Also hard to be more cryptic. The amount of times I regoogle that syntax, or what exactly those numbers mean is basically uncountable.
jdblair 2y ago
I've done something similar, but unlike the author, I always reserved one extra byte and I always null terminated the string. This was so I could use existing string output functions.
WalterBright 2y ago
struct str { char *dat; sz len; }; It's the same solution D uses, except that it's a builtin type, and works for all arrays. I proposed this solution for C: https://www.digitalmars.com/articles/C-biggest-mistake.html https://www.digitalmars.com/articles/C-biggest-mistake.html It's hard to overstate what a huge win this is. D has had 23 years of experience with it, and the virtual elimination of array overflow bugs is just win, win, win. I will never understand why C keeps adding extensions consisting of marginal features, and ignores this foundational fix. I guess they still aren't tired of buffer overflow bugs always being the #1 security vulnerability of shipped C code (and C++, too!).
- Levitating 2y ago
  > It's the same solution D uses As well as most other languages and many C codebases right? Often with a separate length/capacity so the buffer can be larger than the string.
  - WalterBright 2y ago
    True, but it turns out that very few arrays need to be resizeable.
- Koshkin 2y ago
  > I will never understand why C keeps Well, I, for one, do like the idea of C (in contrast to D or C++) still being sort of the lowest-level high-level programming language - one that's just a notch above the assembler.
  - gpderetta 2y ago
    What's lower level on C compared to C++ or D?
    - WalterBright 2y ago
      Nothing. You can get just as down and dirty in D as you can in C. You just don't have to suffer under the preprocessor as it blasts your kingdom. And you've got a fully functional inline assembler, and modules.
up2isomorphism 2y ago
For all the complaints ,all you need to do is to include an another .h files from some string lib and that’s it. But I would say for 95% percent using a fixed length char array with strncpy will work just fine.
codr7 2y ago
I would consider putting the buffer last in the structure and making it flexible to allow skipping one allocation.
- ncruces 2y ago
  That misses the point. These are passed by value.
  - deleted 2y ago
    [deleted]
  - deleted 2y ago
    [deleted]
Levitating 2y ago
> I liked this kind of pattern at the bottom of OpenAI's site :) Where on OpenAI's site do I find a footer like that?
zwnow 2y ago
Never had a string related bug in any programming language in 4 years. I sincerely don't know what people talk about when they claim strings are buggy? What kinda tasks do these happen in?
- Koshkin 2y ago
  It's just that the "traditional" implementations of the operations on C strings (strcat etc.) are considered unsafe - which they are, strictly speaking. (But, to be fair, I haven't ever had problems using them, either.)
- paulddraper 2y ago
  I don’t know who you’re talking to or what they’re saying, do I can’t say. This article is about C strings FYI.
  - zwnow 2y ago
    The article claims they are buggy, thats what I am refering to.
superjared 2y ago
The bstring library[0] has been around a _long_ time. [0]: https://bstring.sourceforge.net/ https://bstring.sourceforge.net/
teo_zero 2y ago
Good attempt at a topic that annoys many programmers. I see a problem with the separation between str and str_buf, though: you create new strings with the latter, but most functions take the former as arguments. Do you convert them every time? Isn't your code littered with str_from_buf()? Put it in another way, it's like the mess with const that you mention in your article. If str is the type you use for a const read-only string, and str_buf for a non-const mutable string, you would like to pass a non-const even to those functions that "only" require a const. (I say "only" because being const is a weaker requirement than being mutable; the fact that it's more wordy is another thing that C's syntax makes confusing, but this is an entirely different topic!) It would be nice if the compiler could be instructed to automatically cast str_buf into str and not vice versa, just like it does for non-const to const. The only way out I can think of, would be to get rid of the two types and only use the one with the cap field, with the convention that if cap is zero, then the string is read-only. The drawback is that certain mistakes are only detected at run-time and not enforced by the compiler. For example, a function than takes a string s and replaces every substring s1 with s2 could have the following prototype in the two-type system: replace(str_buf s, str s1, str s2); And it would be immediate to recognize that you cannot pass a read-only string as the first argument. With a one-type system you loose this ability. Oh well, I guess if a perfect solution existed, it would have been adopted by the C committee, wouldn't it? /s
- kelnos 2y ago
  Do you convert them every time? No, the article addresses this: since the memory layout of the first two struct members is the same in both structs, you can use a pointer to str_buf anywhere a function calls for a pointer to str, after casting it.
  - teo_zero 2y ago
    > you can use a pointer to str_buf anywhere a function calls for a pointer to str Yes, you could, but I see no function mentioned in TFA that wants a pointer to str, only functions that want a str: print_str(), print_fmt(), com_write(). At the same time, the functions that return strings return a struct, never a pointer: str_new(), str_from_range(), str_from_buf(), fmt_buf_new(), and the pseudo-function STR(). To use the memory layout trick you should go through reference + cast + dereference: *((struct str *)&...) My question still holds: is the code littered by such conversion artifacts?
Quis_sum 2y ago
Sorry, but there is a significant misunderstanding: There is no such thing as a string in C. What you call a string is a pointer to char (typically "int8") - nothing more nothing less. The \0 termination is just a convention/convenience to avoid passing the bounds of the memory segment, resp. when to stop processing earlier. Once you go down the route proposed by many of the comments here - why not enhance it to deal with UTF8... Or rather implement a proper "array" type? What about the lack of multidimensional arrays instead of the pointer to pointer to ... approach? Idiosyncracies such as "int a[2][3];" being of type "int *" and not "int **"? C was never intended to shield you from mistakes, but rather replace a macro assembler. ANSI C addressed some of the issues in the original K&R C, but that is about it. If your use case would benefit from all of these protections, there are plenty of higher level language alternatives...
- __d 2y ago
  String literals are one place where the compiler implements the null-termination, so it is built-in in that sense. As per the OP’s example, a wrapper macro like their STR can work around this.
  - Quis_sum 2y ago
    I don't have an issue with the OP's example, which is quite nifty indeed, despite the penalties incurred.
- kelnos 2y ago
  That's incorrect. If I write this in my .c file: char *s = "Hi"; The compiler will not treat that as a simple mere pointer to char when allocating space for it in the binary. It will see that the rhs is surrounded by double quote characters, and allocate 3 bytes for it, instead of 2, and put a NUL byte after the bytes for 'H' and 'i'. Nul-tetminated strings are absolutely a part of the language. Certainly you can make and store strings in a different way if you'd like, but the language itself defines what a string and string literal is.
  - Quis_sum 2y ago
deleted 2y ago
[deleted]

32 points by ushakov 2y ago

deleted 2y ago

[deleted]

There are quite a few of these "better C string" idioms floating around. Another one to consider is e.g. https://github.com/antirez/sds https://github.com/antirez/sds (used by Redis), which instead stores the string contents in-line with the metadata.

cozzyd 2y ago

Why not have the null terminator so you can pass to normal printf? You could even do something crazy with packing a null byte with sz on 64-bit systems (since you will never have a string that long anyway...)

D4ckard 2y ago
Yes, there are really cool packing techniques. See this talk for example: https://www.youtube.com/watch?v=kPR8h4-qZdk https://www.youtube.com/watch?v=kPR8h4-qZdk I don't include the null-terminator because I use this type in my own environment where I never use null-terminated strings so there is no need for it.
bigpingo 2y ago
printf("%.*s", len, str); lets you pass the length as an int argument.

ropejumper 2y ago

Two people have already mentioned things like storing the length inline or including a null-terminator to be backwards-compatible. What's described there is basically the same as std::string_view or &str, and to me one of the biggest reasons to use these structures is that your particular view of the string doesn't interfere with someone else's. You can slice your string in the middle and just look at it piecewise without bothering anyone else. Choosing between these trade-offs just depends on what you're doing. I'd definitely choose this pattern if I were to write a parser for instance.

Dwedit 2y ago
The problem with string views is that they are borrowing the parent string, so you'd need to hold a strong reference to the parent string. This is easy to do in a garbage collected language, because you don't have to do anything. But it's a lot more complicated if you need to do this with reference counting. Do you make every single string view update the reference counter? Do you make a special lighter string view that doesn't keep a counted reference, and is subject to memory safety issues?
- wruza 2y ago
  These are regular questions in languages with (and without) reference counting, what’s so special about string views?
  - Dwedit 2y ago
    There's basically no distinction between a string view and an array slice. It's borrowing an array, and the view is nothing but a reference to the parent, start position, and length. But views are also implemented as a plain pointer and a length, and that's where the memory safety issues from borrowing begin.
    - wruza 2y ago
      I understand the concern, but can’t you just maintain an actual reference field to parent_str in a string view? Unless I missed some no-extra-fields constraint itt, then sorry for the noise.
      - Dwedit 2y ago
        Let's say you took a document as a string, and split it up into words using a lot of string views. Every string view created would affect the reference count of the parent string. Then every time you work with the string views, saving temporary instances, passing them to a function, assigning them, whatever, you're affecting the parent string's reference count. And reference counts are often atomic integer operations, so it might not be a regular memory increment, instead it would be an interlocked increment. And if there's multiple threads, the CPU cores will be competing over who gets to keep the reference counter in their L1 cache line. (There is a way around this where you can give threads an their own reference counter)

zabzonk 2y ago

I have been using null terminated strings since the mid 1970s - before using C, and have never had any problems with them.I have never seen an explanation from someone that has that makes any sense.

atiedebee 2y ago
In my experience, the standard library is inconsistent with its 0 terminator handling. fgets will treat the length passed as the capacity of the buffer, and terminate the last byte with a 0. scanf however treats the length as the number of characters to read, meaning that you need a capacity of n+1 to make sure the 0 terminator is stored properly as well. Its quite easy to mess up placing the 0 terminator yourself too. It's an overall unnecessary burden that could've been fixed quite easily.
- zabzonk 2y ago
  the standard library certainly has a few problems, if you haven't read the docs, but that does not mean that i do.
  - kelnos 2y ago
    I'm sorry that not every programmer in the world has achieved your level of supreme perfection. We shouldn't design our languages or stdlib with the assumption that everyone will have read every line of documentation about them, and (even if they have) remember everything every time they sit down in front of their text editor. That's unrealistic. I don't actually believe you that you've been programming for 50 years and never misused a string or a string API in C or a language with similar string handling. But even if I did believe you, it wouldn't matter. Many people make mistakes, and those mistakes have cost people a lot of time, money, and stress. If you've not read about any of these instances, then I suggest you've been living under a rock and are incredibly out of touch. Or you're just trolling.
    - Quis_sum 2y ago
      C was designed in the 1970s with the goal of giving you minimal overhead, bar using a macro-assembler. The way C handles "strings" provides exactly that. The OP clearly stated that he did not mind the overhead (in terms of executable size, memory consumption, execution speed) in his particular use case.

kevin_thibedeau 2y ago

> Current compilers warn you if the format string doesn’t match its arguments. But this only works on functions that have the same signature as printf so it doesn’t work on my implementation. GCC has the format attribute that lets you have printf type checking on your own variadic functions: https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/Common-Function-Attributes.html#index-Wformat-3 https://gcc.gnu.org/onlinedocs/gcc-14.2.0/gcc/Common-Functio...

WalterBright 2y ago
So does D: https://dlang.org/spec/pragma.html#printf https://dlang.org/spec/pragma.html#printf It was a huge win, at least for me. I implemented it because I was really sick of mismatches. (Although I was careful to use the right formats, when refactoring I'd change a type and then the printf's would go awry. Having the compiler flag them made for quick fixing.)
norir 2y ago
Format strings are a technique that I think largely should be left behind anyway. String interpolation is in my experience usually shorter, easier to read and will always be checked by the compiler.
- jonhohle 2y ago
  It’s shorter until you need any type of formatting. It’s hard to get more terse than `%8.2f`.
  - gizmo686 2y ago
    You can do that with string interpolation too. In python it would be: print(f"{foo:8.2f}") compared with C-style printf: printf("%8.2f", foo)
  - rowanG077 2y ago
    Also hard to be more cryptic. The amount of times I regoogle that syntax, or what exactly those numbers mean is basically uncountable.

jdblair 2y ago

I've done something similar, but unlike the author, I always reserved one extra byte and I always null terminated the string. This was so I could use existing string output functions.

WalterBright 2y ago

struct str { char *dat; sz len; }; It's the same solution D uses, except that it's a builtin type, and works for all arrays. I proposed this solution for C: https://www.digitalmars.com/articles/C-biggest-mistake.html https://www.digitalmars.com/articles/C-biggest-mistake.html It's hard to overstate what a huge win this is. D has had 23 years of experience with it, and the virtual elimination of array overflow bugs is just win, win, win. I will never understand why C keeps adding extensions consisting of marginal features, and ignores this foundational fix. I guess they still aren't tired of buffer overflow bugs always being the #1 security vulnerability of shipped C code (and C++, too!).

Levitating 2y ago
> It's the same solution D uses As well as most other languages and many C codebases right? Often with a separate length/capacity so the buffer can be larger than the string.
- WalterBright 2y ago
  True, but it turns out that very few arrays need to be resizeable.
Koshkin 2y ago
> I will never understand why C keeps Well, I, for one, do like the idea of C (in contrast to D or C++) still being sort of the lowest-level high-level programming language - one that's just a notch above the assembler.
- gpderetta 2y ago
  What's lower level on C compared to C++ or D?
  - WalterBright 2y ago
    Nothing. You can get just as down and dirty in D as you can in C. You just don't have to suffer under the preprocessor as it blasts your kingdom. And you've got a fully functional inline assembler, and modules.

up2isomorphism 2y ago

For all the complaints ,all you need to do is to include an another .h files from some string lib and that’s it. But I would say for 95% percent using a fixed length char array with strncpy will work just fine.

codr7 2y ago

I would consider putting the buffer last in the structure and making it flexible to allow skipping one allocation.

ncruces 2y ago
That misses the point. These are passed by value.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]

Levitating 2y ago

> I liked this kind of pattern at the bottom of OpenAI's site :) Where on OpenAI's site do I find a footer like that?

zwnow 2y ago

Never had a string related bug in any programming language in 4 years. I sincerely don't know what people talk about when they claim strings are buggy? What kinda tasks do these happen in?

Koshkin 2y ago
It's just that the "traditional" implementations of the operations on C strings (strcat etc.) are considered unsafe - which they are, strictly speaking. (But, to be fair, I haven't ever had problems using them, either.)
paulddraper 2y ago
I don’t know who you’re talking to or what they’re saying, do I can’t say. This article is about C strings FYI.
- zwnow 2y ago
  The article claims they are buggy, thats what I am refering to.

superjared 2y ago

The bstring library[0] has been around a _long_ time. [0]: https://bstring.sourceforge.net/ https://bstring.sourceforge.net/

teo_zero 2y ago

Good attempt at a topic that annoys many programmers. I see a problem with the separation between str and str_buf, though: you create new strings with the latter, but most functions take the former as arguments. Do you convert them every time? Isn't your code littered with str_from_buf()? Put it in another way, it's like the mess with const that you mention in your article. If str is the type you use for a const read-only string, and str_buf for a non-const mutable string, you would like to pass a non-const even to those functions that "only" require a const. (I say "only" because being const is a weaker requirement than being mutable; the fact that it's more wordy is another thing that C's syntax makes confusing, but this is an entirely different topic!) It would be nice if the compiler could be instructed to automatically cast str_buf into str and not vice versa, just like it does for non-const to const. The only way out I can think of, would be to get rid of the two types and only use the one with the cap field, with the convention that if cap is zero, then the string is read-only. The drawback is that certain mistakes are only detected at run-time and not enforced by the compiler. For example, a function than takes a string s and replaces every substring s1 with s2 could have the following prototype in the two-type system: replace(str_buf s, str s1, str s2); And it would be immediate to recognize that you cannot pass a read-only string as the first argument. With a one-type system you loose this ability. Oh well, I guess if a perfect solution existed, it would have been adopted by the C committee, wouldn't it? /s

kelnos 2y ago
Do you convert them every time? No, the article addresses this: since the memory layout of the first two struct members is the same in both structs, you can use a pointer to str_buf anywhere a function calls for a pointer to str, after casting it.
- teo_zero 2y ago
  > you can use a pointer to str_buf anywhere a function calls for a pointer to str Yes, you could, but I see no function mentioned in TFA that wants a pointer to str, only functions that want a str: print_str(), print_fmt(), com_write(). At the same time, the functions that return strings return a struct, never a pointer: str_new(), str_from_range(), str_from_buf(), fmt_buf_new(), and the pseudo-function STR(). To use the memory layout trick you should go through reference + cast + dereference: *((struct str *)&...) My question still holds: is the code littered by such conversion artifacts?

Quis_sum 2y ago

Sorry, but there is a significant misunderstanding: There is no such thing as a string in C. What you call a string is a pointer to char (typically "int8") - nothing more nothing less. The \0 termination is just a convention/convenience to avoid passing the bounds of the memory segment, resp. when to stop processing earlier. Once you go down the route proposed by many of the comments here - why not enhance it to deal with UTF8... Or rather implement a proper "array" type? What about the lack of multidimensional arrays instead of the pointer to pointer to ... approach? Idiosyncracies such as "int a[2][3];" being of type "int *" and not "int **"? C was never intended to shield you from mistakes, but rather replace a macro assembler. ANSI C addressed some of the issues in the original K&R C, but that is about it. If your use case would benefit from all of these protections, there are plenty of higher level language alternatives...

__d 2y ago
String literals are one place where the compiler implements the null-termination, so it is built-in in that sense. As per the OP’s example, a wrapper macro like their STR can work around this.
- Quis_sum 2y ago
  I don't have an issue with the OP's example, which is quite nifty indeed, despite the penalties incurred.
kelnos 2y ago
That's incorrect. If I write this in my .c file: char *s = "Hi"; The compiler will not treat that as a simple mere pointer to char when allocating space for it in the binary. It will see that the rhs is surrounded by double quote characters, and allocate 3 bytes for it, instead of 2, and put a NUL byte after the bytes for 'H' and 'i'. Nul-tetminated strings are absolutely a part of the language. Certainly you can make and store strings in a different way if you'd like, but the language itself defines what a string and string literal is.
- Quis_sum 2y ago

deleted 2y ago

[deleted]