8 ms·

Show HN: A nice C string API

107 points by mickjc750 4y ago

A convenient C string API, friendly alongside classic C strings.

KingLancelot 4y ago
gjvc 4y ago
worth comparing to https://cr.yp.to/lib/stralloc.html https://cr.yp.to/lib/stralloc.html
- plan999 4y ago
  You should look at an even better string library. Much more functions and safety for split/joint/tokenizer/etc it's a fork of the plan9 string library bstring. https://bstring.sourceforge.net/ https://bstring.sourceforge.net/
deleted 4y ago
[deleted]
the-printer 4y ago
Are the any good resources that explain the concept of strings in C, particularly why they’re considered to be so difficult to manage? I’m interested in the language, and that along with its safety concerns seem to be the two most frequent complaints against it that I read about online.
- e-dant 4y ago
  C strings are pointers to memory. There are semantics and assumptions encouraging null-character delimited strings, but not every API follows those rules (just got done working with a Windows API that doesn’t). Often, you have to both null-delimit your string and store its length somewhere. That’s the dangerous part. Messing either of those up, or passing your string to an API that messes that up, is not safe. C strings are pointers to memory, either the stack or the heap, and follow exactly the same rules as everything else in that chaotic space: Not many.
  - the-printer 4y ago
    Thank you for this. C programming sounds almost like some sort of combat sport. Riveting.
    - lelanthran 4y ago
      > Thank you for this. C programming sounds almost like some sort of combat sport. Riveting. I've done it for decades; it isn't really as bad as hype-attracting headlines would have you believe. Munitions control, aircraft management systems, industrial automation systems, and many more life-critical systems were programmed in C for decades with comparatively little danger from the language intrinsics leading to death. It's easy to look at the stats and say "there's a few dozen CVEs annually due to C footguns", but that's a few dozen out of hundreds of millions of deployed systems that are written in C. In practice, very few lines of C code bypass the type system, so you get much fewer bugs than an equivalent system in the more usual dynamic programming languages (Python, Javascript, etc).
      - thesnide 4y ago
        Wondering if the big influx of C derived CVE are old or new code. If it is new code, I'm also wondering about the brain damage that those safe languages causes. Yes, it is better to have memory safe languages. But it encourages sloppiness as "nothing can happen". Then those folks aren't fit to write anything else. Which closes the feedback loop on inefficient but safe languages. Which becomes the same thing in airplanes. Pilots don't really know how to fly without instruments anymore.
        wadd1e 4y ago
        >Which becomes the same thing in airplanes. Pilots don't really know how to fly without instruments anymore. Well that's just a blatantly wrong generalisation you made there, curious as to where you got that from. Consider looking up how pilot training is done before making such assumptions. Even though modern airplanes make heavy use of technology, there are emergency scenarios where lots of instruments may not work, and pilots receive more than enough training to fly an airplane in that scenario just to give one example among tons of others. edit: grammar
- jstimpfle 4y ago
  "Strings" are quite an abstract concept. They are a linear sequence of characters. But there are a number of ways to represent them - the simplest of which is a contiguous memory allocation, but depending on the use case you'd need more complex schemes. There are also different ways to do the necessary memory management (e.g. allocate statically at compile time vs dynamically at run time). One of the most complex representations is probably the string rope datastructure - a balanced tree of string chunks, supporting efficient insertion and removal anywhere in the string. Specific to C, as well as lots of low-level APIs, is only that strings are often expected to be contiguously laid out in memory and terminated with a NUL (0) byte. So you need to make sure that you always terminate with a NUL after writing to string storage. Other than that, strings aren't any harder than other aspects of programming with manually managed memory. Maybe motivated from higher-level dynamic or managed languages, is the popular idea that strings should always be allocated dynamically (like std::string for example), and support operations like string-append with automatic reallocation if the currently allocated memory isn't enough to store the new string. In practice, that's not true at all - unless you are in a domain where lots of small intermediate strings are generated. This is pretty inefficient anyway and there is likely no point to use C in this case. By far most strings in most domains are either completely static (use string literals), or are created once in a sequence of append operations and then never changed again. I get by, doing many different things from GUI apps to networking to parsers and interpreters, without any sophisticated string type. All I do is define some printf-like APIs to do logging, for example. Those typically just use a fixed size buffer for the formatting, and then flush that buffer to e.g. stderr. or flush it to a dynamically allocated memory buffer, but there almost never is a need to reallocate that string later.
- Quentak 4y ago
  I have written a short article explaining why null terminated strings as they exist in C cannot represent proper ASCII and UTF-8 because of the null terminator. It's not a full explanation of how strings work but it might be helpful for you. https://kttnr.net/blog/null-terminated-strings-are-incorrect/ https://kttnr.net/blog/null-terminated-strings-are-incorrect...
  - lifthrasiir 4y ago
    SQLite does store a null character in strings, it has lots of documented [1] issues in the API level though. [1] https://www.sqlite.org/nulinstr.html https://www.sqlite.org/nulinstr.html
    - Quentak 4y ago
      Thanks for this link. How do you get the null byte into the string? Is it through casting blob to string? The way I have encountered this is when using the C API in which string arguments for prepared statements are passed as char pointers. If those contain the null byte then the string is cut off. Allowing null characters and then mishandling them is worse than not allowing them.
- Someone 4y ago
  The problem is that C doesn’t have strings; it has functions that treat sequences of non-zero bytes followed by a zero bytes as if they are strings. So, you can’t ask it to create a string that contains the result of appending a string to another one. If you want to append two ‘strings’, you have to create a buffer large enough to hold the result, and then copy in the two sequences of bytes. And even for doing that, the library functions aren’t optimal. The basic “append this string’s data to that string, assuming there’s enough space to do so” function is strcat. It walks the first string to find the zero byte, but to “create a buffer large enough to hold the result” you already must do that. See for example https://stackoverflow.com/questions/21880730/c-what-is-the-best-and-fastest-way-to-concatenate-strings https://stackoverflow.com/questions/21880730/c-what-is-the-b...
  - jstimpfle 4y ago
    You can use snprintf to easily achieve any concatentation you'd like. len = snprintf(buffer, buffersize, "%s%s%d", string_1, string_2, int_1); if (len + 1 /*NUL*/ > buffersize) { // not enough space } You can also use this to dynamically allocate any formatted string len = snprintf(NULL, 0, ....); buffersize = len + 1; /* NUL */ buffer = allocate(buffersize); snprintf(buffer, buffersize, ... /*same args as before*/);
    - tom_ 4y ago
      I like the printf family too. Any time you're doing a bunch of strcat or whatever it's almost always massively easier to use a format string to get the same result. Very easy to get the desired width/precision/alignment, and if you need numbers, printf has your back. It even does the bounds checking for you! (And how often do you get that in C.) It won't be as fast, but it's almost always not a problem, and the nice thing about C and C++ is that the char-by-char route is still available when it is. I like to use asprintf, when available: https://man7.org/linux/man-pages/man3/asprintf.3.html https://man7.org/linux/man-pages/man3/asprintf.3.html - and when not available, I add it, along the lines of the snippet you present. Here's something I've found a useful upgrade to asprintf, as it frees the passed-in buffer after expanding the format string. You can just pass the same char ** repeatedly and it'll update the char * appropriately each time. int xasprintf(char**p,const char *fmt,..) { int n=0; char *p2=nullptr; if(fmt) { va_list v; va_start(v); n=asprintf(&p2,fmt,v); va_end(v); } if(n>=0) { free(*p); *p=p2; } return n; }
- WalterBright 4y ago
  > why they’re considered to be so difficult to manage? Back in the 90s, I was very experienced with C strings and managing them. Then I chanced to look at BASIC again, and realized that strings in BASIC were so simple and intuitive. Why couldn't C be like that? When I started on the design of D, I decided that it had to make strings as easy to do as BASIC did. And D does. The trouble with C strings is the 0 termination of them. This means: 1. to get the length of the string, you have to scan it. This is expensive. 2. when manipulating strings, a common error is to get off by one in the storage because of the 0 termination 3. you cannot get a subset of the string without making a copy. Not only is a copy expensive, but then you have to keep track of the memory for it 4. there's no way to check for buffer overflows D's design, which uses a phat pointer (length, ptr) for strings, solves these problems.
  - tored 4y ago
    A year ago I picked up the BASIC dialect PureBasic. Pleasant surprise actually, the syntax of the PureBasic dialect is a bit archaic, but if you accept that it is much easier and faster to get anything done compared to C (and C++). Personally I find low level topics easier to grok in PureBasic than in C even though they mirror the same concepts. PureBasic has Unicode strings built in. It is a bit shame that BASIC has such a bad reputation, there are many BASIC dialects that does the job well still today. https://www.purebasic.com/documentation/reference/ug_string.html https://www.purebasic.com/documentation/reference/ug_string....
cassepipe 4y ago
I always use antirez's (Redis creator) `sds` and advertise it whenvever I get the chance. Thanks to the someone who recommended it on HN some years ago. It's a joy to use. https://github.com/antirez/sds https://github.com/antirez/sds The trick is the size is hidden before the adress of the buffer.("Learn this one simple trick that will change your life for ever"). From the Readme: ``` Advantage #1: you can pass SDS strings to functions designed for C functions without accessing a struct member or calling a function Advantage #2: accessing individual chars is straightforward. Advantage #3: single allocation has better cache locality. Usually when you access a string created by a string library using a structure, you have two different allocations for the structure representing the string, and the actual buffer holding the string. Over the time the buffer is reallocated, and it is likely that it ends in a totally different part of memory compared to the structure itself. Since modern programs performances are often dominated by cache misses, SDS may perform better in many workloads. ```
- kazinator 4y ago
  Thus, sds it cannot be used for the use cases that this library allows. This library takes string slices without having to allocate or copy memory; it seems to be for use cases involving breaking down strings in complex ways, where good ergonomics and efficiency of obtaining a null-terminated C string are secondary.
- quelsolaar 4y ago
  Optimizing text is hard because you seldom up front know how much memory will be needed and allocations are slow. Another way to do it is to use: typedef struct{ size_t allocated; size_t used; char buffer[]; }String; This lets the header and the string be the same allocation. Thats a huge saving. Its also useful to store the allocation and use size separatly so that you can reuse / modify buffers. The used field lets you use memcpy without looking for string termination. You can make it even more complex by adding flags if the string is on the stack or on the heap. That way you can do things like: String buffer = MACRO_TO_CREATE_STRING_BUFFER_ON_STACK(256), *b; b = &buffer; b = do_processing_with_buffer(&b); // allocates on heap a larger buffer if needed string_free(b); // frees buffer if its on heap.
program 4y ago
It's better not to use types that end with a '_t' because the suffix is reserved in POSIX systems. https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap02.html#tag_22_02_12 https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh...
- cassepipe 4y ago
  On the other hand it is quite handy as a prefix, s_ for structs, e_ for enums, g_ for globals, t_ for simple typedefs, f_ for function pointers typedefs, u_ for unions... sky is the limit ! And it's quite easy to create an highlighting rule for it in vim if you still did not convert to treesitter. Just put in ~/.vim/after/c.vim : ``` syn match cType /\<\(t\|s\|e\|u\)_\w\+\>/ ``` Boom, custom type highlighting for C ! Pick the the letters you will use.
  - lifthrasiir 4y ago
    No wonder why a significant portion of C programmers actually want to keep tags (`struct` in `struct foo`) instead of removing them wih typedef.
- kazinator 4y ago
  I don't agree with that at all; it is "the sky might fall" reasoning. Just * have sane naming in your program. * respect namespaces like _[A-Z] and __ * solve clashes that actually happen Historically, revisions of POSIX have introduced identifiers that were not in any previously announced namespace. There is no way you can name an identifier that is guaranteed not to clash with POSIX, or any other vendor. For instance the name "openat" was fine to use in a POSIX program once upon a time. Consider that all strings have the empty string as a suffix. The string "abc" has four suffixes: "abc", "bc", "c" and "". So, every current and future POSIX identifier has "" as a suffix. This is not just a threat; it is guaranteed! Since every identifier in your program also has a "" suffix, it clashes with that namespace. What's wrong with the argument is that identifiers don't just have a suffix; they have to be identical in order to actually clash. (Or have identical prefixes, due to truncation of external names in a linker: decades ago, the limits were ridiculously small.) I doubt that even one person in POSIX standardization would be dumb enough to approve str_t being added as a typedef name in some existing or new header, and multiple approvals are required. Nobody should be losing any sleep over using _t typedef names in their C code.
e-dant 4y ago
I guess it’s nice for a C string API, but what’s the motivation to use and create this? Wouldn’t externing some C++ symbols (or Rust) work more smoothly?
- lelanthran 4y ago
  > Wouldn’t externing some C++ symbols (or Rust) work more smoothly? For the C++ case, it's not that easy due to C code that cannot handle exceptions thrown in C++ code. For the Rust bit, I'm not sure - creating the library in Rust and letting it be called from C makes the whole rust library unsafe because the data returned from the Rust API would lose ownership information, and is no more safe than simply writing it in C.
kazinator 4y ago
> Attempting to split a string using non-existent delimiter with str_pop_first_split() [returns an invalid string with .data == NULL]. But that seems like a valid case: e.g. these are comma-delimited lists of numbers: "" // empty "1" // one number "20,30" // two numbers the above remark in the documentation seems to be saying (perhaps falsely) that if we try to extract a token from the "1" string using "," as a delimiter, we get an invalid str_t rather than "1". I don't see coverage for this in the tests. There is a test which uses "123/456/789", which extracts the first two splits, and then just verifies that "789" remains. What the programmer wants is to be able to write a loop which will extract "123", "456" and "789" and then* hit the terminating case where the invalid str_t is returned. How many items are in "1,2,3," viewed as comma-separated: three or four? It would also be a code improvement to replace umpteen repetitions of "(str_t){.data = NULL, .size = 0}" throughout the code with a macro.
- mickjc750 4y ago
  That is a good point... Perhaps str_pop_first_split() should pop, even if no delimiters are found. I'll give this some thought. I'll put that macro in too. Thanks.
- mickjc750 4y ago
  Thank you kazinator! I quickly realised you were right about this. It's now fixed. I may have posted this project a little early, but on the other hand it's great to get others input and read all this discussion.
habibur 4y ago
I use a lib like this, but a few changes. printf("The string is %"PRIstr"\n", PRIstrarg(mystring)); Simpler: printf("the string is : %.*s",mystr.size, mystr.data) But that's tedious to write. So create a small macro #define ls(x) (x).size,(x).data And then printf becomes as simple as : printf("the string is : %.*s", ls(mystr)); Though OP's macro is possibly doing more.
- masklinn 4y ago
  > But that's tedious to write. So create a small macro > #define ls(x) (x).size,(x).data Doesn't that double-evaluate `x`?
  - habibur 4y ago
    It does.
naasking 4y ago
Why not ropes? https://github.com/josephg/librope https://github.com/josephg/librope
- abcd_f 4y ago
  An empty rope is 16 bytes on 64bit machines and whooping 24 bytes if it's a wchar rope. In some cases it might be ok, in many others it's not.
pjmlp 4y ago
This is the kind of string libraries that WG14 should care about. Kudos for having a go at it.
gkfasdfasdf 4y ago
Hoping someone can educate me, what are the advantages of having the last member of strbuf_t be a variable length array (char cstr[]) instead of just a char*?
- ComputerGuru 4y ago
  You can store the string as part of the same heap/stack allocation rather than as a separate allocation.
- ksherlock 4y ago
  With inline data, only one malloc is needed for the buffer housekeeping and character data. It's also probably slightly better for cache performance since the housekeeping data and string data are together.
  - gkfasdfasdf 4y ago
    Ah I see. If you want to refer to a string that was not part of this allocation you would use the other str_t type,
Diggsey 4y ago
Slightly ironic that Rust is criticized for having multiple string types, and yet the solution to simplify string handling in C is to introduce the exact same types (str_t == &str, strbuf_t == String) albeit without the safety guarantees.
- estebank 4y ago
  It is still frustrating to me that C still doesn't have a non-allocating method to handle substring references, which both C++ and Rust have. On the other hand I see people trying to parse files, like JSON, in a non-allocating way in Rust and hit a wall until they realize that nodes need to be escaped for anything useful, which requires owning the node's memory (meaning, you need a String or at least Cow<'_, str>, can't get away with a &str).
- adamdusty 4y ago
  I don't think anyone minds that rust has multiple string types just that they're effectively named the same thing so people new to rust have no clue which does what without looking it up. Furthermore people without c/c++ experience mostly wont even know there is a difference since most languages don't give you that control over strings. If rust string were str and strvec or strbuf no one would care.
jandrese 4y ago
This is kind of a bikeshed argument, but I'd prefer if the view was labeled as such. So instead of str_t it would be strview. Rust makes this same mistake IMHO and it causes a lot of confusion for beginners. I would personally call the strbuf_t strstore but that's even more nitpicky. Naming things is one of the hardest problems in CS.
- mickjc750 4y ago
  Hmm. I think it's a good argument. I'll think on it for a day or 2. I went to some effort to explain the difference between str_t and strbuf_t in the readme, but if someone want's to modify some code using this library one day, it's unlikely they will have seen the readme. In 10 years I'll probably have forgotten how this whole thing works, and that someone may be me.
zajio1am 4y ago
1. Ditching null termination makes it cumbersome for interoperability with C ecosystem. 2. It has terrible overhead.
- LegionMammal978 4y ago
  > All strbuf functions maintain a null terminator at the end of the buffer, and the buffer may be accessed as a regular c string using mybuffer->cstr. So effectively a str_t works like an std::string_view from C++, and strbuf_t works like an inline std::string. To produce a null-terminated string from a section of a longer string requires an allocation, unless you can temporarily modify the original string to replace one of its characters with a terminator.
  - zajio1am 4y ago
    Well, the documentation says that null terminator is maintained at the end of the buffer (i.e. mybuffer->cstr[mybuffer->capacity - 1]), not at the end of the string stored in the buffer (i.e. mybuffer->cstr[mybuffer->size]).
    - LegionMammal978 4y ago
      Not sure where you're getting that interpretation from. If you look at the actual code, it sets buf->cstr[buf->size] = 0 every time the string is resized. After all, what else could "the buffer may be accessed as a regular c string" possibly mean?
      - zajio1am 4y ago
        > Not sure where you're getting that interpretation from. That is just plain reading of "null terminator at the end of the buffer", as 'buffer' is just place in memory, regardless of what is stored in it. 'End of the buffer' is commonly used for end of such reserved memory, not end of valid data in that memory. But maintaining the null-terminated string in the buffer is much more useful behavior than just maintaining null terminator at the end of the buffer, so it is likely just sloppiness in the documentation.
andrewmcwatters 4y ago
I want C strings that are compatible with string.h. I want some struct that is a pointer to the char array `s’ with size_t `n’. To meaningfully do this, it means you need auxiliary functions that you execute after calling string.h functions, or you write wrappers that do this for you after calling the relevant string.h functions. I’m OK with that. SDS doesn’t do this. Most other C string libraries like this one basically do what I’m asking for, but not quite. I don’t want separate structs for reading and writing strings. I just want authors to keep it as simple as possible without diverging too hard from how C strings already work today.
- kevin_thibedeau 4y ago
  I have a personal lib that works like this. It maintains a simple struct with a start pointer and a one-past-the-end pointer. You can use it to construct a view or point into unused space at the end of a string for building ops. NUL termination is preserved so interop with stdlib is always available. This allows for nicer string handling while always allowing interop with anything expecting a char *. Libraries with their own string implementation always exact a penalty to get a cstr out.
kaba0 4y ago
All in all, C is still not expressive enough for even such a basic data structure as strings.
thesz 4y ago
Having a string type that has "invalid string" value which is different from empty string value is a bliss. What is important there is that the invalid string value is completely compatible with most C functions - despite actual data pointer is NULL, the length of data is zero so memcmp, memmove/memcpy and most other functions will not segfault. This is really thought out approach. Thank you!
musicale 4y ago
> str.h defines the following str_t type: typedef struct str_t { const char* data; size_t size; } str_t; Sort of a hybrid of C style (pointer) and Pascal style (bounded array) strings?

107 points by mickjc750 4y ago

A convenient C string API, friendly alongside classic C strings.

KingLancelot 4y ago

gjvc 4y ago

worth comparing to https://cr.yp.to/lib/stralloc.html https://cr.yp.to/lib/stralloc.html

plan999 4y ago
You should look at an even better string library. Much more functions and safety for split/joint/tokenizer/etc it's a fork of the plan9 string library bstring. https://bstring.sourceforge.net/ https://bstring.sourceforge.net/

deleted 4y ago

[deleted]

the-printer 4y ago

Are the any good resources that explain the concept of strings in C, particularly why they’re considered to be so difficult to manage? I’m interested in the language, and that along with its safety concerns seem to be the two most frequent complaints against it that I read about online.

e-dant 4y ago
C strings are pointers to memory. There are semantics and assumptions encouraging null-character delimited strings, but not every API follows those rules (just got done working with a Windows API that doesn’t). Often, you have to both null-delimit your string and store its length somewhere. That’s the dangerous part. Messing either of those up, or passing your string to an API that messes that up, is not safe. C strings are pointers to memory, either the stack or the heap, and follow exactly the same rules as everything else in that chaotic space: Not many.
- the-printer 4y ago
  Thank you for this. C programming sounds almost like some sort of combat sport. Riveting.
  - lelanthran 4y ago
    > Thank you for this. C programming sounds almost like some sort of combat sport. Riveting. I've done it for decades; it isn't really as bad as hype-attracting headlines would have you believe. Munitions control, aircraft management systems, industrial automation systems, and many more life-critical systems were programmed in C for decades with comparatively little danger from the language intrinsics leading to death. It's easy to look at the stats and say "there's a few dozen CVEs annually due to C footguns", but that's a few dozen out of hundreds of millions of deployed systems that are written in C. In practice, very few lines of C code bypass the type system, so you get much fewer bugs than an equivalent system in the more usual dynamic programming languages (Python, Javascript, etc).
    - thesnide 4y ago
      Wondering if the big influx of C derived CVE are old or new code. If it is new code, I'm also wondering about the brain damage that those safe languages causes. Yes, it is better to have memory safe languages. But it encourages sloppiness as "nothing can happen". Then those folks aren't fit to write anything else. Which closes the feedback loop on inefficient but safe languages. Which becomes the same thing in airplanes. Pilots don't really know how to fly without instruments anymore.
      - wadd1e 4y ago
        >Which becomes the same thing in airplanes. Pilots don't really know how to fly without instruments anymore. Well that's just a blatantly wrong generalisation you made there, curious as to where you got that from. Consider looking up how pilot training is done before making such assumptions. Even though modern airplanes make heavy use of technology, there are emergency scenarios where lots of instruments may not work, and pilots receive more than enough training to fly an airplane in that scenario just to give one example among tons of others. edit: grammar
jstimpfle 4y ago
"Strings" are quite an abstract concept. They are a linear sequence of characters. But there are a number of ways to represent them - the simplest of which is a contiguous memory allocation, but depending on the use case you'd need more complex schemes. There are also different ways to do the necessary memory management (e.g. allocate statically at compile time vs dynamically at run time). One of the most complex representations is probably the string rope datastructure - a balanced tree of string chunks, supporting efficient insertion and removal anywhere in the string. Specific to C, as well as lots of low-level APIs, is only that strings are often expected to be contiguously laid out in memory and terminated with a NUL (0) byte. So you need to make sure that you always terminate with a NUL after writing to string storage. Other than that, strings aren't any harder than other aspects of programming with manually managed memory. Maybe motivated from higher-level dynamic or managed languages, is the popular idea that strings should always be allocated dynamically (like std::string for example), and support operations like string-append with automatic reallocation if the currently allocated memory isn't enough to store the new string. In practice, that's not true at all - unless you are in a domain where lots of small intermediate strings are generated. This is pretty inefficient anyway and there is likely no point to use C in this case. By far most strings in most domains are either completely static (use string literals), or are created once in a sequence of append operations and then never changed again. I get by, doing many different things from GUI apps to networking to parsers and interpreters, without any sophisticated string type. All I do is define some printf-like APIs to do logging, for example. Those typically just use a fixed size buffer for the formatting, and then flush that buffer to e.g. stderr. or flush it to a dynamically allocated memory buffer, but there almost never is a need to reallocate that string later.
Quentak 4y ago
I have written a short article explaining why null terminated strings as they exist in C cannot represent proper ASCII and UTF-8 because of the null terminator. It's not a full explanation of how strings work but it might be helpful for you. https://kttnr.net/blog/null-terminated-strings-are-incorrect/ https://kttnr.net/blog/null-terminated-strings-are-incorrect...
- lifthrasiir 4y ago
  SQLite does store a null character in strings, it has lots of documented [1] issues in the API level though. [1] https://www.sqlite.org/nulinstr.html https://www.sqlite.org/nulinstr.html
  - Quentak 4y ago
    Thanks for this link. How do you get the null byte into the string? Is it through casting blob to string? The way I have encountered this is when using the C API in which string arguments for prepared statements are passed as char pointers. If those contain the null byte then the string is cut off. Allowing null characters and then mishandling them is worse than not allowing them.
Someone 4y ago
The problem is that C doesn’t have strings; it has functions that treat sequences of non-zero bytes followed by a zero bytes as if they are strings. So, you can’t ask it to create a string that contains the result of appending a string to another one. If you want to append two ‘strings’, you have to create a buffer large enough to hold the result, and then copy in the two sequences of bytes. And even for doing that, the library functions aren’t optimal. The basic “append this string’s data to that string, assuming there’s enough space to do so” function is strcat. It walks the first string to find the zero byte, but to “create a buffer large enough to hold the result” you already must do that. See for example https://stackoverflow.com/questions/21880730/c-what-is-the-best-and-fastest-way-to-concatenate-strings https://stackoverflow.com/questions/21880730/c-what-is-the-b...
- jstimpfle 4y ago
  You can use snprintf to easily achieve any concatentation you'd like. len = snprintf(buffer, buffersize, "%s%s%d", string_1, string_2, int_1); if (len + 1 /*NUL*/ > buffersize) { // not enough space } You can also use this to dynamically allocate any formatted string len = snprintf(NULL, 0, ....); buffersize = len + 1; /* NUL */ buffer = allocate(buffersize); snprintf(buffer, buffersize, ... /*same args as before*/);
  - tom_ 4y ago
    I like the printf family too. Any time you're doing a bunch of strcat or whatever it's almost always massively easier to use a format string to get the same result. Very easy to get the desired width/precision/alignment, and if you need numbers, printf has your back. It even does the bounds checking for you! (And how often do you get that in C.) It won't be as fast, but it's almost always not a problem, and the nice thing about C and C++ is that the char-by-char route is still available when it is. I like to use asprintf, when available: https://man7.org/linux/man-pages/man3/asprintf.3.html https://man7.org/linux/man-pages/man3/asprintf.3.html - and when not available, I add it, along the lines of the snippet you present. Here's something I've found a useful upgrade to asprintf, as it frees the passed-in buffer after expanding the format string. You can just pass the same char ** repeatedly and it'll update the char * appropriately each time. int xasprintf(char**p,const char *fmt,..) { int n=0; char *p2=nullptr; if(fmt) { va_list v; va_start(v); n=asprintf(&p2,fmt,v); va_end(v); } if(n>=0) { free(*p); *p=p2; } return n; }
WalterBright 4y ago
> why they’re considered to be so difficult to manage? Back in the 90s, I was very experienced with C strings and managing them. Then I chanced to look at BASIC again, and realized that strings in BASIC were so simple and intuitive. Why couldn't C be like that? When I started on the design of D, I decided that it had to make strings as easy to do as BASIC did. And D does. The trouble with C strings is the 0 termination of them. This means: 1. to get the length of the string, you have to scan it. This is expensive. 2. when manipulating strings, a common error is to get off by one in the storage because of the 0 termination 3. you cannot get a subset of the string without making a copy. Not only is a copy expensive, but then you have to keep track of the memory for it 4. there's no way to check for buffer overflows D's design, which uses a phat pointer (length, ptr) for strings, solves these problems.
- tored 4y ago
  A year ago I picked up the BASIC dialect PureBasic. Pleasant surprise actually, the syntax of the PureBasic dialect is a bit archaic, but if you accept that it is much easier and faster to get anything done compared to C (and C++). Personally I find low level topics easier to grok in PureBasic than in C even though they mirror the same concepts. PureBasic has Unicode strings built in. It is a bit shame that BASIC has such a bad reputation, there are many BASIC dialects that does the job well still today. https://www.purebasic.com/documentation/reference/ug_string.html https://www.purebasic.com/documentation/reference/ug_string....

cassepipe 4y ago

I always use antirez's (Redis creator) `sds` and advertise it whenvever I get the chance. Thanks to the someone who recommended it on HN some years ago. It's a joy to use. https://github.com/antirez/sds https://github.com/antirez/sds The trick is the size is hidden before the adress of the buffer.("Learn this one simple trick that will change your life for ever"). From the Readme: ``` Advantage #1: you can pass SDS strings to functions designed for C functions without accessing a struct member or calling a function Advantage #2: accessing individual chars is straightforward. Advantage #3: single allocation has better cache locality. Usually when you access a string created by a string library using a structure, you have two different allocations for the structure representing the string, and the actual buffer holding the string. Over the time the buffer is reallocated, and it is likely that it ends in a totally different part of memory compared to the structure itself. Since modern programs performances are often dominated by cache misses, SDS may perform better in many workloads. ```

kazinator 4y ago
Thus, sds it cannot be used for the use cases that this library allows. This library takes string slices without having to allocate or copy memory; it seems to be for use cases involving breaking down strings in complex ways, where good ergonomics and efficiency of obtaining a null-terminated C string are secondary.
quelsolaar 4y ago
Optimizing text is hard because you seldom up front know how much memory will be needed and allocations are slow. Another way to do it is to use: typedef struct{ size_t allocated; size_t used; char buffer[]; }String; This lets the header and the string be the same allocation. Thats a huge saving. Its also useful to store the allocation and use size separatly so that you can reuse / modify buffers. The used field lets you use memcpy without looking for string termination. You can make it even more complex by adding flags if the string is on the stack or on the heap. That way you can do things like: String buffer = MACRO_TO_CREATE_STRING_BUFFER_ON_STACK(256), *b; b = &buffer; b = do_processing_with_buffer(&b); // allocates on heap a larger buffer if needed string_free(b); // frees buffer if its on heap.

program 4y ago

It's better not to use types that end with a '_t' because the suffix is reserved in POSIX systems. https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap02.html#tag_22_02_12 https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh...

cassepipe 4y ago
On the other hand it is quite handy as a prefix, s_ for structs, e_ for enums, g_ for globals, t_ for simple typedefs, f_ for function pointers typedefs, u_ for unions... sky is the limit ! And it's quite easy to create an highlighting rule for it in vim if you still did not convert to treesitter. Just put in ~/.vim/after/c.vim : ``` syn match cType /\<\(t\|s\|e\|u\)_\w\+\>/ ``` Boom, custom type highlighting for C ! Pick the the letters you will use.
- lifthrasiir 4y ago
  No wonder why a significant portion of C programmers actually want to keep tags (`struct` in `struct foo`) instead of removing them wih typedef.
kazinator 4y ago
I don't agree with that at all; it is "the sky might fall" reasoning. Just * have sane naming in your program. * respect namespaces like _[A-Z] and __ * solve clashes that actually happen Historically, revisions of POSIX have introduced identifiers that were not in any previously announced namespace. There is no way you can name an identifier that is guaranteed not to clash with POSIX, or any other vendor. For instance the name "openat" was fine to use in a POSIX program once upon a time. Consider that all strings have the empty string as a suffix. The string "abc" has four suffixes: "abc", "bc", "c" and "". So, every current and future POSIX identifier has "" as a suffix. This is not just a threat; it is guaranteed! Since every identifier in your program also has a "" suffix, it clashes with that namespace. What's wrong with the argument is that identifiers don't just have a suffix; they have to be identical in order to actually clash. (Or have identical prefixes, due to truncation of external names in a linker: decades ago, the limits were ridiculously small.) I doubt that even one person in POSIX standardization would be dumb enough to approve str_t being added as a typedef name in some existing or new header, and multiple approvals are required. Nobody should be losing any sleep over using _t typedef names in their C code.

e-dant 4y ago

I guess it’s nice for a C string API, but what’s the motivation to use and create this? Wouldn’t externing some C++ symbols (or Rust) work more smoothly?

lelanthran 4y ago
> Wouldn’t externing some C++ symbols (or Rust) work more smoothly? For the C++ case, it's not that easy due to C code that cannot handle exceptions thrown in C++ code. For the Rust bit, I'm not sure - creating the library in Rust and letting it be called from C makes the whole rust library unsafe because the data returned from the Rust API would lose ownership information, and is no more safe than simply writing it in C.

kazinator 4y ago

> Attempting to split a string using non-existent delimiter with str_pop_first_split() [returns an invalid string with .data == NULL]. But that seems like a valid case: e.g. these are comma-delimited lists of numbers: "" // empty "1" // one number "20,30" // two numbers the above remark in the documentation seems to be saying (perhaps falsely) that if we try to extract a token from the "1" string using "," as a delimiter, we get an invalid str_t rather than "1". I don't see coverage for this in the tests. There is a test which uses "123/456/789", which extracts the first two splits, and then just verifies that "789" remains. What the programmer wants is to be able to write a loop which will extract "123", "456" and "789" and then* hit the terminating case where the invalid str_t is returned. How many items are in "1,2,3," viewed as comma-separated: three or four? It would also be a code improvement to replace umpteen repetitions of "(str_t){.data = NULL, .size = 0}" throughout the code with a macro.

mickjc750 4y ago
That is a good point... Perhaps str_pop_first_split() should pop, even if no delimiters are found. I'll give this some thought. I'll put that macro in too. Thanks.
mickjc750 4y ago
Thank you kazinator! I quickly realised you were right about this. It's now fixed. I may have posted this project a little early, but on the other hand it's great to get others input and read all this discussion.

habibur 4y ago

I use a lib like this, but a few changes. printf("The string is %"PRIstr"\n", PRIstrarg(mystring)); Simpler: printf("the string is : %.*s",mystr.size, mystr.data) But that's tedious to write. So create a small macro #define ls(x) (x).size,(x).data And then printf becomes as simple as : printf("the string is : %.*s", ls(mystr)); Though OP's macro is possibly doing more.

masklinn 4y ago
> But that's tedious to write. So create a small macro > #define ls(x) (x).size,(x).data Doesn't that double-evaluate `x`?
- habibur 4y ago
  It does.

naasking 4y ago

Why not ropes? https://github.com/josephg/librope https://github.com/josephg/librope

abcd_f 4y ago
An empty rope is 16 bytes on 64bit machines and whooping 24 bytes if it's a wchar rope. In some cases it might be ok, in many others it's not.

pjmlp 4y ago

This is the kind of string libraries that WG14 should care about. Kudos for having a go at it.

gkfasdfasdf 4y ago

Hoping someone can educate me, what are the advantages of having the last member of strbuf_t be a variable length array (char cstr[]) instead of just a char*?

ComputerGuru 4y ago
You can store the string as part of the same heap/stack allocation rather than as a separate allocation.
ksherlock 4y ago
With inline data, only one malloc is needed for the buffer housekeeping and character data. It's also probably slightly better for cache performance since the housekeeping data and string data are together.
- gkfasdfasdf 4y ago
  Ah I see. If you want to refer to a string that was not part of this allocation you would use the other str_t type,

Diggsey 4y ago

Slightly ironic that Rust is criticized for having multiple string types, and yet the solution to simplify string handling in C is to introduce the exact same types (str_t == &str, strbuf_t == String) albeit without the safety guarantees.

estebank 4y ago
It is still frustrating to me that C still doesn't have a non-allocating method to handle substring references, which both C++ and Rust have. On the other hand I see people trying to parse files, like JSON, in a non-allocating way in Rust and hit a wall until they realize that nodes need to be escaped for anything useful, which requires owning the node's memory (meaning, you need a String or at least Cow<'_, str>, can't get away with a &str).
adamdusty 4y ago
I don't think anyone minds that rust has multiple string types just that they're effectively named the same thing so people new to rust have no clue which does what without looking it up. Furthermore people without c/c++ experience mostly wont even know there is a difference since most languages don't give you that control over strings. If rust string were str and strvec or strbuf no one would care.

jandrese 4y ago

This is kind of a bikeshed argument, but I'd prefer if the view was labeled as such. So instead of str_t it would be strview. Rust makes this same mistake IMHO and it causes a lot of confusion for beginners. I would personally call the strbuf_t strstore but that's even more nitpicky. Naming things is one of the hardest problems in CS.

mickjc750 4y ago
Hmm. I think it's a good argument. I'll think on it for a day or 2. I went to some effort to explain the difference between str_t and strbuf_t in the readme, but if someone want's to modify some code using this library one day, it's unlikely they will have seen the readme. In 10 years I'll probably have forgotten how this whole thing works, and that someone may be me.

zajio1am 4y ago

1. Ditching null termination makes it cumbersome for interoperability with C ecosystem. 2. It has terrible overhead.

LegionMammal978 4y ago
> All strbuf functions maintain a null terminator at the end of the buffer, and the buffer may be accessed as a regular c string using mybuffer->cstr. So effectively a str_t works like an std::string_view from C++, and strbuf_t works like an inline std::string. To produce a null-terminated string from a section of a longer string requires an allocation, unless you can temporarily modify the original string to replace one of its characters with a terminator.
- zajio1am 4y ago
  Well, the documentation says that null terminator is maintained at the end of the buffer (i.e. mybuffer->cstr[mybuffer->capacity - 1]), not at the end of the string stored in the buffer (i.e. mybuffer->cstr[mybuffer->size]).
  - LegionMammal978 4y ago
    Not sure where you're getting that interpretation from. If you look at the actual code, it sets buf->cstr[buf->size] = 0 every time the string is resized. After all, what else could "the buffer may be accessed as a regular c string" possibly mean?
    - zajio1am 4y ago
      > Not sure where you're getting that interpretation from. That is just plain reading of "null terminator at the end of the buffer", as 'buffer' is just place in memory, regardless of what is stored in it. 'End of the buffer' is commonly used for end of such reserved memory, not end of valid data in that memory. But maintaining the null-terminated string in the buffer is much more useful behavior than just maintaining null terminator at the end of the buffer, so it is likely just sloppiness in the documentation.

andrewmcwatters 4y ago

I want C strings that are compatible with string.h. I want some struct that is a pointer to the char array `s’ with size_t `n’. To meaningfully do this, it means you need auxiliary functions that you execute after calling string.h functions, or you write wrappers that do this for you after calling the relevant string.h functions. I’m OK with that. SDS doesn’t do this. Most other C string libraries like this one basically do what I’m asking for, but not quite. I don’t want separate structs for reading and writing strings. I just want authors to keep it as simple as possible without diverging too hard from how C strings already work today.

kevin_thibedeau 4y ago
I have a personal lib that works like this. It maintains a simple struct with a start pointer and a one-past-the-end pointer. You can use it to construct a view or point into unused space at the end of a string for building ops. NUL termination is preserved so interop with stdlib is always available. This allows for nicer string handling while always allowing interop with anything expecting a char *. Libraries with their own string implementation always exact a penalty to get a cstr out.

kaba0 4y ago

All in all, C is still not expressive enough for even such a basic data structure as strings.

thesz 4y ago

Having a string type that has "invalid string" value which is different from empty string value is a bliss. What is important there is that the invalid string value is completely compatible with most C functions - despite actual data pointer is NULL, the length of data is zero so memcmp, memmove/memcpy and most other functions will not segfault. This is really thought out approach. Thank you!

musicale 4y ago

> str.h defines the following str_t type: typedef struct str_t { const char* data; size_t size; } str_t; Sort of a hybrid of C style (pointer) and Pascal style (bounded array) strings?