13 ms·

Ushering out strlcpy()

166 points by magnio 4y ago

throw0101a 4y ago
The original strlcpy() was written for OpenBSD (2.4) by Todd C. Miller (who has also been the maintainer of sudo since 1994): * https://github.com/openbsd/src/blob/master/lib/libc/string/strlcpy.c https://github.com/openbsd/src/blob/master/lib/libc/string/s... * https://cvsweb.openbsd.org/src/lib/libc/string/strlcpy.c https://cvsweb.openbsd.org/src/lib/libc/string/strlcpy.c
- ncmncm 4y ago
  Sudo has had its own debacles.
  - throw0101a 4y ago
    Feel free to use one of the alternatives: * https://www.sudo.ws/docs/alternatives/ https://www.sudo.ws/docs/alternatives/
- wahern 4y ago
  Don't forget the original USENIX paper: https://www.usenix.org/legacy/events/usenix99/full_papers/millert/millert.pdf https://www.usenix.org/legacy/events/usenix99/full_papers/mi...
matheusmoreira 4y ago
So why insist on this NUL termination business? I expected this to be necessary in C libraries due to legacy code but the kernel can always do whatever it wants internally. struct bytes { size_t size; unsigned char *pointer; }; Seems simple enough... A little structure like that has always been useful in my projects. I find myself agreeing with the glibc maintainer in the extended discussion linked from the article: https://lwn.net/Articles/612244/ https://lwn.net/Articles/612244/ > Correct string handling means that you always know how long your strings are and therefore you can you memcpy (instead of strcpy). I think he's absolutely right. The str* functions are just worse versions of the mem* functions.
- _8j50 4y ago
  Can you imagine the conversion effort?
  - XorNot 4y ago
    I can imagine it's not been analysed so we're all assuming it's just "impossible". Sometimes you just need to do the boring work so it's done. The Linux kernel is one of the most important pieces of software on the planet - limiting it's performance and safety due to C's string handling legacy is madness.
    - _8j50 4y ago
      Have you read the article? Linus' resistance to auto-conversion means one or a few people can't do it, every maintainer has to do a project involving many orgs and thousands of people over several years. It is not impossible, it takes time but it is doable, the question is is this the best way to solve the underlying issue (and what exactly is that?) ? Assuming of course the NUL replacement is backwards compatible in some way, if not then I say focus on migrating to Rust instead.
      - JdeBP 4y ago
        M. Torvalds never expressed an objection to auto-conversion. Xe expressed an objection to mass conversion, with patches that mass change function calls with little scrutiny and even less actual regression testing of all of the affected code paths. Because xyr past experience was that that was what happened in practice, and stuff broke as a consequence.
        pwdisswordfish9 4y ago
        Who is M. Torvalds? I am not aware of anyone with first initial M. named Torvalds that is relevant to this discussion and would require such bizarre pronouns to be applied to them.
Night_Thastus 4y ago
It's so funny to read all this. In college, we were always told, "don't use strcpy! Use strncpy instead!" just to find out later on that strncpy isn't that great either. Then I heard about the better strlcpy, and even now I'm hearing that isn't quite good enough.
- masklinn 4y ago
  > It's so funny to read all this. In college, we were always told, "don't use strcpy! Use strncpy instead!" just to find out later on that strncpy isn't that great either. TBF that was definitely incorrect, as strncpy was never intended to work with C strings. Most (though not all, that would be too easy) of the n functions work with fixed-size strings. The behaviour of strncpy makes perfect sense then.
  - saagarjha 4y ago
    I (and most documentation) prefers the term “character array” to make the distinction clear.
    - techwiz137 4y ago
      I guess I should refer to people as "collection of atoms", then. Or even further to probability clouds, to make a distinction.
      - saagarjha 4y ago
        In which case people who are deceased can be said to be termin…nevermind.
  - knorker 4y ago
    Why would you use strncpy on non-c-strings? Why not use memcpy?
    - masklinn 4y ago
userbinator 4y ago
It's funny to see all these trends come and go for such a fundamentally simple problem. I'm still of the opinion that strcpy() with a good calculation of lengths is the simplest on the occasion that you truly do need to copy a string, which IMHO is often done far more than really necessary. The main point is that the length calculation and check should've been done before the copy; if you find out only when you do the copy that it's too long, then it's already too late. An "anti-pattern" I seem to be seeing increasingly often in newer code is that of allocating a string (or worse, a dynamically expanding buffer), copying several other strings to it, then only calling another function with that concatenated copy before freeing it, when the other function could've simply been called several times with the individual parts successively.
- jandrese 4y ago
  strlcpy's flaw is its return value. In order to get the length of the string it has to walk the entire thing. If you are copying 20 character strings out of a 20TB mmaped file it will be outrageously slow and that's an unnecessary footgun. It should have just returned the number of bytes copied to dst if the string was successfully copied and a -1 with an errno of E2BIG if the dst was too small for the src. It would still do the copy and termination of course, and the programmer will know the length in this case because they specified it in the function call. Of course this is what strscpy does. If you aren't sure about the length of the src buffer strlcpy can be a ticking time bomb. If you aren't sure if the src string is NULL terminated it's also a problem, which is especially bad since one of the big reasons to use strlcpy is to avoid buffer overruns. GTA Online suffered from outrageous load times due to this very same API quirk. This would also make it easy for the function to return -1 and set errno to EINVAL if you specify NULL for the src or dst.
- deleted 4y ago
  [deleted]
- xorvoid 4y ago
  I don’t understand. If you’re doing length checks before “strcpy()”, then you can easily call “memcpy()” instead. And.. memcpy() is faster because it can go word at a time. So, why would anyone ever need “strcpy()”?
zzo38computer 4y ago
You can use strncpy if you are careful to do it right (e.g. if you know the end of the destination buffer is null-terminated, specify the length one less than the destination buffer size). Sometimes memcpy is better, though. You can use both in one program.
- masklinn 4y ago
  > You can use strncpy if you are careful to do it right That’s like saying if you’re careful to do it right you can use a Bowie knife as a screwdriver. Technically correct but practically will end in tears. If you’re not working with fixed-size strings (which few are these days) just copy over strscpy, or strlcpy, 5 lines won’t kill you.
- kevin_thibedeau 4y ago
  You're still stuck with the zero pad performance hit when the destination is significantly larger than the source.
- noselasd 4y ago
  But you don't want to use strncpy, especially not inside the kernel - since it may waste a lot of useless cpu. char big_enough[512]; strncpy(big_enough, sizeof 512, small_string]; strncpy now fills the rest of big_enough with nuls, a lot of waste if small_string is 4 characters long.
- formerly_proven 4y ago
  The express purpose of strncpy is to handle fixed-length strings in classic unix (file) structures (e.g. tar), I'm not convinced it's correct to use in basically any other situation.
dwheeler 4y ago
To be fair, they are replacing strlcpy() with the very similar strscpy(). I do think that strscpy is an improvement, but it's quite similar.
Cloudef 4y ago
The only string manipulation function you need is snprintf, every other str* function is dumb
- adrian_b 4y ago
  Using snprintf is equivalent with using strlcpy and also equivalent with using a strlen followed by a memcpy. As explained in the parent article, using something equivalent with strlen is inefficient when the source is much longer than the destination, because you do not need the length of the source, you just need to know that it is longer than the destination. Instead of defining yet another strcpy-like function, I would have preferred an alternative to strlen, taking 2 arguments, the size of the destination and a pointer to the source, and returning either the length of the source or an error when the source is longer than the provided size. Then every strcpy can be replaced with that strlen-replacement function followed by memcpy, and the same strlen replacement can be used with all other string functions, e.g. with strcat. POSIX 2008 has added the strnlen function ("size_t strnlen(const char *s, size_t maxlen);"), which does not have the inefficiency of strlen, by returning at most the size of the destination, but it does not signal an error to indicate truncation. You can still use strnlen, if you give it a size larger by 1 and you check the return value to detect an oversize source, but it is slightly less convenient than if the strnlen return value would have been defined like the return value of strscpy.
  - saagarjha 4y ago
    memchr?
    - adrian_b 4y ago
      memchr is less convenient even than strnlen, because it returns a pointer, so you must subtract 2 pointers to get the size that needs to be used in memcpy. memchr is indeed an alternative for someone who would need to use some old libc, without the strnlen function. The strscpy function is available only inside the Linux kernel, but strnlen should be available in any environment that conforms to POSIX 2008, so strnlen + memcpy is what I would recommend for copying null-terminated strings in any C program.
hedora 4y ago
I love how, from reading the article, you can't tell if the count parameter of the proposed replacement includes the null terminator (is the length of the dest buffer) or not (is the length - 1). Also, it is unclear what side effects the function has if it returns ERR2BIG. Is dest null terminated or not in that case? I could figure these things out, but my point is that all these str*cpy functions are fundamentally error prone because other people can't keep those details straight either, apparently. I've never had an issue using 1+strlen() to figure out how many bytes to copy, checking the destination buffer size and then invoking memcpy. It's a bit inefficient (two passes, though both are usually using vectorized instructions) but at least it is really clear to the reader. The main problem is that C lets you typo this: strlen(s+1) When you mean this: strlen(s)+1 Also, of course, for untrusted buffers, you can't use strlen().
- saagarjha 4y ago
  strscpy always null terminates, unless the destination buffer has size zero. -E2BIG is returned when the entire string could not be copied. The return value is essentially strlen of the resulting string. It’s good all around.
  - hedora 4y ago
    Even with that, I still don't have enough information to safely invoke it. (What is count?) These APIs are too complicated/subtle. Honestly, at this point, I just use (ptr, len) pairs that point to non-null terminated strings whenever possible.
    - masklinn 4y ago
      > Even with that, I still don't have enough information to safely invoke it. (What is count?) Count is necessarily the target buffer’s size since it’s what you don’t want overrun.
      - deleted 4y ago
        [deleted]
saagarjha 4y ago
strscpy, unlike literally everything in the standard but memccpy, is a good replacement. Not only does it avoid the usual performance pitfalls, it lets you safely create your own desired functionality on top of it: in other words, it’s is composable. This is not true of any of the other functions. You can make your own strlcpy or strncpy (not sure why you would, but bear with me) out of strscpy. Trying to go in the reverse direction without paying a large cost is not possible. (For a more detailed discussion, I wrote a whole blog post about why strncpy is good and it got posted here a while back: https://news.ycombinator.com/item?id=27537900 https://news.ycombinator.com/item?id=27537900)
- saagarjha 4y ago
  *why strscpy is good, my iPhone has unfortunately learned the spelling of commonly-used C functions
ncmncm 4y ago
It seems like a rare bout of sanity for strlcpy to have fallen out of favor, in the kernel. It never got into glibc, which was I guess another. But of course it is all over the BSDs, and not going anywhere AFaIK, more's the pity.
- kevin_thibedeau 4y ago
  Glibc isn't much of a standard of excellence for rational behavior considering they have malloc_info() dumping out XML rather than actually returning a useful data structure.
  - ncmncm 4y ago
    Maybe "rare bout of sanity" is an unfamiliar concept?
- JdeBP 4y ago
  The BSD kernels do not have nearly as many APIs that employ variable-length special-character-terminated strings that have to be parsed from human-readable forms, however. Whilst this is common for things like procfs in the Linux world, the BSDs favour general purpose interfaces like sysctl(3) where data are transferred in machine-parseable forms rather than human-readable ones. So "all over the BSDs" rather glosses over the fact that whilst common in applications code, there's less need for these sorts of string functions in the kernel. (And it is the Linux kernel that is the headlined subject here.) strlcpy() is in libkern, but it's not anywhere near as frequent in the kernel as it is in applications code.
hyperman1 4y ago
I've always wondered why the C standard didn't just bite the bullet and added a string object, with a malloc-backed expanding string inside.
- Ekaros 4y ago
  I think there is sufficient membership in standard committee that doesn't want such "automated" features specially when touch memory allocation and freeing. And I too find it somewhat antithetical to C as language. Still. Why isn't there a good quality library or set of libraries that would cover these features is my question.
  - rabf 4y ago
    https://github.com/antirez/sds https://github.com/antirez/sds
    - hyperman1 4y ago
      This is exactly the kind of thing I was thinking about. Great!
    - deleted 4y ago
      [deleted]
- tialaramex 4y ago
  Unlike C++ the C language takes its freestanding mode seriously. In this mode there's no malloc() and so your string type can't exist. But today in freestanding I can write "ERROR" and get a string literal. So that would mean either you abolish string literals in freestanding (making C worse) or you have a separate type for these string literals from the type for a expandable string. Now, Rust pulls off the latter in my view fairly elegantly, but that takes a lot of sophisticated type features in the language including a DeRef and several AsRef implementations. In Rust these are general features open to other types, C could special case them instead, but that's still a lot of engineering for a small feature. The result is definitely not the elegant language which can be described in a slim book. Maybe it's better anyway, but it's not C.
agumonkey 4y ago
50 years down the road, and still arguing about strings
1500100900 4y ago
> The BSD answer to the problems with strncpy() was to introduce a new function called strlcpy() Not BSD, it was OpenBSD specifically.
minusf 4y ago
i havent used C in quite a while but I did not remember strlcpy using strlen, and it seems the linux kernel's implementation differs quite a bit from the openbsd one: https://elixir.bootlin.com/linux/v5.19.3/source/lib/string.c#L125 https://elixir.bootlin.com/linux/v5.19.3/source/lib/string.c... https://github.com/openbsd/src/blob/master/lib/libc/string/strlcpy.c https://github.com/openbsd/src/blob/master/lib/libc/string/s...
- qhwudbebd 4y ago
  The while (*src++); on line 45 of lib/libc/string/strlcpy.c is an inline implementation of strlen here. It's done solely for the benefit of the return value (src - osrc - 1) but means that it traverses the entire length of src until the first NUL - even if that's gigabytes or wanders out of mapped memory.
ufo 4y ago
Are there any userspace libraries that provide strscpy, or do we need to reimplement it ourselves?
avidphantasm 4y ago
Dumb(?) question from a non-kernel programmer: can strings in the kernel be limited to those with lengths <2^16 (65536)? Then the length can be stored in the unused bits of the pointer. This probably creates more problems than it solves, especially if we ever have CPUs that want to address more than 256 GiB of memory. This would also only work on 64-bit architectures.
- avidphantasm 4y ago
  Or if say, most strings tend to be small (<256 chars), use 8 unused bits of the pointer for the length. If the string is longer than that, mask the bits to 0. Then the string handling functions can have a fast path that uses the length from pointer if available, if not, walks the entire string.
  - jabl 4y ago
    I think most(?) C++ implementations of std::string use something called a short-string optimization, where the string type is something like struct string { size_t len; union { char* ptr_to_malloced_string; char short_string[sizeof(char*)]; } } So if the length of the string is shorter than the size of a pointer, the string is stored inline in the struct, and if the length is longer it's a separate allocated object.
- notamy 4y ago
  > especially if we ever have CPUs that want to address more than 256 GiB of memory. Given how common of a scenario this is for Linux, I imagine this sort of a change would be more trouble than it's worth...
- mort96 4y ago
  FYI, we have consumer systems with over a terabyte of RAM.
deleted 4y ago
[deleted]
thayne 4y ago
Is there a userland function equivalent to strscpy in glibc?
trasz 4y ago
Tangentially related: FreeBSD tends to use sbuf(3) (https://www.freebsd.org/cgi/man.cgi?query=sbuf https://www.freebsd.org/cgi/man.cgi?query=sbuf) for anything but the very simplest string manipulation. It’s safer and more convenient than NULL-terminated strings.
benj111 4y ago
I'm confused about the race conditions mentioned. Can't you just copy a byte to a buffer checking for null, counting down the length. At the assembly level you have to copy a byte at a time anyway. I don't see why you'd want to check the entire length of the input string for strlcpy either. It seems less than optimal. I'm guessing the people involved know more than me. So...?

166 points by magnio 4y ago

The original strlcpy() was written for OpenBSD (2.4) by Todd C. Miller (who has also been the maintainer of sudo since 1994): * https://github.com/openbsd/src/blob/master/lib/libc/string/strlcpy.c https://github.com/openbsd/src/blob/master/lib/libc/string/s... * https://cvsweb.openbsd.org/src/lib/libc/string/strlcpy.c https://cvsweb.openbsd.org/src/lib/libc/string/strlcpy.c

ncmncm 4y ago
Sudo has had its own debacles.
- throw0101a 4y ago
  Feel free to use one of the alternatives: * https://www.sudo.ws/docs/alternatives/ https://www.sudo.ws/docs/alternatives/
wahern 4y ago
Don't forget the original USENIX paper: https://www.usenix.org/legacy/events/usenix99/full_papers/millert/millert.pdf https://www.usenix.org/legacy/events/usenix99/full_papers/mi...

matheusmoreira 4y ago

So why insist on this NUL termination business? I expected this to be necessary in C libraries due to legacy code but the kernel can always do whatever it wants internally. struct bytes { size_t size; unsigned char *pointer; }; Seems simple enough... A little structure like that has always been useful in my projects. I find myself agreeing with the glibc maintainer in the extended discussion linked from the article: https://lwn.net/Articles/612244/ https://lwn.net/Articles/612244/ > Correct string handling means that you always know how long your strings are and therefore you can you memcpy (instead of strcpy). I think he's absolutely right. The str* functions are just worse versions of the mem* functions.

_8j50 4y ago
Can you imagine the conversion effort?
- XorNot 4y ago
  I can imagine it's not been analysed so we're all assuming it's just "impossible". Sometimes you just need to do the boring work so it's done. The Linux kernel is one of the most important pieces of software on the planet - limiting it's performance and safety due to C's string handling legacy is madness.
  - _8j50 4y ago
    Have you read the article? Linus' resistance to auto-conversion means one or a few people can't do it, every maintainer has to do a project involving many orgs and thousands of people over several years. It is not impossible, it takes time but it is doable, the question is is this the best way to solve the underlying issue (and what exactly is that?) ? Assuming of course the NUL replacement is backwards compatible in some way, if not then I say focus on migrating to Rust instead.
    - JdeBP 4y ago
      M. Torvalds never expressed an objection to auto-conversion. Xe expressed an objection to mass conversion, with patches that mass change function calls with little scrutiny and even less actual regression testing of all of the affected code paths. Because xyr past experience was that that was what happened in practice, and stuff broke as a consequence.
      - pwdisswordfish9 4y ago
        Who is M. Torvalds? I am not aware of anyone with first initial M. named Torvalds that is relevant to this discussion and would require such bizarre pronouns to be applied to them.

Night_Thastus 4y ago

It's so funny to read all this. In college, we were always told, "don't use strcpy! Use strncpy instead!" just to find out later on that strncpy isn't that great either. Then I heard about the better strlcpy, and even now I'm hearing that isn't quite good enough.

masklinn 4y ago
> It's so funny to read all this. In college, we were always told, "don't use strcpy! Use strncpy instead!" just to find out later on that strncpy isn't that great either. TBF that was definitely incorrect, as strncpy was never intended to work with C strings. Most (though not all, that would be too easy) of the n functions work with fixed-size strings. The behaviour of strncpy makes perfect sense then.
- saagarjha 4y ago
  I (and most documentation) prefers the term “character array” to make the distinction clear.
  - techwiz137 4y ago
    I guess I should refer to people as "collection of atoms", then. Or even further to probability clouds, to make a distinction.
    - saagarjha 4y ago
      In which case people who are deceased can be said to be termin…nevermind.
- knorker 4y ago
  Why would you use strncpy on non-c-strings? Why not use memcpy?
  - masklinn 4y ago

userbinator 4y ago

It's funny to see all these trends come and go for such a fundamentally simple problem. I'm still of the opinion that strcpy() with a good calculation of lengths is the simplest on the occasion that you truly do need to copy a string, which IMHO is often done far more than really necessary. The main point is that the length calculation and check should've been done before the copy; if you find out only when you do the copy that it's too long, then it's already too late. An "anti-pattern" I seem to be seeing increasingly often in newer code is that of allocating a string (or worse, a dynamically expanding buffer), copying several other strings to it, then only calling another function with that concatenated copy before freeing it, when the other function could've simply been called several times with the individual parts successively.

jandrese 4y ago
strlcpy's flaw is its return value. In order to get the length of the string it has to walk the entire thing. If you are copying 20 character strings out of a 20TB mmaped file it will be outrageously slow and that's an unnecessary footgun. It should have just returned the number of bytes copied to dst if the string was successfully copied and a -1 with an errno of E2BIG if the dst was too small for the src. It would still do the copy and termination of course, and the programmer will know the length in this case because they specified it in the function call. Of course this is what strscpy does. If you aren't sure about the length of the src buffer strlcpy can be a ticking time bomb. If you aren't sure if the src string is NULL terminated it's also a problem, which is especially bad since one of the big reasons to use strlcpy is to avoid buffer overruns. GTA Online suffered from outrageous load times due to this very same API quirk. This would also make it easy for the function to return -1 and set errno to EINVAL if you specify NULL for the src or dst.
deleted 4y ago
[deleted]
xorvoid 4y ago
I don’t understand. If you’re doing length checks before “strcpy()”, then you can easily call “memcpy()” instead. And.. memcpy() is faster because it can go word at a time. So, why would anyone ever need “strcpy()”?

zzo38computer 4y ago

You can use strncpy if you are careful to do it right (e.g. if you know the end of the destination buffer is null-terminated, specify the length one less than the destination buffer size). Sometimes memcpy is better, though. You can use both in one program.

masklinn 4y ago
> You can use strncpy if you are careful to do it right That’s like saying if you’re careful to do it right you can use a Bowie knife as a screwdriver. Technically correct but practically will end in tears. If you’re not working with fixed-size strings (which few are these days) just copy over strscpy, or strlcpy, 5 lines won’t kill you.
kevin_thibedeau 4y ago
You're still stuck with the zero pad performance hit when the destination is significantly larger than the source.
noselasd 4y ago
But you don't want to use strncpy, especially not inside the kernel - since it may waste a lot of useless cpu. char big_enough[512]; strncpy(big_enough, sizeof 512, small_string]; strncpy now fills the rest of big_enough with nuls, a lot of waste if small_string is 4 characters long.
formerly_proven 4y ago
The express purpose of strncpy is to handle fixed-length strings in classic unix (file) structures (e.g. tar), I'm not convinced it's correct to use in basically any other situation.

dwheeler 4y ago

To be fair, they are replacing strlcpy() with the very similar strscpy(). I do think that strscpy is an improvement, but it's quite similar.

Cloudef 4y ago

The only string manipulation function you need is snprintf, every other str* function is dumb

adrian_b 4y ago
Using snprintf is equivalent with using strlcpy and also equivalent with using a strlen followed by a memcpy. As explained in the parent article, using something equivalent with strlen is inefficient when the source is much longer than the destination, because you do not need the length of the source, you just need to know that it is longer than the destination. Instead of defining yet another strcpy-like function, I would have preferred an alternative to strlen, taking 2 arguments, the size of the destination and a pointer to the source, and returning either the length of the source or an error when the source is longer than the provided size. Then every strcpy can be replaced with that strlen-replacement function followed by memcpy, and the same strlen replacement can be used with all other string functions, e.g. with strcat. POSIX 2008 has added the strnlen function ("size_t strnlen(const char *s, size_t maxlen);"), which does not have the inefficiency of strlen, by returning at most the size of the destination, but it does not signal an error to indicate truncation. You can still use strnlen, if you give it a size larger by 1 and you check the return value to detect an oversize source, but it is slightly less convenient than if the strnlen return value would have been defined like the return value of strscpy.
- saagarjha 4y ago
  memchr?
  - adrian_b 4y ago
    memchr is less convenient even than strnlen, because it returns a pointer, so you must subtract 2 pointers to get the size that needs to be used in memcpy. memchr is indeed an alternative for someone who would need to use some old libc, without the strnlen function. The strscpy function is available only inside the Linux kernel, but strnlen should be available in any environment that conforms to POSIX 2008, so strnlen + memcpy is what I would recommend for copying null-terminated strings in any C program.

hedora 4y ago

I love how, from reading the article, you can't tell if the count parameter of the proposed replacement includes the null terminator (is the length of the dest buffer) or not (is the length - 1). Also, it is unclear what side effects the function has if it returns ERR2BIG. Is dest null terminated or not in that case? I could figure these things out, but my point is that all these str*cpy functions are fundamentally error prone because other people can't keep those details straight either, apparently. I've never had an issue using 1+strlen() to figure out how many bytes to copy, checking the destination buffer size and then invoking memcpy. It's a bit inefficient (two passes, though both are usually using vectorized instructions) but at least it is really clear to the reader. The main problem is that C lets you typo this: strlen(s+1) When you mean this: strlen(s)+1 Also, of course, for untrusted buffers, you can't use strlen().

saagarjha 4y ago
strscpy always null terminates, unless the destination buffer has size zero. -E2BIG is returned when the entire string could not be copied. The return value is essentially strlen of the resulting string. It’s good all around.
- hedora 4y ago
  Even with that, I still don't have enough information to safely invoke it. (What is count?) These APIs are too complicated/subtle. Honestly, at this point, I just use (ptr, len) pairs that point to non-null terminated strings whenever possible.
  - masklinn 4y ago
    > Even with that, I still don't have enough information to safely invoke it. (What is count?) Count is necessarily the target buffer’s size since it’s what you don’t want overrun.
    - deleted 4y ago
      [deleted]

saagarjha 4y ago

strscpy, unlike literally everything in the standard but memccpy, is a good replacement. Not only does it avoid the usual performance pitfalls, it lets you safely create your own desired functionality on top of it: in other words, it’s is composable. This is not true of any of the other functions. You can make your own strlcpy or strncpy (not sure why you would, but bear with me) out of strscpy. Trying to go in the reverse direction without paying a large cost is not possible. (For a more detailed discussion, I wrote a whole blog post about why strncpy is good and it got posted here a while back: https://news.ycombinator.com/item?id=27537900 https://news.ycombinator.com/item?id=27537900)

saagarjha 4y ago
*why strscpy is good, my iPhone has unfortunately learned the spelling of commonly-used C functions

ncmncm 4y ago

It seems like a rare bout of sanity for strlcpy to have fallen out of favor, in the kernel. It never got into glibc, which was I guess another. But of course it is all over the BSDs, and not going anywhere AFaIK, more's the pity.

kevin_thibedeau 4y ago
Glibc isn't much of a standard of excellence for rational behavior considering they have malloc_info() dumping out XML rather than actually returning a useful data structure.
- ncmncm 4y ago
  Maybe "rare bout of sanity" is an unfamiliar concept?
JdeBP 4y ago
The BSD kernels do not have nearly as many APIs that employ variable-length special-character-terminated strings that have to be parsed from human-readable forms, however. Whilst this is common for things like procfs in the Linux world, the BSDs favour general purpose interfaces like sysctl(3) where data are transferred in machine-parseable forms rather than human-readable ones. So "all over the BSDs" rather glosses over the fact that whilst common in applications code, there's less need for these sorts of string functions in the kernel. (And it is the Linux kernel that is the headlined subject here.) strlcpy() is in libkern, but it's not anywhere near as frequent in the kernel as it is in applications code.

hyperman1 4y ago

I've always wondered why the C standard didn't just bite the bullet and added a string object, with a malloc-backed expanding string inside.

Ekaros 4y ago
I think there is sufficient membership in standard committee that doesn't want such "automated" features specially when touch memory allocation and freeing. And I too find it somewhat antithetical to C as language. Still. Why isn't there a good quality library or set of libraries that would cover these features is my question.
- rabf 4y ago
  https://github.com/antirez/sds https://github.com/antirez/sds
  - hyperman1 4y ago
    This is exactly the kind of thing I was thinking about. Great!
  - deleted 4y ago
    [deleted]
tialaramex 4y ago
Unlike C++ the C language takes its freestanding mode seriously. In this mode there's no malloc() and so your string type can't exist. But today in freestanding I can write "ERROR" and get a string literal. So that would mean either you abolish string literals in freestanding (making C worse) or you have a separate type for these string literals from the type for a expandable string. Now, Rust pulls off the latter in my view fairly elegantly, but that takes a lot of sophisticated type features in the language including a DeRef and several AsRef implementations. In Rust these are general features open to other types, C could special case them instead, but that's still a lot of engineering for a small feature. The result is definitely not the elegant language which can be described in a slim book. Maybe it's better anyway, but it's not C.

agumonkey 4y ago

50 years down the road, and still arguing about strings

1500100900 4y ago

> The BSD answer to the problems with strncpy() was to introduce a new function called strlcpy() Not BSD, it was OpenBSD specifically.

minusf 4y ago

i havent used C in quite a while but I did not remember strlcpy using strlen, and it seems the linux kernel's implementation differs quite a bit from the openbsd one: https://elixir.bootlin.com/linux/v5.19.3/source/lib/string.c#L125 https://elixir.bootlin.com/linux/v5.19.3/source/lib/string.c... https://github.com/openbsd/src/blob/master/lib/libc/string/strlcpy.c https://github.com/openbsd/src/blob/master/lib/libc/string/s...

qhwudbebd 4y ago
The while (*src++); on line 45 of lib/libc/string/strlcpy.c is an inline implementation of strlen here. It's done solely for the benefit of the return value (src - osrc - 1) but means that it traverses the entire length of src until the first NUL - even if that's gigabytes or wanders out of mapped memory.

ufo 4y ago

Are there any userspace libraries that provide strscpy, or do we need to reimplement it ourselves?

avidphantasm 4y ago

Dumb(?) question from a non-kernel programmer: can strings in the kernel be limited to those with lengths <2^16 (65536)? Then the length can be stored in the unused bits of the pointer. This probably creates more problems than it solves, especially if we ever have CPUs that want to address more than 256 GiB of memory. This would also only work on 64-bit architectures.

avidphantasm 4y ago
Or if say, most strings tend to be small (<256 chars), use 8 unused bits of the pointer for the length. If the string is longer than that, mask the bits to 0. Then the string handling functions can have a fast path that uses the length from pointer if available, if not, walks the entire string.
- jabl 4y ago
  I think most(?) C++ implementations of std::string use something called a short-string optimization, where the string type is something like struct string { size_t len; union { char* ptr_to_malloced_string; char short_string[sizeof(char*)]; } } So if the length of the string is shorter than the size of a pointer, the string is stored inline in the struct, and if the length is longer it's a separate allocated object.
notamy 4y ago
> especially if we ever have CPUs that want to address more than 256 GiB of memory. Given how common of a scenario this is for Linux, I imagine this sort of a change would be more trouble than it's worth...
mort96 4y ago
FYI, we have consumer systems with over a terabyte of RAM.

deleted 4y ago

[deleted]

thayne 4y ago

Is there a userland function equivalent to strscpy in glibc?

trasz 4y ago

Tangentially related: FreeBSD tends to use sbuf(3) (https://www.freebsd.org/cgi/man.cgi?query=sbuf https://www.freebsd.org/cgi/man.cgi?query=sbuf) for anything but the very simplest string manipulation. It’s safer and more convenient than NULL-terminated strings.

benj111 4y ago

I'm confused about the race conditions mentioned. Can't you just copy a byte to a buffer checking for null, counting down the length. At the assembly level you have to copy a byte at a time anyway. I don't see why you'd want to check the entire length of the input string for strlcpy either. It seems less than optimal. I'm guessing the people involved know more than me. So...?