18 ms·
Strlcpy and strlcat added to glibc 2.38
- sillywalk 3y agoIt must be snowing in hell right now. :) It says that they might be added to POSIX. edit apparently Ulrich Drepper (major glibc contributer & former glibc leader is back at Red Hat [0] https://research.redhat.com/blog/project_member/ulrich-drepper/ https://research.redhat.com/blog/project_member/ulrich-drepp...
- wahern 3y agoThey're already in draft 3 of the forthcoming 202x revision. The glibc work resulted in a request to clarify the draft 3 specification: https://www.austingroupbugs.net/view.php?id=1726 https://www.austingroupbugs.net/view.php?id=1726
- chris_wot 3y agoI'll bet he would be a laugh to work with.
- sillywalk 3y agoPerhaps working at Goldman Sachs mellowed him out some.
- dezgeg 3y agoI hope not, as entire reason he went there was to punish the bankers for their role in the 2008 financial crisis ;) (https://www.reddit.com/r/linux/comments/zzhd9/comment/c69camq/ https://www.reddit.com/r/linux/comments/zzhd9/comment/c69cam...)
- deleted 3y ago[deleted]
- classified 3y agoSo what is RedHat being punished for now?
- zshrc 3y agoDepends who you ask.
- rwmj 3y agoI work with him and he's smarter than almost everyone.
- Thoreandan 3y ago25 years on. Congrats OpenBSD
- classified 3y agoI'll never understand how Linux won over BSD.
- noobermin 3y agoThe end of an era, a rather pedantic and overdrawn one.
- panzi 3y agoWhat I'd like to have is snprintf_l(). It's not standard, but it's available in FreeBSD, macOS, and Windows (as _snprintf_l()). Just not in glibc (and probably musl).
- jlokier 3y agosnprintf_l() is not standard, but you can simulate it with uselocale() to save and restore the per-thread locale, which is standard (POSIX.1-2008) and supported by Glibc, FreeBSD, MacOS and other OSes, though Windows requires its own, different way of doing per-thread locale. It seems it was once an explicit design choice in Glibc. Here (https://sourceware.org/bugzilla/show_bug.cgi?id=10891 https://sourceware.org/bugzilla/show_bug.cgi?id=10891) a Glibc ticket refers to documentation "Thread-aware Local Model, A Proposal" of the plans for the _l() functions (https://akkadia.org/drepper/tllocale.ps.gz https://akkadia.org/drepper/tllocale.ps.gz) long ago, which are nowadays implemented in Glibc. It looks like snprintf_l() and other <stdio.h> functions were not part of that plan. That plan or something like it also made its way into POSIX.1-2008, so it seems likely that the committee gave some thought to including strftime_l() and not snprintf_l(). The paper linked above includes a rationale that tiny, potentially performance-critical functions like isalpha() need a fast version that takes a local parameter, thus isalpha_l(), because of the overhead of fetching a thread-local value inside the function. Perhaps the intent is that only those tiny functions, or even macros, whose performance would be greatly affected by the cost of fetching the thread-local locale, need a _l() version. That mostly makes sense for the functions which have _l() versions in Glibc. But with that rationale, I don't see why there is strftime_l() but not snprintf_l().
- panzi 3y agoI want to output a lot of numbers (double) and be sure they are in "C" locale (generating JSON output). So it's actually fprintf_l. I can set/restore LC_NUMERIC in a wrapped function, but having a _l version would be nicer.
- TazeTSchnitzel 3y agoIf you ever find yourself writing strcpy() followed by strcat(), consider: snprintf(buf, sizeof(buf), "%s%s", a, b); This as safe as strlcpy and strlcat, but more efficient, and has been standard for 24 years.
- saagarjha 3y ago1. This returns an int. 2. This is inefficient in a similar way that strlcpy is: it does more work than the size of the buffer.
- jstimpfle 3y ago1. This is only slightly less irrelevant than "this returns a size_t" if that were so. 2. You have the option to provide the length, snprintf(buf, sizeof buf, "%.*s%.*s", len1, str1, len2, str2); If you're bottlenecked by snprintf (hint: you aren't) then snprintf isn't your API anyway. Write some more custom code, probably some memcpy's etc.
- saagarjha 3y agoFor 1: returning an int means that you get some unspecified behavior on overflow. For 2 then you need to call strlen yourself which kind of defeats the point of using snprintf because you can just use memcpy instead.
- jstimpfle 3y agoFor 2: No you don't? There is no reason to fault snprintf() if you didn't already know the length of the string. And when you do know the lengths, no, there absolutely is a reason to use snprintf -- convenience. snprintf(buf, sizeof buf, "%s/%s", dirpath, filename); is much easier to write than an equivalent sequence of manual copies with temporary index variables and pointer arithmetic. Same for snprintf(buf, sizeof buf, "%.*s/%.*s", dirlen, dirpath, filelen, filename); if you really cared about squeezing the last drop from the API. For 1: Then don't overflow. It's not practical to process strings that big to allow a 32-bit overflow (or 16-bit overflow, on a 16-bit system), so it's unlikely anybody here has ever been in that situation anyway. Apart from that I'm not so sure that the API is specified to allow overflow to happen. It's probably an under-specified area of the contract, but I would first check that the result couldn't be -1 for example. Apart from that, what would be the reasonable specification in case of size_t overflow that would result in a controllable situation? Sometimes I think, C being probably better formalized than any other language is also a big reason for it being criticized so much. Language nerds just love to take the specs and try to shred it on theoretic grounds without any consideration of the practical.
- avar 3y agoAs a bit of historical context: "This is horribly inefficient BSD crap. Using these function only leads to other errors. Correct string handling means that you always know how long your strings are and therefore you can you memcpy (instead of strcpy). Beside, those who are using strcat or variants deserved to be punished." - Ulrich Drepper, around 23 years ago: https://sourceware.org/legacy-ml/libc-alpha/2000-08/msg00053.html https://sourceware.org/legacy-ml/libc-alpha/2000-08/msg00053...
- chris_wot 3y agoHe was a jerk, but often he had a reason for his abusiveness. Was the reason in this case valid?
- sanxiyn 3y agoYes.
- pnpnp 3y agoWhy?
- saagarjha 3y agoPeople typically do not realize that it has a return value that is expensive to compute.
- dezgeg 3y agoIt's not any slower in the typical case where destination buffer is large enough to fit the source thing. And if that's not the case then we are most likely in a error case (either caller notices the truncation and decides to abort, or ignores the truncation and things may soon go boom), and not many people care about optimizing error paths. Furthermore, when coders dont't have strlcpy() the alternatives are often even worse than strlcpy(): 1) They use strcpy() and have buffer overflows. 2) They use strncpy() which is slower than strlcpy() in the common (non truncating) case, and in the truncating case leave the string unterminated (thus segfault potential) 3) They use snprintf(dst, len, "%s", src); which is strictly slower than strlcpy()
- wyldfire 3y agoSo was Drepper wrong? Did he just get worn down? Or did it not involve him at all?
- sanxiyn 3y agoI think the fact that it will be added to POSIX was decisive.
- masklinn 3y agoI understand Drepper pretty much considers string.h a lost cause, and I can’t fault him for that. It’s rather that POSIX decided to add the strl functions, so adding them (and verifying their semantics) is a necessity.
- silon42 3y agoA purist would support removing strcpy, etc... but that wouldn't go well, so adding an improvement is acceptable.
- masklinn 3y ago> A purist would support removing strcpy, etc... That is very much Drepper’s position: > Correct string handling means that you always know how long your strings are and therefore you can you memcpy (instead of strcpy). > Beside, those who are using strcat or variants deserved to be punished.
- Gibbon1 3y agoThey should have added a warning for strcpy 20 years ago and shipped the compiler with it default enabled.
- kzrdude 3y agoHe isn't involved anymore, is he?
- fs111 3y agoI think he is doing other things these days https://research.redhat.com/blog/project_member/ulrich-drepper/ https://research.redhat.com/blog/project_member/ulrich-drepp...
- eatbitseveryday 3y agoLinux uses strscpy. See [1] [2] [3]. The issues of concern are to always NUL-terminate, and to effectively know if the result was truncated. Truncation can lead to big issues, especially if the string being composed refers to paths, device names, other resources, etc. For example you may truncate a path from /foo/bar/baz to /foo/bar and inadvertently operate on other files. An API that makes this confusing is dangerous. See the confused deputy problem description [4]. [1] https://mafford.com/text/the-many-ways-to-copy-a-string-in-c/ https://mafford.com/text/the-many-ways-to-copy-a-string-in-c... [2] https://lwn.net/Articles/659214/ https://lwn.net/Articles/659214/ [3] https://docs.kernel.org/core-api/kernel-api.html#c.strscpy https://docs.kernel.org/core-api/kernel-api.html#c.strscpy [4] https://en.wikipedia.org/wiki/Confused_deputy_proble https://en.wikipedia.org/wiki/Confused_deputy_proble
- askiiart 3y agoFixed Wikipedia link: https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Confused_deputy_problem
- dchest 3y agoHere's discussion on why strscpy shouldn't be included in POSIX: https://www.austingroupbugs.net/view.php?id=986 https://www.austingroupbugs.net/view.php?id=986 (scroll to 0002897)
- saagarjha 3y agoIt's not a very good argument. Notably: > strlcpy() fits within the existing set of functions like a glove. strlcpy(a, b, n) behaves identically to snprintf(a, n, "%s", b). The return value always corresponds to the number of non-null bytes that would have been written. If we truly think that this is bad design, should we come up with a new version of snprintf() that also doesn't do this? I don't think so. People typically do not consider snprintf and strlcpy to be a similar family of functions. There's no need to transpose the weird behavior to a new string copying routine.
- deleted 3y ago
- psychphysic 3y agoUrgh more str functions. I look forwards to forgetting about this and/or discovering a new foot gun.
- jstimpfle 3y agoJust ignore them. I suspect they're adding them (to POSIX) for portability of historic software. Here is an issue tracker I've found (Haven't read through it; not interested): https://www.austingroupbugs.net/view.php?id=986 https://www.austingroupbugs.net/view.php?id=986
- 10g1k 3y agoAbbreviations are just bad code. Always. Tradition is no excuse for writing bad code. Names of everything should be sufficiently descriptive that absolutely anyone will know its purpose upon first glance.
- Gibbon1 3y agoI'd be okay with renaming strcpy() to string_you_big_dummy_copy()
- notbeuller 3y agoYou joke, but this is almost reasonable. When refactoring a large codebase riddled with strncpy, strcpy and strcmp, understanding unambiguously what code does shouldn't come down to my middle aged eyes being able to parse better than a compiler. I did a global search and replace with a #define, verified the object code diff'd against the original version, and never looked back.
- Gibbon1 3y agoAs usual I'm joking but somewhat serious. Step one is better replacement functions. Step two actually should be make the bad ones feel sleezy. One thing I think is the problem with making safer string functions is it's hard to do that while staying at the same very low level of abstraction. And I think a lot of code out there sets up string functions to work off incomplete information. (here is a pointer to a string buffer, trust me it's big enough to hold what you'll stuff in it)
- jeroenhd 3y agoThis mere suggestion will annoy many C programmers but I completely agree with you here. With all of the ioctls, the atois and the strncpy_ls I just stopped trying to understand what the names are supposed to mean and use them for their effects only. strlcpy may as well be called qpdixhwnai for all I care, I'll have to look it up when I need it anyway. I've learned C on Windows and the Windows API is friendly in comparison to the C API. When Windows beats you in API design, you should really reconsider some of your policies. Is mem_copy really that much worse than memcpy? Why not memcopy? What do we gain by leaving out that single o? Why is settimeofday not sttmod if munmap is how you write memoryunmap? It feels to me like POSIX is still being optimized for people manually poking holes into punchcards. We've had autocomplete for decades now, a few extra characters won't hurt, I promise.
- zbuf 3y agoInteresting to see the implementation is basically a two-pass process: strlen() to count the source string, followed by a memcpy(). My intuition would be to give some importance to not looping on the source string radically beyond the length of the destination buffer. https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strlcpy.c;h=7a0df3ebb6b279cf151daf14a8cde3b43e705f6c;hb=454a20c8756c9c1d55419153255fc7692b3d2199 https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strl...
- sirwhinesalot 3y agoJust make your own string functions from scratch when using C, you'll thank me later. First, you make two structs: str_buf { capacity, len, data[] } and str_view { len, data* }. Then write your string handling functions to write into a str_buf* and read from str_views. You have the length and capacity available so memcpy is easy and safe to use. The str_buf treats capacity as 1 less than it really is such that it can ensure there's always a stupid 0 at the end for compatibility with APIs that expect 0 terminated strings. There you go, no more security bugs, no more nonsense.
- jacquesm 3y ago> The str_buf treats capacity as 1 less than it really is such that it can ensure there's always a stupid 0 at the end for compatibility with APIs that expect 0 terminated strings. Off-by-one errors are a thing. > Just make your own string functions from scratch when using C, you'll thank me later. No, if you're going to use C and you need a string type use a well supported string library so that you don't end up reinventing the wheel (probably in a buggy way) and benefit from the battle testing that that code has gone through. If we're looking at actual strings (as in text) then I'd use 'libunistring'.
- hyperman1 3y agoI fear all of this is dancing around the nasty core of the problem: generic writing to C style strings can't be done without extra information. You can't write stuff to memory without negotiating how much room is needed and available, and optionally moving the string. Silent truncation will cause bugs. Buffer overflow even more. Fixing this now is hard: Writing to 0-ended strings require manually tracking lengths. Expanding a string without allowing malloc is misery. The only way out I see is basically starting from zero: ISO C should define an API with a (pointer,current length, max length) struct at its core, pointer pointing to a 0-terminated C string. You can read it, but changing it requires using functions that can error out and/or malloc more memory. There are already multiple libs like this, but C has none. If the struct would be ABI, non-C programming languages can pass strings between them.
- samtho 3y agoC had the opportunity to include this but they did not. It is my understanding that they wanted to design everything in C as inherent to the language, rather than magic types, especially a struct. There is an elegance in the notion that a string is just an array of characters. If I’m working with a significant amount of strings in C, I can keep track of lengths, not a huge deal.
- jstimpfle 3y agoExactly this. There are no literals in C that create composite types. There are no composite types inherent to the language. All these types are defined in (system) includes. And zero-terminated strings are not strictly worse than other length-prefixed string forms. They save some space -- sure, less relevant today -- as well as provide an in-band termination signal -- which is hacky, again sure, but it is convenient when looking at a hex dump for example.
- nicoburns 3y ago> There are no literals in C that create composite types Float/double literals do
- t43562 3y agoNULL terminated strings are a fact of life in many cases and C just needs lots of string functions to cater for different use cases. e.g. usually for me truncation is worse than crashing (corrupted data basically). When I read arguments they are full of people thinking that their one size fits all and that somehow having too many variations would be bad. This seems illogical to me since I've had to write my own string copying routines enough times because the one that fitted my need wasn't commonly available. Purism with C is just stu.... well anyhow. A "good" fat pointer library for C would help a lot - something that could pop a NUL onto the end when you needed to put the string into an OS function but I would also have to groan at the idea that NUL termination should be outlawed in some way. At the C level you want options not limitations.
- orwin 3y agoYeah agree. Roll your own linked lists (or rather: avoid them but if you have to use them, roll your own, it's like 30 min to 2 hours at most), your own btrees, your own ring buffer. I guess the gnu hashmap is fine 90% of the time, but if your use case is weird, you might want to run your own too. And BTW the glibc isn't really bad, just that in C, generic solutions will miss your specifics edge cases. And this is true for strings too. If you use only static strings in your whole project, the standard is enough, but I won't write an IRC server or a posix Shell without my own strings (not anymore :p).
- GoblinSlayer 3y agoodbc interface can accept both null terminated and length bounded strings by using NTS sentinel value for null terminated string length.
- alxlaz 3y agoThere are two things I have learned from this thread: 1. glibc has been handling things with a lot more civility post-Drepper, and while it's had as hard a time as any OS infrastructure project, that's helped a lot when it came to managing more polarizing issues. 2. The greatest enemy to wider Rust adoption is not that C-educated programmers are reluctant to learn and apply more solid principles. The greatest enemy to wider Rust adoption is that, despite having ample material about the past mistakes of their forerunners, the Rust community is only learning from those that relate to language design and ignoring all the other ones.
- PoignardAzur 3y ago> The greatest enemy to wider Rust adoption is that, despite having ample material about the past mistakes of their forerunners, the Rust community is only learning from those that relate to language design and ignoring all the other ones. What? How is that your takeaway from the thread? There's only a few mentions of Rust and none of them are abrasive.
- alxlaz 3y agoThere are quite a few abrasive replies upthread from some Rustaceans. They don't mention Rust by name, just like not every one of Drepper's mails contained the world "glibc", but they're in the same vein. This is particularly important at a point in a language's lifetime when community support is not just the best, but usually the only kind of support you can get. I like Rust and I'm very productive with it, but if anyone thinks I'm going to ask junior devs on my team to put up with the kind of stuff I see upstream, they're wrong. Just because we developed a thick skin for it on FOSS mailing lists back in the nineties doesn't mean everyone needs to.
- estebank 3y agoIf you see poor behavior in the project, call it out. I would personally be interested in being made aware of it to help eliminate it, but the moderation team[1] exists specifically to deal with this. 1: https://www.rust-lang.org/governance/teams/moderation https://www.rust-lang.org/governance/teams/moderation
- deleted 3y ago[deleted]
- vogon_laureate 3y agoKudos and congrats to Todd Miller and the OpenBSD folks! [1] [1]: https://man.openbsd.org/strlcpy.3