4 ms·

> But this isn't the common scenario that strl* family of functions were designed for I will give you that. They definitely weren't designed for this scenario.

by byuu 13y ago

> But this isn't the common scenario that strl* family of functions were designed for I will give you that. They definitely weren't designed for this scenario. But in practice, people tend to ignore the return value (which is using the functions incorrectly), and if you were used to strncpy (which has its own performance problems), you might wrongly assume that you could use these to pull substrings out as well. The regrettable part is that it's easy to combine overflow detection with substring copy functionality at the same time. > Sorry but that is just wrong. Sorry, but it is not wrong. See the actual implementation source: http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/strlcat.c?rev=1.13 http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/st... "Returns strlen(src) + MIN(siz, strlen(initial dst))." Even the official man page says the return value is confusing: "While this may seem somewhat confusing, it was done to make truncation detection simple." I mean no disrespect, but you're proving my point. Even most proponents of these functions do not understand all of the nuances in how they work. That's a real problem, as if you don't understand strlcat's return value, you can't use it safely. The functionality is too specialized to be general purpose functions. > It's not "requested length", it's "this many bytes fit in my buffer" Yes. Sorry if my wording was poor, I'm not really good at explaining things. > These functions are not supposed to validate your input any more than are they supposed to check the validity of your pointers, intent, etc. Exactly. They're useful functions for certain purposes. My only point is that they are by no means perfect. There are many use cases where they are not appropriate. Unfortunately people tend to either praise this family as the solution to C memory safety problems, or they dismiss them entirely. Absolutely use them when they're appropriate, but don't simply use them everywhere and think you're making your program safer. There's performance implications, out-of-bounds memory violations, security issues raised by truncating values without catching it, etc. These are all the programmer's fault for misusing the functions, of course, but safer and/or more versatile functions can be made.

clarry 13y ago
> But in practice, people tend to ignore the return value (which is using the functions incorrectly) That might be true. It would be very interesting to see some statistics. However, with these functions, it is very easy to check whether the truncation detection is being done or not. Grep is mostly enough, but it can even be automated for most cases. Also, that is hardly an argument against these functions: any string utility function will inevitably have to deal with errors somehow (or worse, ignore them and continue!). > if you were used to strncpy (which has its own performance problems), you might wrongly assume that you could use these to pull substrings out as well. Yes, and sadly I've seen something like that (and worse) in the wild. These are easy to spot though. > Sorry, but it is not wrong. > > See the actual implementation source: http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/st.. http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/st.... > > "Returns strlen(src) + MIN(siz, strlen(initial dst))." Interesting. You're right about the code and its return value. However, the current manual documents it a little bit differently; the weird return value for the case when the initial destination string is not a string is never mentioned (but the fact that NUL-termination won't occur in this case is there). So in practice you can forget the MIN() and say it is strlen(dst) + strlen(src). From my perspective it makes all the sense, because, going back to what I've argued before, a string handling function should at least be allowed to assume you're giving it strings to work with; it's not there to validate your input or check pointers, etc. > Even the official man page says the return value is confusing: "While this may seem somewhat confusing, it was done to make truncation detection simple." You quoted a line that is not describing the corner-case return value mentioned above. It is describing the fact that the return value is (assuming actual strings are processed!) the total length of the string that would result from the operation if the destination buffer was large enough to hold it without truncation. There is nothing confusing about that. Perhaps the author initially thought it would be confusing for people who expect it to resemble functions such as read() that return the number of characters actually given to you. But there is nothing to be confused for, and the official man page has been simplified to reflect that: commit: http://marc.info/?l=openbsd-cvs&m=133338810915771&w=2 current version: http://mdoc.su/o/strlcat > I mean no disrespect, but you're proving my point. Even most proponents of these functions do not understand all of the nuances in how they work. That's a real problem, as if you don't understand strlcat's return value, you can't use it safely. The correct way to detect truncation with strlcpy and strlcat has always been exactly the same, and documented: if (strlcat(dst, src, dstsize) >= dstsize) { /* truncated */ } The fact that I wasn't aware of a corner-case scenario where dst does not contain a string to begin with has in no way impeded my ability to type the above line correctly. The branch will still be taken if the string would be too large for the buffer, but your code is broken and strlcat cannot fix it for you. > The functionality is too specialized to be general purpose functions Yes, they perform a very specific function; they are not Perl. Yet these are "general purpose" enough that the less secure variants (and not only these as of C11) were introduced in the C standard, and are used everywhere. These functions are convenient, available everywhere, and recognized by any C programmer, and that is why they are used. The OpenBSD functions came to be because of the widespread (mis)use of the C functions, to make it easier to do it right. > Exactly. They're useful functions for certain purposes. My only point is that they are by no means perfect. There are many use cases where they are not appropriate. Unfortunately people tend to either praise this family as the solution to C memory safety problems, or they dismiss them entirely. Good to know we are in agreement. :-) Perhaps I interpreted you too strongly when you said one may as well use another implementation; to me that sounded like a suggestion to dismiss these entirely. > These are all the programmer's fault for misusing the functions, of course, but safer and/or more versatile functions can be made. I would be interested in a concrete example. More versatile? Yes, that is easy. Safer? Well I'm not so sure about that. Perhaps by trading some correctness. Perhaps in some non-critical user-land application where unbounded allocations are OK and aborting on memory exhaustion is "safe". Simpler and safer? Almost definitely not, but please show me I'm wrong :-)
- byuu 13y ago
  > The fact that I wasn't aware of a corner-case scenario where dst does not contain a string to begin with has in no way impeded my ability to type the above line correctly. True. And again I was hesitant to say that, so I hope you didn't take any offense. These functions surprised me as well at first (I was personally bitten by that strlcpy() running full strlen(source) issue in my own XML parser.) > I would be interested in a concrete example. Well I don't really have a super-secure version. Honestly 99% of string stuff I do is using a C++ library that keeps track of the allocation size (and also has small-string optimization.) What I came up with in place of strlcpy/cat, which worked better for my own purposes, is below, which I call strmcpy/cat: //return = strlen(target) unsigned strmcpy(char *target, const char *source, unsigned length) { const char *origin = target; if(length) { while(*source && --length) *target++ = *source++; *target = 0; } return target - origin; } //return = strlen(target) unsigned strmcat(char *target, const char *source, unsigned length) { const char *origin = target; while(*target && length) target++, length--; return (target - origin) + strmcpy(target, source, length); } Truncation is detected by seeing if source[strmcpy/cat(...)] != 0, and very easy to wrap with another function. And in general, I find strcat functions kind of unpleasant (eg Schlemel the painter), so for that a strpcpy() that writes to a pointer will let you chain those, eg: void strpcpy(char *&target, const char *source, unsigned &length) { unsigned offset = strmcpy(target, source, length); target += offset, length -= offset; } Now I am sure you can point out a myriad of serious problems with these functions, just as I have for strlcpy/cat. Hell they're probably much worse than strl, but they work for me at least. If nothing else, they are at least easier to read =) Full info on these functions is at the bottom of this dead article: http://web.archive.org/web/20130121234920/http://byuu.org/articles/programming/strcpycat http://web.archive.org/web/20130121234920/http://byuu.org/ar...

by byuu 13y ago

clarry 13y ago

> But in practice, people tend to ignore the return value (which is using the functions incorrectly) That might be true. It would be very interesting to see some statistics. However, with these functions, it is very easy to check whether the truncation detection is being done or not. Grep is mostly enough, but it can even be automated for most cases. Also, that is hardly an argument against these functions: any string utility function will inevitably have to deal with errors somehow (or worse, ignore them and continue!). > if you were used to strncpy (which has its own performance problems), you might wrongly assume that you could use these to pull substrings out as well. Yes, and sadly I've seen something like that (and worse) in the wild. These are easy to spot though. > Sorry, but it is not wrong. > > See the actual implementation source: http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/st.. http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libc/string/st.... > > "Returns strlen(src) + MIN(siz, strlen(initial dst))." Interesting. You're right about the code and its return value. However, the current manual documents it a little bit differently; the weird return value for the case when the initial destination string is not a string is never mentioned (but the fact that NUL-termination won't occur in this case is there). So in practice you can forget the MIN() and say it is strlen(dst) + strlen(src). From my perspective it makes all the sense, because, going back to what I've argued before, a string handling function should at least be allowed to assume you're giving it strings to work with; it's not there to validate your input or check pointers, etc. > Even the official man page says the return value is confusing: "While this may seem somewhat confusing, it was done to make truncation detection simple." You quoted a line that is not describing the corner-case return value mentioned above. It is describing the fact that the return value is (assuming actual strings are processed!) the total length of the string that would result from the operation if the destination buffer was large enough to hold it without truncation. There is nothing confusing about that. Perhaps the author initially thought it would be confusing for people who expect it to resemble functions such as read() that return the number of characters actually given to you. But there is nothing to be confused for, and the official man page has been simplified to reflect that: commit: http://marc.info/?l=openbsd-cvs&m=133338810915771&w=2 current version: http://mdoc.su/o/strlcat > I mean no disrespect, but you're proving my point. Even most proponents of these functions do not understand all of the nuances in how they work. That's a real problem, as if you don't understand strlcat's return value, you can't use it safely. The correct way to detect truncation with strlcpy and strlcat has always been exactly the same, and documented: if (strlcat(dst, src, dstsize) >= dstsize) { /* truncated */ } The fact that I wasn't aware of a corner-case scenario where dst does not contain a string to begin with has in no way impeded my ability to type the above line correctly. The branch will still be taken if the string would be too large for the buffer, but your code is broken and strlcat cannot fix it for you. > The functionality is too specialized to be general purpose functions Yes, they perform a very specific function; they are not Perl. Yet these are "general purpose" enough that the less secure variants (and not only these as of C11) were introduced in the C standard, and are used everywhere. These functions are convenient, available everywhere, and recognized by any C programmer, and that is why they are used. The OpenBSD functions came to be because of the widespread (mis)use of the C functions, to make it easier to do it right. > Exactly. They're useful functions for certain purposes. My only point is that they are by no means perfect. There are many use cases where they are not appropriate. Unfortunately people tend to either praise this family as the solution to C memory safety problems, or they dismiss them entirely. Good to know we are in agreement. :-) Perhaps I interpreted you too strongly when you said one may as well use another implementation; to me that sounded like a suggestion to dismiss these entirely. > These are all the programmer's fault for misusing the functions, of course, but safer and/or more versatile functions can be made. I would be interested in a concrete example. More versatile? Yes, that is easy. Safer? Well I'm not so sure about that. Perhaps by trading some correctness. Perhaps in some non-critical user-land application where unbounded allocations are OK and aborting on memory exhaustion is "safe". Simpler and safer? Almost definitely not, but please show me I'm wrong :-)

byuu 13y ago
> The fact that I wasn't aware of a corner-case scenario where dst does not contain a string to begin with has in no way impeded my ability to type the above line correctly. True. And again I was hesitant to say that, so I hope you didn't take any offense. These functions surprised me as well at first (I was personally bitten by that strlcpy() running full strlen(source) issue in my own XML parser.) > I would be interested in a concrete example. Well I don't really have a super-secure version. Honestly 99% of string stuff I do is using a C++ library that keeps track of the allocation size (and also has small-string optimization.) What I came up with in place of strlcpy/cat, which worked better for my own purposes, is below, which I call strmcpy/cat: //return = strlen(target) unsigned strmcpy(char *target, const char *source, unsigned length) { const char *origin = target; if(length) { while(*source && --length) *target++ = *source++; *target = 0; } return target - origin; } //return = strlen(target) unsigned strmcat(char *target, const char *source, unsigned length) { const char *origin = target; while(*target && length) target++, length--; return (target - origin) + strmcpy(target, source, length); } Truncation is detected by seeing if source[strmcpy/cat(...)] != 0, and very easy to wrap with another function. And in general, I find strcat functions kind of unpleasant (eg Schlemel the painter), so for that a strpcpy() that writes to a pointer will let you chain those, eg: void strpcpy(char *&target, const char *source, unsigned &length) { unsigned offset = strmcpy(target, source, length); target += offset, length -= offset; } Now I am sure you can point out a myriad of serious problems with these functions, just as I have for strlcpy/cat. Hell they're probably much worse than strl, but they work for me at least. If nothing else, they are at least easier to read =) Full info on these functions is at the bottom of this dead article: http://web.archive.org/web/20130121234920/http://byuu.org/articles/programming/strcpycat http://web.archive.org/web/20130121234920/http://byuu.org/ar...