7 ms·

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's need

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines yourself.

nly 8y ago
Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.
- agumonkey 8y ago
  any attempt has been made to build a nicer foundational C/OS library since ?
  - nly 8y ago
    BSD has some minor improvements to libc. There's also glib. Generally though, C is just a terrible language to do anything other than write extremely low level routines in. You should probably just never use strtok() and go straight to something like Ragel or re2c for building high performance tokenizers... then call those from a higher level language.
    - yakovdk 8y ago
      Hey! I take exception to that. I've been writing shellcode all week. C is uncomfortably high level for writing low level routines!
  - jstimpfle 8y ago
    There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.
    - dkrikun 8y ago
      There are numerous reasons to write in C, apart from caring so much. Using strtok() is not that bad after all because its a simple function after all.
    - Matthias247 8y ago
      You should. However there are lots of organizations around that use C because that’s what’s used in their domain (E.g. embedded) AND use the standard library, because they don’t know how to do better. The results are often worse than if they would have used a higher level language right from the start.
- nwmcsween 8y ago
  libc aka ISO, POSIX, GNU wasn't designed it was more or less evolved from what was in use.
yason 8y ago
If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In contrast, an example of truly sickening part of stdlib is converting strings to number. The atoi()/atol() family doesn't check for errors at all so you want to use strtol(). But the way error checking works in strtol() is so complex that the man page has a specific example of how to do it correctly. All sane programmers quickly write a clean wrapper around strtol() to encode the complexity once. Now, strtok() is nothing like that. In its simplicity, strtok() is quite versatile. A few strtok() calls can easily parse lines like: keyword=value1, value2, value3 that you might find in configuration files. And I mean truly in just a few lines which you might expect in Python but with C string handling? No.
- jstimpfle 8y ago
  Here is the musl implementation. > https://github.com/esmil/musl/blob/master/src/string/strtok.c https://github.com/esmil/musl/blob/master/src/string/strtok.... It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory. So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.
  - yason 8y ago
    So, here's that implementation: static char *p; if (!s && !(s = p)) return NULL; s += strspn(s, sep); if (!*s) return p = 0; p = s + strcspn(s, sep); if (*p) *p++ = 0; else p = 0; return s; Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library. Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy. As I pointed in the other reply, strtok() does not break on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form. If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.
fanf2 8y ago
Use strcspn() instead
- jstimpfle 8y ago
  Token tok; start_token(&tok); for (;;) { int c = look_next_char(); if (('A' <= c && c <= 'Z') || ('a' <= c && c <= 'z')) { /* or whatever test */ consume_char(); add_to_token(tok, c); } else { break; } } end_token(tok); Done. There's no point in going through a weird API.
  - tptacek 8y ago
    strcspn is ANSI C90.
  - torstenvl 8y ago
    You've not only hard-coded your tokenization rules inside your logic, but you've managed to make it break on anything non-ASCII. I suppose you consider isalpha() to have a weird API?
    - jstimpfle 8y ago
      I look at it the other way: I've hard coded the reading and writing routines inside the tokenization logic. Being able to do that is exactly the point why it's so much simpler to avoid a silly API such as strcspn (or, god forbid, strtok). > non-ASCII yeah i know... Do you prefer strcspn("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVXYZ")? Do you think it's faster? If you're pedantic, you could lex (0x41 <= c && c <= 0x5A). That way at least you consistently read ASCII, even on non-ASCII implementations. But I don't care and it's less readable. > I suppose you consider isalpha() to have a weird API? Yes. I do not even understand what it does. >> isalpha() checks for an alphabetic character; in the standard "C" locale, it is equivalent to (isupper(c) || islower(c)). In some locales, there may be additional characters for which isalpha() is true-letters which are neither upper case nor lower case. Well in any case I'm sure that's not what I wanted... By the way locale is super hard to use as well. Locale is a process global property. I'm not aware of any way to pass explicit locales to library functions.
tails4e 8y ago
strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.
- jstimpfle 8y ago
  I'd say just use memcpy.
  - tptacek 8y ago
    How does that help? You have the same bookkeeping problem with memcpy as you do with strncat.
    - jstimpfle 8y ago
      str[ln]cat is like memcpy, but it does unnecessary work (search for the end of the string which the caller should long have figured out). It's like memcpy but less general and more confusing to use due to the implicit string length of the target operand. It's a completely unnecessary API, likely a remainder from the times when people used C for scripting. It's also a complete idiocy to possibly not copy everything. I've never wanted that. It's begging for bugs. If there's not enough room to copy all the source bytes, you either know that in advance or you want to crash hard. Even memcpy itself is hardly necessary as a library function. It's a simple loop. But at least it's what you would write anyway, and in some cases compilers are able to optimize it (with questionable benefits).
      - repiret 8y ago
        If you use memcpy and your destination buffer is too small you don’t crash hard with any certainty, your program might even appear to work. Moreover, most of the time I’m either certain the output buffer is large enough (maybe because I just allocated it) or I really do want the output buffer to have a truncated string. Strncpy’s two dumb behaviors that I almost never want is to pad the destination buffer with zeros if the source string is smaller and to leave the destination I terminated if the source is longer.
fredmorcos 8y ago
strtok is there for things like and similar to /etc/hosts and /etc/fstab
- ygra 8y ago
  Or have those file formats been designed around having to parse them in a C program?
  - fredmorcos 8y ago
    Could be. I don't know which one came first.
simias 8y ago
I use strtok_r from time to time, it does the job if you have a mutable input. Of course having to write zeroes is a bit cumbersome but it's one of the drawbacks of C-style strings. The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. Splitting, concatenating, regex-ing... All of that is a huge pain in C. If you need to write a high-performance parser then it might be worth it but if you're just parsing a fancy command line format and performances don't matter it's just incredibly frustrating and error prone. Rust fares better here because its str type is not NUL-terminated but actually keeps track of the length separately which makes it significantly more flexible and versatile. Of course you could do that in C but you'll be incompatible with any code dealing with native C-strings. And of course you make one mistake and you have a buffer overflow vulnerability... So yeah, if you program in C please use strtok_r if applicable, otherwise considering offloading the parsing to an other part of your application written in a language better suited for that and hand over binary data to the C library. If everything else fails then consider handwriting your parser and may god be with you. Oh and if your grammar is complex enough to warrant it, there's always lex/yacc.
- paavoova 8y ago
  > but you'll be incompatible with any code dealing with native C-strings Not entirely, see https://github.com/antirez/sds https://github.com/antirez/sds Basically, you have a header storing length, etc, but still null terminate, so library functions like strlen are none the wiser.
  - the_duke 8y ago
    Nim (the programming language) implements strings this way, AFAIK, with the same reason: to be easily compatible with the C ecosystem.
  - saagarjha 8y ago
    You lose the ability to pass create zero-cost slices, though. C++11 actually implements strings like this, but iterators allow you to pass in parts of the string as necessary.
- 8y ago
hermitdev 8y ago
There are valuable use cases where this matters. For example, parsing FIX messages in finance, this allows you to parse the tag/value pairs with no memory allocations, which matters in low latency HFT applications.
- quotemstr 8y ago
  If you want speed, just build a lexer with ragel [1]. It's hard to go faster than a DFA. [1] http://www.colm.net/open-source/ragel/ http://www.colm.net/open-source/ragel/
avar 8y ago
> it writes zeros to the input array [which] means it's unfit for most use cases[...] Other issues with strtok() aside, this seems like a silly reason to discount a standard library function. If you don't want your input munged you can strdup() it. It's rare to find a C program that's so specialized that the performance hit of a strdup() would be unacceptable in a case where strtok() could otherwise have been used.
- jstimpfle 8y ago
  It would be wasteful to duplicate the input string which includes so much garbage. I would rather just go through the input string and append token by token to the output string, terminated with a single NUL.
Gibbon1 8y ago
> And it writes zeros to the input array. If you pass strtok_r a const string it can and will bus fault in some systems. This happens when it tries to write a /0 to the input string. Being an old crusty firmware guy I'm not sailing on good cargo cult ship HMS Immutability, but generating side effects in your input data stream is terrible. There is no way to back up/undo when using strtok_r. When your parsing involves a decision tree that kinda sucks.
Hypergraphe 8y ago
Agreed, I basically avoid using strtok because of that. Why would you write zeros in my input...

nly 8y ago

Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.

agumonkey 8y ago
any attempt has been made to build a nicer foundational C/OS library since ?
- nly 8y ago
  BSD has some minor improvements to libc. There's also glib. Generally though, C is just a terrible language to do anything other than write extremely low level routines in. You should probably just never use strtok() and go straight to something like Ragel or re2c for building high performance tokenizers... then call those from a higher level language.
  - yakovdk 8y ago
    Hey! I take exception to that. I've been writing shellcode all week. C is uncomfortably high level for writing low level routines!
- jstimpfle 8y ago
  There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.
  - dkrikun 8y ago
    There are numerous reasons to write in C, apart from caring so much. Using strtok() is not that bad after all because its a simple function after all.
  - Matthias247 8y ago
    You should. However there are lots of organizations around that use C because that’s what’s used in their domain (E.g. embedded) AND use the standard library, because they don’t know how to do better. The results are often worse than if they would have used a higher level language right from the start.
nwmcsween 8y ago
libc aka ISO, POSIX, GNU wasn't designed it was more or less evolved from what was in use.

yason 8y ago

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In contrast, an example of truly sickening part of stdlib is converting strings to number. The atoi()/atol() family doesn't check for errors at all so you want to use strtol(). But the way error checking works in strtol() is so complex that the man page has a specific example of how to do it correctly. All sane programmers quickly write a clean wrapper around strtol() to encode the complexity once. Now, strtok() is nothing like that. In its simplicity, strtok() is quite versatile. A few strtok() calls can easily parse lines like: keyword=value1, value2, value3 that you might find in configuration files. And I mean truly in just a few lines which you might expect in Python but with C string handling? No.

jstimpfle 8y ago
Here is the musl implementation. > https://github.com/esmil/musl/blob/master/src/string/strtok.c https://github.com/esmil/musl/blob/master/src/string/strtok.... It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory. So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.
- yason 8y ago
  So, here's that implementation: static char *p; if (!s && !(s = p)) return NULL; s += strspn(s, sep); if (!*s) return p = 0; p = s + strcspn(s, sep); if (*p) *p++ = 0; else p = 0; return s; Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library. Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy. As I pointed in the other reply, strtok() does not break on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form. If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.

fanf2 8y ago

Use strcspn() instead

jstimpfle 8y ago
Token tok; start_token(&tok); for (;;) { int c = look_next_char(); if (('A' <= c && c <= 'Z') || ('a' <= c && c <= 'z')) { /* or whatever test */ consume_char(); add_to_token(tok, c); } else { break; } } end_token(tok); Done. There's no point in going through a weird API.
- tptacek 8y ago
  strcspn is ANSI C90.
- torstenvl 8y ago
  You've not only hard-coded your tokenization rules inside your logic, but you've managed to make it break on anything non-ASCII. I suppose you consider isalpha() to have a weird API?
  - jstimpfle 8y ago
    I look at it the other way: I've hard coded the reading and writing routines inside the tokenization logic. Being able to do that is exactly the point why it's so much simpler to avoid a silly API such as strcspn (or, god forbid, strtok). > non-ASCII yeah i know... Do you prefer strcspn("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVXYZ")? Do you think it's faster? If you're pedantic, you could lex (0x41 <= c && c <= 0x5A). That way at least you consistently read ASCII, even on non-ASCII implementations. But I don't care and it's less readable. > I suppose you consider isalpha() to have a weird API? Yes. I do not even understand what it does. >> isalpha() checks for an alphabetic character; in the standard "C" locale, it is equivalent to (isupper(c) || islower(c)). In some locales, there may be additional characters for which isalpha() is true-letters which are neither upper case nor lower case. Well in any case I'm sure that's not what I wanted... By the way locale is super hard to use as well. Locale is a process global property. I'm not aware of any way to pass explicit locales to library functions.

tails4e 8y ago

strncat is another good example. It's a buffer overflow waiting to happen, and worse it's the 'safe' version of strcat. The trick is the parameter you pass is not the length of the destination buffer, but the remaining length of the destination buffer. Most people really want the behaviour of strlcat.

jstimpfle 8y ago
I'd say just use memcpy.
- tptacek 8y ago
  How does that help? You have the same bookkeeping problem with memcpy as you do with strncat.
  - jstimpfle 8y ago
    str[ln]cat is like memcpy, but it does unnecessary work (search for the end of the string which the caller should long have figured out). It's like memcpy but less general and more confusing to use due to the implicit string length of the target operand. It's a completely unnecessary API, likely a remainder from the times when people used C for scripting. It's also a complete idiocy to possibly not copy everything. I've never wanted that. It's begging for bugs. If there's not enough room to copy all the source bytes, you either know that in advance or you want to crash hard. Even memcpy itself is hardly necessary as a library function. It's a simple loop. But at least it's what you would write anyway, and in some cases compilers are able to optimize it (with questionable benefits).
    - repiret 8y ago
      If you use memcpy and your destination buffer is too small you don’t crash hard with any certainty, your program might even appear to work. Moreover, most of the time I’m either certain the output buffer is large enough (maybe because I just allocated it) or I really do want the output buffer to have a truncated string. Strncpy’s two dumb behaviors that I almost never want is to pad the destination buffer with zeros if the source string is smaller and to leave the destination I terminated if the source is longer.

fredmorcos 8y ago

strtok is there for things like and similar to /etc/hosts and /etc/fstab

ygra 8y ago
Or have those file formats been designed around having to parse them in a C program?
- fredmorcos 8y ago
  Could be. I don't know which one came first.

simias 8y ago

I use strtok_r from time to time, it does the job if you have a mutable input. Of course having to write zeroes is a bit cumbersome but it's one of the drawbacks of C-style strings. The plain truth is that string handling in C is a huge pain in the ass no matter how you look at it. Splitting, concatenating, regex-ing... All of that is a huge pain in C. If you need to write a high-performance parser then it might be worth it but if you're just parsing a fancy command line format and performances don't matter it's just incredibly frustrating and error prone. Rust fares better here because its str type is not NUL-terminated but actually keeps track of the length separately which makes it significantly more flexible and versatile. Of course you could do that in C but you'll be incompatible with any code dealing with native C-strings. And of course you make one mistake and you have a buffer overflow vulnerability... So yeah, if you program in C please use strtok_r if applicable, otherwise considering offloading the parsing to an other part of your application written in a language better suited for that and hand over binary data to the C library. If everything else fails then consider handwriting your parser and may god be with you. Oh and if your grammar is complex enough to warrant it, there's always lex/yacc.

paavoova 8y ago
> but you'll be incompatible with any code dealing with native C-strings Not entirely, see https://github.com/antirez/sds https://github.com/antirez/sds Basically, you have a header storing length, etc, but still null terminate, so library functions like strlen are none the wiser.
- the_duke 8y ago
  Nim (the programming language) implements strings this way, AFAIK, with the same reason: to be easily compatible with the C ecosystem.
- saagarjha 8y ago
  You lose the ability to pass create zero-cost slices, though. C++11 actually implements strings like this, but iterators allow you to pass in parts of the string as necessary.
8y ago

hermitdev 8y ago

There are valuable use cases where this matters. For example, parsing FIX messages in finance, this allows you to parse the tag/value pairs with no memory allocations, which matters in low latency HFT applications.

quotemstr 8y ago
If you want speed, just build a lexer with ragel [1]. It's hard to go faster than a DFA. [1] http://www.colm.net/open-source/ragel/ http://www.colm.net/open-source/ragel/

avar 8y ago

> it writes zeros to the input array [which] means it's unfit for most use cases[...] Other issues with strtok() aside, this seems like a silly reason to discount a standard library function. If you don't want your input munged you can strdup() it. It's rare to find a C program that's so specialized that the performance hit of a strdup() would be unacceptable in a case where strtok() could otherwise have been used.

jstimpfle 8y ago
It would be wasteful to duplicate the input string which includes so much garbage. I would rather just go through the input string and append token by token to the output string, terminated with a single NUL.

Gibbon1 8y ago

> And it writes zeros to the input array. If you pass strtok_r a const string it can and will bus fault in some systems. This happens when it tries to write a /0 to the input string. Being an old crusty firmware guy I'm not sailing on good cargo cult ship HMS Immutability, but generating side effects in your input data stream is terrible. There is no way to back up/undo when using strtok_r. When your parsing involves a decision tree that kinda sucks.

Hypergraphe 8y ago

Agreed, I basically avoid using strtok because of that. Why would you write zeros in my input...