3 ms·

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for

by yason 8y ago

If you program in C please just write those four obvious lines yourself. Those are not necessarily obvious lines, there are several pitfalls to avoid, and for that reason strtok() is much longer than four lines. When it comes to the standard library functions strtok() has well-defined behaviour that is easy to reason with and near-magically approaches the string-splitting convenience close to scripting languages. In contrast, an example of truly sickening part of stdlib is converting strings to number. The atoi()/atol() family doesn't check for errors at all so you want to use strtol(). But the way error checking works in strtol() is so complex that the man page has a specific example of how to do it correctly. All sane programmers quickly write a clean wrapper around strtol() to encode the complexity once. Now, strtok() is nothing like that. In its simplicity, strtok() is quite versatile. A few strtok() calls can easily parse lines like: keyword=value1, value2, value3 that you might find in configuration files. And I mean truly in just a few lines which you might expect in Python but with C string handling? No.

jstimpfle 8y ago
Here is the musl implementation. > https://github.com/esmil/musl/blob/master/src/string/strtok.c https://github.com/esmil/musl/blob/master/src/string/strtok.... It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory. So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.
- yason 8y ago
  So, here's that implementation: static char *p; if (!s && !(s = p)) return NULL; s += strspn(s, sep); if (!*s) return p = 0; p = s + strcspn(s, sep); if (*p) *p++ = 0; else p = 0; return s; Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library. Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy. As I pointed in the other reply, strtok() does not break on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form. If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.
  - jstimpfle 8y ago
    As someone else pointed out what you really want is the implementation of strspn/strcspn which is where the loop is. You don't "carry" that code along. You just write while (i < len && !is_token_begin(buf[i])) i++; if (i == len) error("End of input\n"); start_token(tok); while (i < len && is_token_char(buf[i])) { i++; add_to_token(tok, buf[i]); } end_token(tok); or something along those lines. Whatever you need. It's not rocket science. Putting highly fluctuating and project-specific code like this in a library would only have disadvantages. Not everything should be in a library. In fact, most things should not be.
nly 8y ago
Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. This is a non-trivial task requiring a real parser operating over a well-specified grammar. A tokenization pipeline just won't cut it. I worked on a project a few years ago that read its custom-format config file in line by line, chopped everything off each line following the first '#' character (to support comments), and then trimmed the whitespace. This sounds like a reasonable and elegant approach until you consider that now none of your user controlled fields (via a GUI in our case) can contain the '#' character. This effected customers, but nobody ever fixed it. With the tools and languages out there now, there's just no excuse for this crap.
- yason 8y ago
  Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. If you have needs that require a general parsing library then why are you criticizing strtok()? It doesn't parse XML either, not C source code, nor any unspecified configuration file formats.
  - nly 8y ago
    My point is that strtok has almost no reasonable uses in real programs.
Someone 8y ago
> A few strtok() calls can easily parse lines like: keyword=value1, value2, value3 The challenge with parsing isn’t parsing correct inputs; it’s generating useful error messages and recovering on incorrect inputs such as keyword=,,value1, value2, value3,,,, or even =keyword=,,value1, value2, value3,,,, strtok isn’t the best tool for doing that. (Yes, those could be valid inputs, but if they are, chances are they should be parsed differently)
- yason 8y ago
  That's what I meant: strtok() is well-defined in what it does. The man page is really short. I don't understand people who complain it doesn't do something it's not supposed to do. Yet it's very useful in what it does. Your mileage may vary but it's also a common issue having to filter out empty items out of something like ",foo,bar ,,,,baz,,xyzzy," where you only really want those four words. You will especially encounter this in parsing user input which might have extra whitespace, or badly formed lists of items. Use strtok() to split C strings into tokens delimited by the given set of delimiters. If you want to catch each comma but skip over any whitespace and split at the first '=' only, then use something else. Let's say if I'm going to need a configuration file for my program I'd most certainly start with something I can parse with strtok(). I would really need very specific needs to warrant a more complex format that would require a more sophisticated parser in which case using one would be a no-brainer.

by yason 8y ago

jstimpfle 8y ago

Here is the musl implementation. > https://github.com/esmil/musl/blob/master/src/string/strtok.c https://github.com/esmil/musl/blob/master/src/string/strtok.... It's a bit longer than 4 lines because strtok does things you should not want. If you insist on parsing that configuration line with strtok, go ahead and write that brittle code. It breaks as soon as you want empty strings (try "keyword=value1, , value3" with strtok) or escape sequences or other transformations, or as soon as you want to do something as basic as parsing from a stream instead of a string that is completely in memory. So to clarify, of course you are never done with parsing in 4 lines. But even if it wasn't as braindead to overwrite the input string, the functionality strtok provides would not be worth more than 4 lines.

yason 8y ago
So, here's that implementation: static char *p; if (!s && !(s = p)) return NULL; s += strspn(s, sep); if (!*s) return p = 0; p = s + strcspn(s, sep); if (*p) *p++ = 0; else p = 0; return s; Instead of carrying that code, or something similar, with my source code or my own utility library I'd much rather have the already debugged version from the standard library. Overwriting the input in C is more efficient than maintaining more internal state and returning a pointer and the length of each token which you would need to strncpy() to get the token into a C string. strtok() does not want to do the initial strdup() for you because only you will know whether your input can already be mutated or whether you need to use a copy. As I pointed in the other reply, strtok() does not break on strings like "keyword=value1,, , value3" unless you skipped RTFM and expect it to do something completely different. And more often than not that's exactly what you want when parsing non-computer readable input which you can expect to take a specific form. If you want to handle escape sequences, parse from a stream (without having the option to fgets() the next line into memory), or parse CSV tables without collapsin colunms then you will want to use something more specific to that. Luckily, strtok() was not advertised as a Swiss army knife so it's off the hook for specific parsing purposes like those.
- jstimpfle 8y ago
  As someone else pointed out what you really want is the implementation of strspn/strcspn which is where the loop is. You don't "carry" that code along. You just write while (i < len && !is_token_begin(buf[i])) i++; if (i == len) error("End of input\n"); start_token(tok); while (i < len && is_token_char(buf[i])) { i++; add_to_token(tok, buf[i]); } end_token(tok); or something along those lines. Whatever you need. It's not rocket science. Putting highly fluctuating and project-specific code like this in a library would only have disadvantages. Not everything should be in a library. In fact, most things should not be.

nly 8y ago

Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. This is a non-trivial task requiring a real parser operating over a well-specified grammar. A tokenization pipeline just won't cut it. I worked on a project a few years ago that read its custom-format config file in line by line, chopped everything off each line following the first '#' character (to support comments), and then trimmed the whitespace. This sounds like a reasonable and elegant approach until you consider that now none of your user controlled fields (via a GUI in our case) can contain the '#' character. This effected customers, but nobody ever fixed it. With the tools and languages out there now, there's just no excuse for this crap.

yason 8y ago
Unless you're using a pre-specified configuration file format (e.g. TOML), then parsing configuration files requires a general parsing library. If you have needs that require a general parsing library then why are you criticizing strtok()? It doesn't parse XML either, not C source code, nor any unspecified configuration file formats.
- nly 8y ago
  My point is that strtok has almost no reasonable uses in real programs.

Someone 8y ago

> A few strtok() calls can easily parse lines like: keyword=value1, value2, value3 The challenge with parsing isn’t parsing correct inputs; it’s generating useful error messages and recovering on incorrect inputs such as keyword=,,value1, value2, value3,,,, or even =keyword=,,value1, value2, value3,,,, strtok isn’t the best tool for doing that. (Yes, those could be valid inputs, but if they are, chances are they should be parsed differently)

yason 8y ago
That's what I meant: strtok() is well-defined in what it does. The man page is really short. I don't understand people who complain it doesn't do something it's not supposed to do. Yet it's very useful in what it does. Your mileage may vary but it's also a common issue having to filter out empty items out of something like ",foo,bar ,,,,baz,,xyzzy," where you only really want those four words. You will especially encounter this in parsing user input which might have extra whitespace, or badly formed lists of items. Use strtok() to split C strings into tokens delimited by the given set of delimiters. If you want to catch each comma but skip over any whitespace and split at the first '=' only, then use something else. Let's say if I'm going to need a configuration file for my program I'd most certainly start with something I can parse with strtok(). I would really need very specific needs to warrant a more complex format that would require a more sophisticated parser in which case using one would be a no-brainer.