8 ms·

String tokenization in C

155 points by throwaway2419 8y ago

megous 8y ago
Other approach from library calls and flex is re2c. It preprocesses the source code and inlines regular expression parsing where you needed. It's very powerful in combination with goto.
jfries 8y ago
Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.
- jstimpfle 8y ago
  You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.
  - dahart 8y ago
    > Yout don’t really use regex in C. Speak for yourself. There’s a POSIX standard for regex that is more than 30 years old & a GNU implementation that comes with gcc. C++ has regex in the standard library.
    - jstimpfle 8y ago
      "You" used like "one", or "in common practice it isn't really used". It's just not in C's spirit to use canned libraries. Such use cases have long been transferred to Python and other languages. Sure, there is a regex API in the C standard library. I'd bet not even grep uses it. The one use-case I'm envisioning is quickly exposing POSIX conformant regexen to the command-line.
      - dahart 8y ago
        > I’d bet not even grep uses it. I’ll take you up on that bet. I see #include <regex.h> in every source repo of grep I can find right now. http://git.savannah.gnu.org/cgit/grep.git/tree/src/search.h http://git.savannah.gnu.org/cgit/grep.git/tree/src/search.h https://opensource.apple.com/source/text_cmds/text_cmds-99/grep/grep.h.auto.html https://opensource.apple.com/source/text_cmds/text_cmds-99/g... https://android.googlesource.com/platform/system/core.git/+/android-4.2.2_r1/toolbox/grep/grep.h https://android.googlesource.com/platform/system/core.git/+/... https://github.com/c9/node-gnu-tools/blob/master/grep-src/src/search.h https://github.com/c9/node-gnu-tools/blob/master/grep-src/sr... You owe me a beer. :) BTW, I do super agree with your comment to just not use strtok, and also the idea that most people are better off parsing text in perl or python...
        jstimpfle 8y ago
        Not sure why that header is included there, though. I can't find any uses of the regex library. There are multiple custom matchers implemented. So maybe it's that the GNU regex library uses the grep sources. In any case, there don't seem to be any uses of regexec() or regcomp(), for example. Which would have surprised me anyway since that API is rather limiting (you cannot search incrementally). Let's get that beer sometime, anyway.
      - kahirsch 8y ago
        GNU grep uses the regex library in Gnulib. It can also use the Perl-compatible pcre library. I am finding it hard to believe that you think it's "just not in C's spirit to use canned libraries."
        jstimpfle 8y ago
        I was saying "canned", not "boxed" or "containerized". It's a lot easier to write the little things yourself. (And there are good benefits to be had from writing specialized code yourself, instead of relying on big fat generalized tankers.)
        8y ago
  - barrkel 8y ago
    Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand-coded implementations of regular expression matching. Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to know exactly where you are in the regular language that defines the tokens to write good code. Writing a regex matcher in code like this is like writing code in assembly - mentally, you're mapping to a different set of concepts all the time.
    - jstimpfle 8y ago
      Yes. I don't think anyone is disagreeing here. If you're implying that we should then use a regex implementation instead: Coding up a lexer (for a mainstream programming language) using simple counter increments and such is not a lot of work. It has the advantage that it results in faster code (unless you're going for real heavy machinery) and that you can easily code up additional transformations. For example, how would you parse string literals (including escape sequences) with a regex?
      - mcguire 8y ago
        "(.|\.)*"
        jstimpfle 8y ago
        "Evaluation:\tWrong!\x07\x07\x07\r\n" You want to convert the string literal to an internal buffer, interpreting the escape sequences. In the same way, you want to parse integers. You cannot really do that with a regular expression. RE is for matching, not for transforming.
      - jcranberry 8y ago
        String literals are easy without escaped quotes. With escaped quotes its annoying and non regex is much cleaner.
  - int_19h 8y ago
    Lexer generators are pretty popular in C land. I mean, there's Yacc, obviously. And then there's more low-level stuff like re2c.
    - tptacek 8y ago
      Yacc is a parser generator, not a scanner generator. You meant lex/flex.
- johnisgood 8y ago
  I do not believe that using regex is necessary. I have parsed a lot of code in my life and regex was not a necessity.
  - GordonS 8y ago
    Agreed. Regex can make parsing code much more succinct and easiet to grok (although usually at a small performance cost). So not "necessary", but it can be really useful.
- colejohnson66 8y ago
  I wrote a tokenizer for a language I’m creating, and all I needed was read character and peek character from an iterator
  - barrkel 8y ago
    Guess what? You wrote a state machine (DFA most likely) in code, where the program counter represents the state. There is in all probability (if your language is sane), a 1:1 correspondence between the code you wrote and the regular grammar of your tokens. You implemented a regular language matcher, i.e. a regex matcher, but in code rather than via a regular expression language interpreter or compiler. Typical pattern: start = p; while (isspace(*p) && p < eof) // [ ]* ++p; if (p == eof) return EOF; if (is_ident_start(*p)) { // [a-z] ++p; while (is_ident(*p)) // [a-z0-9]* ++p; set_token(p, p - start); return IDENT; } else if (is_number(*p)) { // [0-9] ++p; while (is_number(*p)) // [0-9]* ++p; set_token(p, p - start); return NUMBER; } // etc. Corresponds to: IDENT ::= [a-z][a-z0-9]* ; NUMBER ::= [0-9][0-9]* ; SPACE ::= [ ]* ; TOKEN ::= SPACE (IDENT | NUMBER) ; Inline those nonterminals, and guess what - regular expression!
    - jstimpfle 8y ago
      That works until you need your own implementation of ++p;. Now wiring up that implementation to a generic library is already more work than just doing it all yourself. Not even considering the integration costs of the library into the sources and the build. And you cannot really do transformation instead of only matching with RE. Those are needed already in simple cases like string and number literals. Now your code will look more like %{ #include "y.tab.h" int num_lines = 1; int comment_mode=0; int stack =0; %} digit ([0-9]) integer ({digit}+) float_num ({digit}+\.{digit}+) %% {integer} { //deal with integer printf("#%d: NUM:",num_lines); ECHO;printf("\n"); yylval.Integer = atoi(yytext); return INT; } {float_num} {// deal with float printf("#%d: NUM:",num_lines);ECHO;printf("\n"); yylval.Float = atof(yytext); return FLOAT; } \n { ++num_lines; } . if(strcmp(yytext," "))ECHO; %% int yywrap() { return 1; } (copied from stackoverflow). And that's before preprocessing. Yeah. Thanks, but no thanks.
pasokan 8y ago
It used to be that gcc will warn against strtok and recommend strsep instead. Do not know what the status is today
- tinus_hn 8y ago
  Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.
  - morbusfonticuli 8y ago
    > Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it. Well, there is already a thread-safe variant [0]: > The strtok() function uses a static buffer while parsing, so it's not thread safe. Use strtok_r() if this matters to you. [0] https://linux.die.net/man/3/strtok_r https://linux.die.net/man/3/strtok_r
    - seba_dos1 8y ago
      ...with different API :P
      - enriquto 8y ago
        of course! the original API cannot be made re-entrant
  - heinrichhartman 8y ago
    Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?
    - Sharlin 8y ago
      Yes, as long as you can guarantee that there’s only one tokenization going on per thread at a time.
caf 8y ago
Note though that strsep() is not as portable, because it is an extension to standard C.
- beefhash 8y ago
  In fact, it's not even in POSIX.
  - teddyh 8y ago
    To quote the GNU C library manual: “This function was introduced in 4.3BSD and therefore is widely available.”¹ 1. https://www.gnu.org/software/libc/manual/html_node/Finding-Tokens-in-a-String.html#index-strsep https://www.gnu.org/software/libc/manual/html_node/Finding-T...
    - johannes1234321 8y ago
      Widely except Windows and different embedded platforms etc.
- tptacek 8y ago
  It's a tiny function, written in ANSI C, so if you're really concerned about this, just include it with your program. It's an extension to the standard C library, not to C itself.
  - beefhash 8y ago
    Except then you have the issue about compilers complaining about double-declarations of the function, meaning you'll either have a lot of warning spam on every #include or now hard require some kind of header defines for HAVE_STRSEP. Once you go that way, there's no going back and it's only gonna become more and more.
    - tptacek 8y ago
      People have been including strsep in packages since the 1990s (people used to include their own snprintfs, ffs). If you're really this freaked out about it, call your local copy "mystrsep" or something like that. You know what else isn't in POSIX? All the rest of your C code.
setquk 8y ago
I just use flex. You don’t have to ship flex as a dependency either.
jstimpfle 8y ago
strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines yourself.
- nly 8y ago
  Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.
  - agumonkey 8y ago
    any attempt has been made to build a nicer foundational C/OS library since ?
    - nly 8y ago
      BSD has some minor improvements to libc. There's also glib. Generally though, C is just a terrible language to do anything other than write extremely low level routines in. You should probably just never use strtok() and go straight to something like Ragel or re2c for building high performance tokenizers... then call those from a higher level language.
      - yakovdk 8y ago
        Hey! I take exception to that. I've been writing shellcode all week. C is uncomfortably high level for writing low level routines!
    - jstimpfle 8y ago
      There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.
lixtra 8y ago
I have an obsession with unsafe example code: strcpy(str,"abc,def,ghi"); token = strtok(str,","); printf("%s \n",token); Even if the author knows how many tokens are returned I would prefer a check for NULL here since a good fraction might not read further than this bad example.
- enriquto 8y ago
  > I have an obsession with unsafe example code: It is perfectly OK for example code to be unsafe. You do not wear a parachute when you learn to fly using a simulator. You realize that things will become more serious and complicated in the future, but you have to start with something simple and unsafe, no big deal. Otherwise you will never see the consequences of unsafe code in simple cases.
  - bqe 8y ago
    I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs.
    - enriquto 8y ago
      > I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs. Even if this is true, the reasoning here is disturbingly short-sighted. Copying code that you do not understand is unacceptable behavior, and I'd say the sooner it blows up in your face, the better. The goal of code examples is to illustrate how things work in a simplified way, and code without error checks is often easier to understand at first. Imagine a hello world with all the possible error checks. That would be incomprehensible.
graycat 8y ago
A lot of experience shows that the string tokenization in Open Object Rexx is darned useful. E.g., for many years, IBM's internal computing was from about 3600 mainframe computers around the world running VM/CMS with a lot of service machines written in Rexx. Rexx is no toy but a powerful, polished, scripting language and really good at handling strings. A little example of some Rexx code with some string parsing is in https://news.ycombinator.com/item?id=18648999 https://news.ycombinator.com/item?id=18648999
stochastic_monk 8y ago
I recommend ksplit/ksplit_core from Heng Li’s excellent klib kstring.{h,c}[0]. It modifies the string in-place, adding null terminators, and provides a list of offsets into the string. This gives you the flexibility of accessing tokens by index without paying costs of copying or memory allocation. [0] https://github.com/attractivechaos/klib https://github.com/attractivechaos/klib
rurban 8y ago
AFAIK strtok has restrict on both args since C99. And the safe variants strtok_s and esp. wcstok_s are missing. Strings are unicode nowadays, not ASCII. https://en.cppreference.com/w/c/string/byte/strtok https://en.cppreference.com/w/c/string/byte/strtok
bsenftner 8y ago
...And then the application is required to implement variable length characters, a la Unicode, and you start your strings logic all over...
- syrrim 8y ago
  As long as you're fine with ascii delimiters, strtok et al. work fine for utf-8 strings.
  - bsenftner 8y ago
    Would you happen to be aware of good Unicode normalization function/lib in C/C++?
kazinator 8y ago
The actions of strtok can easily be coded using strspn and strcspn. https://groups.google.com/forum/message/raw?msg=comp.lang.c/ZhXAlw6VZsA/_Y5evTIkf6kJ https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2001] https://groups.google.com/forum/message/raw?msg=comp.lang.c/ff0xFqRPH_Y/Cen0mgciXn8J https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2011 repost] strspn(s, bag) calculates the length of the prefix of string s which consists only of the characters in string bag. strcspn(s, bag) calculates the length of the prefix of s consisting of characters not in bag. The bag is like a one-character regex class; so that is to say strspn(s, "abcd") is like calculating the length of the token at the front of input s matching the regex [abcd]* , and in the case of strcspn, that becomes [^abcd]* .
- saagarjha 8y ago
  And it’s nicer, since you can pass in a const char * and use it in concurrent code.
saagarjha 8y ago
str = (char *) malloc(sizeof(char) * (strlen(TESTSTRING)+1)); strcpy(str,TESTSTRING); str = strdup(TESTSTRING)?
alexandernst 8y ago
How about just using a properly suited language por string manipulation?
the_clarence 8y ago
Problem is that your token string is going to be quite large. Is there a built-in solution for when tokens are just single chars?
satyenr 8y ago
> Next, strtok is not thread-safe. That's because it uses a static buffer internally. So, you should take care that only one thread in your program calls strtok at a time. I wonder why strtok() does not use an output parameter similar to scanf() — and return the number of tokens. Something like: int strtok(char *str, char *delim, char **tokens); Granted, it would involve dynamic memory allocation and the implementation that immediately comes to mind would be less efficient than the current implementation, but surely it’s worth eliminating the kind of bugs the current strtok() can introduce? Does anyone here have the historical prospective?

155 points by throwaway2419 8y ago

Other approach from library calls and flex is re2c. It preprocesses the source code and inlines regular expression parsing where you needed. It's very powerful in combination with goto.

jfries 8y ago

Well, yes, using strtok works if the data happens to be structured in a certain simple way. Very often you want to do something more advanced though, and using regex for matching tokens is then necessary.

jstimpfle 8y ago
You don't really use regex in C. You just write a few simple loops. Look up the lexer of the programming language of your choice.
- dahart 8y ago
  > Yout don’t really use regex in C. Speak for yourself. There’s a POSIX standard for regex that is more than 30 years old & a GNU implementation that comes with gcc. C++ has regex in the standard library.
  - jstimpfle 8y ago
    "You" used like "one", or "in common practice it isn't really used". It's just not in C's spirit to use canned libraries. Such use cases have long been transferred to Python and other languages. Sure, there is a regex API in the C standard library. I'd bet not even grep uses it. The one use-case I'm envisioning is quickly exposing POSIX conformant regexen to the command-line.
    - dahart 8y ago
      > I’d bet not even grep uses it. I’ll take you up on that bet. I see #include <regex.h> in every source repo of grep I can find right now. http://git.savannah.gnu.org/cgit/grep.git/tree/src/search.h http://git.savannah.gnu.org/cgit/grep.git/tree/src/search.h https://opensource.apple.com/source/text_cmds/text_cmds-99/grep/grep.h.auto.html https://opensource.apple.com/source/text_cmds/text_cmds-99/g... https://android.googlesource.com/platform/system/core.git/+/android-4.2.2_r1/toolbox/grep/grep.h https://android.googlesource.com/platform/system/core.git/+/... https://github.com/c9/node-gnu-tools/blob/master/grep-src/src/search.h https://github.com/c9/node-gnu-tools/blob/master/grep-src/sr... You owe me a beer. :) BTW, I do super agree with your comment to just not use strtok, and also the idea that most people are better off parsing text in perl or python...
      - jstimpfle 8y ago
        Not sure why that header is included there, though. I can't find any uses of the regex library. There are multiple custom matchers implemented. So maybe it's that the GNU regex library uses the grep sources. In any case, there don't seem to be any uses of regexec() or regcomp(), for example. Which would have surprised me anyway since that API is rather limiting (you cannot search incrementally). Let's get that beer sometime, anyway.
    - kahirsch 8y ago
      GNU grep uses the regex library in Gnulib. It can also use the Perl-compatible pcre library. I am finding it hard to believe that you think it's "just not in C's spirit to use canned libraries."
      - jstimpfle 8y ago
        I was saying "canned", not "boxed" or "containerized". It's a lot easier to write the little things yourself. (And there are good benefits to be had from writing specialized code yourself, instead of relying on big fat generalized tankers.)
        8y ago
- barrkel 8y ago
  Most lexers are state machines, either explicit with tables (like you get from lex) or implicit with program counter (with loops and switches). Those state machines implement matchers for regular languages; they're effectively hand-coded implementations of regular expression matching. Regular expressions don't show up outside the spec, sure; but if you're writing the code (for implicit state machine), you need to know exactly where you are in the regular language that defines the tokens to write good code. Writing a regex matcher in code like this is like writing code in assembly - mentally, you're mapping to a different set of concepts all the time.
  - jstimpfle 8y ago
    Yes. I don't think anyone is disagreeing here. If you're implying that we should then use a regex implementation instead: Coding up a lexer (for a mainstream programming language) using simple counter increments and such is not a lot of work. It has the advantage that it results in faster code (unless you're going for real heavy machinery) and that you can easily code up additional transformations. For example, how would you parse string literals (including escape sequences) with a regex?
    - mcguire 8y ago
      "(.|\.)*"
      - jstimpfle 8y ago
        "Evaluation:\tWrong!\x07\x07\x07\r\n" You want to convert the string literal to an internal buffer, interpreting the escape sequences. In the same way, you want to parse integers. You cannot really do that with a regular expression. RE is for matching, not for transforming.
    - jcranberry 8y ago
      String literals are easy without escaped quotes. With escaped quotes its annoying and non regex is much cleaner.
- int_19h 8y ago
  Lexer generators are pretty popular in C land. I mean, there's Yacc, obviously. And then there's more low-level stuff like re2c.
  - tptacek 8y ago
    Yacc is a parser generator, not a scanner generator. You meant lex/flex.
johnisgood 8y ago
I do not believe that using regex is necessary. I have parsed a lot of code in my life and regex was not a necessity.
- GordonS 8y ago
  Agreed. Regex can make parsing code much more succinct and easiet to grok (although usually at a small performance cost). So not "necessary", but it can be really useful.
colejohnson66 8y ago
I wrote a tokenizer for a language I’m creating, and all I needed was read character and peek character from an iterator
- barrkel 8y ago
  Guess what? You wrote a state machine (DFA most likely) in code, where the program counter represents the state. There is in all probability (if your language is sane), a 1:1 correspondence between the code you wrote and the regular grammar of your tokens. You implemented a regular language matcher, i.e. a regex matcher, but in code rather than via a regular expression language interpreter or compiler. Typical pattern: start = p; while (isspace(*p) && p < eof) // [ ]* ++p; if (p == eof) return EOF; if (is_ident_start(*p)) { // [a-z] ++p; while (is_ident(*p)) // [a-z0-9]* ++p; set_token(p, p - start); return IDENT; } else if (is_number(*p)) { // [0-9] ++p; while (is_number(*p)) // [0-9]* ++p; set_token(p, p - start); return NUMBER; } // etc. Corresponds to: IDENT ::= [a-z][a-z0-9]* ; NUMBER ::= [0-9][0-9]* ; SPACE ::= [ ]* ; TOKEN ::= SPACE (IDENT | NUMBER) ; Inline those nonterminals, and guess what - regular expression!
  - jstimpfle 8y ago
    That works until you need your own implementation of ++p;. Now wiring up that implementation to a generic library is already more work than just doing it all yourself. Not even considering the integration costs of the library into the sources and the build. And you cannot really do transformation instead of only matching with RE. Those are needed already in simple cases like string and number literals. Now your code will look more like %{ #include "y.tab.h" int num_lines = 1; int comment_mode=0; int stack =0; %} digit ([0-9]) integer ({digit}+) float_num ({digit}+\.{digit}+) %% {integer} { //deal with integer printf("#%d: NUM:",num_lines); ECHO;printf("\n"); yylval.Integer = atoi(yytext); return INT; } {float_num} {// deal with float printf("#%d: NUM:",num_lines);ECHO;printf("\n"); yylval.Float = atof(yytext); return FLOAT; } \n { ++num_lines; } . if(strcmp(yytext," "))ECHO; %% int yywrap() { return 1; } (copied from stackoverflow). And that's before preprocessing. Yeah. Thanks, but no thanks.

pasokan 8y ago

It used to be that gcc will warn against strtok and recommend strsep instead. Do not know what the status is today

tinus_hn 8y ago
Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it.
- morbusfonticuli 8y ago
  > Strtok is not thread safe and can’t be made thread safe without changing the API. You should not use it. Well, there is already a thread-safe variant [0]: > The strtok() function uses a static buffer while parsing, so it's not thread safe. Use strtok_r() if this matters to you. [0] https://linux.die.net/man/3/strtok_r https://linux.die.net/man/3/strtok_r
  - seba_dos1 8y ago
    ...with different API :P
    - enriquto 8y ago
      of course! the original API cannot be made re-entrant
- heinrichhartman 8y ago
  Well, strtok could use thread local variables to store intermediate state, to make it threadsafe while maintaining the same API. Not saying this is a good idea, but technically it would work, no?
  - Sharlin 8y ago
    Yes, as long as you can guarantee that there’s only one tokenization going on per thread at a time.

caf 8y ago

Note though that strsep() is not as portable, because it is an extension to standard C.

beefhash 8y ago
In fact, it's not even in POSIX.
- teddyh 8y ago
  To quote the GNU C library manual: “This function was introduced in 4.3BSD and therefore is widely available.”¹ 1. https://www.gnu.org/software/libc/manual/html_node/Finding-Tokens-in-a-String.html#index-strsep https://www.gnu.org/software/libc/manual/html_node/Finding-T...
  - johannes1234321 8y ago
    Widely except Windows and different embedded platforms etc.
tptacek 8y ago
It's a tiny function, written in ANSI C, so if you're really concerned about this, just include it with your program. It's an extension to the standard C library, not to C itself.
- beefhash 8y ago
  Except then you have the issue about compilers complaining about double-declarations of the function, meaning you'll either have a lot of warning spam on every #include or now hard require some kind of header defines for HAVE_STRSEP. Once you go that way, there's no going back and it's only gonna become more and more.
  - tptacek 8y ago
    People have been including strsep in packages since the 1990s (people used to include their own snprintfs, ffs). If you're really this freaked out about it, call your local copy "mystrsep" or something like that. You know what else isn't in POSIX? All the rest of your C code.

setquk 8y ago

I just use flex. You don’t have to ship flex as a dependency either.

jstimpfle 8y ago

strtok is one of the silliest parts of the standard library. (And there are many bad ones). It's broken. It's not thread safe (yes there is strtok_r). It's needlessly hard to use. And it writes zeros to the input array. The latter means it's unfit for most use cases, including non-trivial tokenization where you want e.g. to split "a+1" into three tokens. If you program in C please just write those four obvious lines yourself.

nly 8y ago
Much of libc is terrible from an API design perspective, even given the limitations of C as a language. libc has somehow managed to hit the sweet spot and have APIs that are both inconvenient to use properly, and perform poorly.
- agumonkey 8y ago
  any attempt has been made to build a nicer foundational C/OS library since ?
  - nly 8y ago
    BSD has some minor improvements to libc. There's also glib. Generally though, C is just a terrible language to do anything other than write extremely low level routines in. You should probably just never use strtok() and go straight to something like Ragel or re2c for building high performance tokenizers... then call those from a higher level language.
    - yakovdk 8y ago
      Hey! I take exception to that. I've been writing shellcode all week. C is uncomfortably high level for writing low level routines!
  - jstimpfle 8y ago
    There are of course tons of code on the net, but there is no need to standardize another grab-bag of bad API calls. If you need a batteries-included library you use python or similar. If you write in C, you care so much that you largely avoid the standard library and design your code from the ground up.

lixtra 8y ago

I have an obsession with unsafe example code: strcpy(str,"abc,def,ghi"); token = strtok(str,","); printf("%s \n",token); Even if the author knows how many tokens are returned I would prefer a check for NULL here since a good fraction might not read further than this bad example.

enriquto 8y ago
> I have an obsession with unsafe example code: It is perfectly OK for example code to be unsafe. You do not wear a parachute when you learn to fly using a simulator. You realize that things will become more serious and complicated in the future, but you have to start with something simple and unsafe, no big deal. Otherwise you will never see the consequences of unsafe code in simple cases.
- bqe 8y ago
  I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs.
  - enriquto 8y ago
    > I think you underestimate how many people blindly copy examples without understanding them. Safe example code results in more correct programs. Even if this is true, the reasoning here is disturbingly short-sighted. Copying code that you do not understand is unacceptable behavior, and I'd say the sooner it blows up in your face, the better. The goal of code examples is to illustrate how things work in a simplified way, and code without error checks is often easier to understand at first. Imagine a hello world with all the possible error checks. That would be incomprehensible.

graycat 8y ago

A lot of experience shows that the string tokenization in Open Object Rexx is darned useful. E.g., for many years, IBM's internal computing was from about 3600 mainframe computers around the world running VM/CMS with a lot of service machines written in Rexx. Rexx is no toy but a powerful, polished, scripting language and really good at handling strings. A little example of some Rexx code with some string parsing is in https://news.ycombinator.com/item?id=18648999 https://news.ycombinator.com/item?id=18648999

stochastic_monk 8y ago

I recommend ksplit/ksplit_core from Heng Li’s excellent klib kstring.{h,c}[0]. It modifies the string in-place, adding null terminators, and provides a list of offsets into the string. This gives you the flexibility of accessing tokens by index without paying costs of copying or memory allocation. [0] https://github.com/attractivechaos/klib https://github.com/attractivechaos/klib

rurban 8y ago

AFAIK strtok has restrict on both args since C99. And the safe variants strtok_s and esp. wcstok_s are missing. Strings are unicode nowadays, not ASCII. https://en.cppreference.com/w/c/string/byte/strtok https://en.cppreference.com/w/c/string/byte/strtok

bsenftner 8y ago

...And then the application is required to implement variable length characters, a la Unicode, and you start your strings logic all over...

syrrim 8y ago
As long as you're fine with ascii delimiters, strtok et al. work fine for utf-8 strings.
- bsenftner 8y ago
  Would you happen to be aware of good Unicode normalization function/lib in C/C++?

kazinator 8y ago

The actions of strtok can easily be coded using strspn and strcspn. https://groups.google.com/forum/message/raw?msg=comp.lang.c/ZhXAlw6VZsA/_Y5evTIkf6kJ https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2001] https://groups.google.com/forum/message/raw?msg=comp.lang.c/ff0xFqRPH_Y/Cen0mgciXn8J https://groups.google.com/forum/message/raw?msg=comp.lang.c/... [2011 repost] strspn(s, bag) calculates the length of the prefix of string s which consists only of the characters in string bag. strcspn(s, bag) calculates the length of the prefix of s consisting of characters not in bag. The bag is like a one-character regex class; so that is to say strspn(s, "abcd") is like calculating the length of the token at the front of input s matching the regex [abcd]* , and in the case of strcspn, that becomes [^abcd]* .

saagarjha 8y ago
And it’s nicer, since you can pass in a const char * and use it in concurrent code.

saagarjha 8y ago

str = (char *) malloc(sizeof(char) * (strlen(TESTSTRING)+1)); strcpy(str,TESTSTRING); str = strdup(TESTSTRING)?

alexandernst 8y ago

How about just using a properly suited language por string manipulation?

the_clarence 8y ago

Problem is that your token string is going to be quite large. Is there a built-in solution for when tokens are just single chars?

satyenr 8y ago

> Next, strtok is not thread-safe. That's because it uses a static buffer internally. So, you should take care that only one thread in your program calls strtok at a time. I wonder why strtok() does not use an output parameter similar to scanf() — and return the number of tokens. Something like: int strtok(char *str, char *delim, char **tokens); Granted, it would involve dynamic memory allocation and the implementation that immediately comes to mind would be less efficient than the current implementation, but surely it’s worth eliminating the kind of bugs the current strtok() can introduce? Does anyone here have the historical prospective?