3 ms·
strtok() altering the source string is a natural consequence of null termination. The traditional Pascalian equivalent (put the string length at the head of the
by philwelch 16y ago
strtok() altering the source string is a natural consequence of null termination. The traditional Pascalian equivalent (put the string length at the head of the string) creates a symmetric problem at the head of the string. My (perhaps naive) idea is to use a struct containing the string length and pointing to the beginning of the string data. Nondestructive tokenization becomes simple.
This post actually gave me an impetus to go back and add tokenization to my string library that's built around this idea.
- tptacek 16y agoThe big problem with tok isn't that it alters the string; it's that it keeps state, and so can't be used in multicontext code. sep changes the interface to delegate state tracking to the caller, which is the way it should be. Just as map and reduce in Python is "unpythonic", interfaces that mint new temp strings to process what should be a simple charstar are un-c-like, and should be avoided.
- philwelch 16y agoJust as map and reduce in Python is "unpythonic", interfaces that mint new temp strings to process what should be a simple charstar are un-c-like, and should be avoided. That's not what I'm doing at all, though. I'm just giving char *'s and size_t's for portions of the original character array, not duplicating subsections of the array at all.
- tptacek 16y agoYou are, in other words, reinventing strspn?
- philwelch 16y agostrspn takes a set of characters to match against and tells you how long to go until you reach a non-matching character. My tokenizer, like strtok, takes a set of delimiter characters. So same thing except with complement sets, I guess. It's also part of a larger library that consistently uses pointer/length pairs to designate strings as opposed to null termination, which admittedly is quite non-C. No clue whether it's actually a good idea. EDIT: If anyone's actually curious, I have it on github: http://github.com/philwelch/string http://github.com/philwelch/string