3 ms·
> Note that just replacing separators and newlines with nulls isn't enough to parse CSVs because the file format supports escape sequences and optional quotes a
by thethirdone 6y ago
> Note that just replacing separators and newlines with nulls isn't enough to parse CSVs because the file format supports escape sequences and optional quotes around values. If you're modifying a buffer in place, you'll still be stuck copying data to replace (longer) escape sequences with their (shorter) final values.
I had forgotten that escape sequences needed to also be changed. Unescaping in place does make it non-trivial.
> I don't understand the line you've quoted from the article regarding why this code is slower than csv2. The csv2 code seemingly does copy from the original buffer rather than manipulating in place. Indeed, the csv2 parser copies data from a buffer into a C++ STL container object, which I would expect to be quite an expensive operation. Something does not add up here, either in my understanding or in the mechanics/description of the benchmark.
I'm not familiar with the csv2 parser, but if what you say is true, I guess the slowdown would come from doing a csv2 doing a larger copy(s) and the given code doing many smaller copies.
- formerly_proven 6y ago> I had forgotten that escape sequences needed to also be changed. Unescaping in place does make it non-trivial. Assuming escapes are rare, backshift unescaping is totally an option, though it scales poorly (interpreting escapes is linear, backshifting is quadratic) and is thus susceptible to slowdowns with malicious inputs. It does need very little code and no extra memory at all, though.
- fanf2 6y agoBackshifting is not quadratic :-) dst = src = ptr; while(*src != '\0') { if(src[0] == '"' && src[1] == '"') { src++; } *dst++ = *src++; }
- formerly_proven 6y agoThis bothered me a little, so I went and dug out the only instance were I remember using this technique, and it's quite literally what you wrote. All these high level immutable languages must be getting to me. Thanks :)