3 ms·
Don't just parse - convert. In a pipeline to split-parseable data, if you like, such as the possibly smaller, faster, and more general: https://github.com/c-b
by cb321 3y ago
Don't just parse - convert. In a pipeline to split-parseable data, if you like, such as the possibly smaller, faster, and more general: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim (And, ideally, convert all the way to a mmap & go binary format like nio so you don't have to re-parse.)
- theendisney 3y agoFun! Convert to js: csv.split('\n").join('".split(\',\'));a.push("'); So that each line becomes: a.push("foo,bar,baz".split(',')); And then we have an array of arrays.
- neonsunset 3y agoIs also expressible (and is vectorized) in C#. But that's author's code, not mine :)
- cb321 3y agoThat's right - pure splitting is much more SIMD-friendly than..the whole syntax melange. This is another charm to the "partitioned" design. The conversion to split-parseable TSV can run on its own CPU core and the SIMD splitting on its own core. As long as pipe bandwidth suffices, you have very easy parallelism. This kind of design/intent was popular on Unix at the dawn of multiprocessing when there were still Giant Kernel Locks. But it still has merits, even on Windows. If you have a lot of data (and space for it, e.g. in /dev/shm) you can also save all the converted data to a TSV file. That's now soundly "partitionable" at the "nearest ASCII newline to 1/N bytes" and you can then go core-parallel as well as SIMD within cores (but that N-wise pass with memory mapping or mem.views). Admittedly this is probably more helpful when you are doing more computation than just splits, like ASCII-to-binary conversion of fields or such. Plus, someone might have actually exported from Excel (or whatever) into some sound TSV instead of weird quote-escaped-CSV that some think is standardized by rfc 4180 (which itself disavows being a "standard"). In that case, at least, you needn't convert at all. So, I see at least 3 reasons to layer this part of a system as a convert-then-split: pipeline parallelism, file parallelism, and entire pass elision.
- twoodfin 3y agoOf course the “hard part” of CSV parsing is dealing with escapes, which break simple splits. But now I’m wondering if a good approach might be to split on the escape character and then reassemble / parse from there, safe in the knowledge that every character has exactly one interpretation.
- int_19h 3y ago"Normal" CSV doesn't have escape characters. Quotes in quoted strings are escaped by doubling then, and everything else (including newlines) is interpreted as is inside quoted strings.
- datascienced 3y agoThere is no normal csv! I always used Excel as the “standard” when writing a CSV parser. If every field is quoted you can indeed remove the first and last “, then split on “,“ and then replace “” with “ in the fields. Excuse my phone converting the quotes!
- int_19h 3y agoThat is precisely why I put "normal" in quotes. Nevertheless, if there is a way to escape anything at all, usually it is the quotation mark, and usually it is escaped by doubling. Pretty much any other scheme is very unlikely to be properly interpreted in this context.
- datascienced 3y agoYes indeed. To make it easy to parse everything has to be quoted. If some things are quoted then you can’t just split on comma because for example:m “, is a cat”,”, is my boyfriend”,123 etc.
- LeonB 3y agoThere is no spec or standard or consensus on “Normal” CSV. I like it when CSV follows RFC 4180 too - but it’s descriptive not prescriptive.