3 ms·
I struggle to think of what parallel processing would mean when parsing(!) a newline delimited file. If the file format had offsets specified (e.g. pdf), then I
by mdaniel 2y ago
I struggle to think of what parallel processing would mean when parsing(!) a newline delimited file. If the file format had offsets specified (e.g. pdf), then I'm on board with delegating the parsing of each declared segment - like a very small map-reduce program
But CSVs are infamous for yolo-ing quoting and EOL treatment so just ripping through and spawning threads for each discovered EOL sequence seems like a recipe for misparse. I guess it's possible to guard the thread worker with the expected number of columns and if their parse attempt doesn't find that many then they can bail out with the expectation that the worker assigned the actual EOL sequence will rescue the error. But what a waste of I/O (and cpu, of course)