2 ms·
Assume all linebreaks aren't part of quoted strings. Split the data in linebreaks. Process each line in parallel (massive speedup, can use many cores or even
by londons_explore 6y ago
Assume all linebreaks aren't part of quoted strings.
Split the data in linebreaks.
Process each line in parallel (massive speedup, can use many cores or even GPU).
Record "errors" (ie. Where a quoted block doesn't end before the newline)
Remove and reprocess all records containing errors or following a record with errors. Repeat until no errors exist.
- burntsushi 6y agoYeah, you kind of fell right into the trap I was talking about. Your approach either requires being able to memory map the CSV data or storing the CSV data in memory. And to be honest, it's not clear to me that your approach would end up being faster anyway. There's a lot of overhead buried in there, and it sounds like it's at least two passes over the data in common cases. And in cases where there are line breaks in the data, your approach will perform very poorly I think.