4 ms·
From looking at their source briefly, I can make two observations: Firstly, their core parsing routine is a state machine not unlike the one found in rust-csv:
by burntsushi 10y ago
From looking at their source briefly, I can make two observations:
Firstly, their core parsing routine is a state machine not unlike the one found in rust-csv: https://github.com/wiseio/paratext/blob/master/src/csv/rowbased_worker.hpp#L48 https://github.com/wiseio/paratext/blob/master/src/csv/rowba...
Secondly, they do indeed appear to be achieving high parsing speed using coarsely grained parallelism. You can see the code that chunks up the CSV data here: https://github.com/wiseio/paratext/blob/master/src/generic/chunker.hpp https://github.com/wiseio/paratext/blob/master/src/generic/c...
It's pretty clever!
- Ygg2 10y agoCan this be used to speed up rust-csv?
- burntsushi 10y agoIt feels like something that should be built on top of rust-csv.
- Ygg2 10y agoI thought this was base ability of the parser to parse CSV in chunks.