3 ms·
> Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself. I ca
by tbrownaw 5y ago
> Removing the ability to embed record delimiters, for example, means you can process the records in parallel. That’s a massive improvement all by itself.
I can process records from a csv in parallel just fine. Lexing isn't slow.
.csv stream -> tokens -> records (-> aggregate into ~100ms blocks) -> work queue with N workers
- UncleEntity 5y agoYou lex the whole file, separate it into records and then process them? Doesn’t sound like you’re processing the file in parallel to me. TFA’s way would allow you to start at any arbitrary file position, just advance until you find a record separator then start parsing from there without having to worry about starting in the middle of a quoted string.
- tbrownaw 5y agoNot like sequential steps, a whole bunch of foreach / yield return loops until the last one dumps things onto a thread-safe queue.