3 ms·
Agree. Espacially, a TSV file is very easy to process in chunks, you don't need a specialized parser like with JSON where most libs just loads the whole thing.
by BiteCode_dev 3y ago
Agree. Espacially, a TSV file is very easy to process in chunks, you don't need a specialized parser like with JSON where most libs just loads the whole thing.
700kb/s smell super fishy.
I can parse that with one Python VM and a few MB of RAM way faster. And node has a JIT.
Besides, let's imagine it would be effectively the case that Python and Node are too slow for this, that would be the perfect case to delegate the task to infra instead of introducing a whole new language and ecosystem plus a rewrite. E.G: a perfect use case for DuckDB.
- didntcheck 3y agoAgreed that there's something else going on here, but it's worth noting that (C)Python uses reference counting (with periodic cycle breaking), rather than tracing GC. Not sure if that actually affects throughput in this scenario though