3 ms·
column-wide map-reduce over large dataframes usually give you a 1000x or so speedup. With rust you can stream each record and leverage the insane parallelism a
by Fiahil 6y ago
column-wide map-reduce over large dataframes usually give you a 1000x or so speedup.
With rust you can stream each record and leverage the insane parallelism and async-io libs (rayon, crossbeam, tokio) and a very small memory footprint. sure you have asyncio in python but that’s nowhere near the speed of tokio.
- datanecdote 6y agoThanks for the pointers, those crates seem great. The flaky multithreading libs are my least favorite part of python, and rust’s strength in this area seems very appealing.