4 ms·
One thing that high-performance polars/arrow enables is to build ETL without data sharding because in most cases, all your data fits into one node. Jupyter and
by rubenfiszel 4y ago
One thing that high-performance polars/arrow enables is to build ETL without data sharding because in most cases, all your data fits into one node. Jupyter and local scripts become sufficient and you avoid a lot of complexity.
We hesitated a lot before mentioning ETL in the OSS project windmill [1] but since we support any python including polars, passing data between steps through shared folders and since polars is blazingly fast, it is a suprisingly good system for ETL waiting for the need of using spark/materialize directly.
[1]: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill
- RockyMcNuts 4y agoVertical is the new horizontal, because of stuff like Epycs with 96 cores/192 threads, AWS instances with 24TB of RAM, M2 MacBook Pros with 96GB RAM, 12 cores, ludicrous SOC bandwidth. What used to be 'big data' fits on an instance, you only need shards/clusters for 'ludicrous data'.
- RockyMcNuts 4y agoAlso, it would make me so happy if someone did a proper benchmark of - pandas - polars - modin with various backends - DuckDB and Postgres SQL from Jupyter/python with the usual SQL dataframe magic - R - on medium and 'pretty big data' i.e. 10GB dataframes with millions of rows. - maybe on consumer Intel, M2, and Epyc with lots of cores