3 ms·
Isn't this also addressed by using Dask?
by tootie 4y ago
Isn't this also addressed by using Dask?
- jwmoz 4y agoI tried dask specifically for this task, processing a larger file on an 8GB mem macbook air. It was wildly ineffective. Ended up having to chop up the data and do it myself.
- TrapLord_Rhodo 4y agoDask can utilize pandas dataframes and make them run across different cores. Also is great for tasks that take up too much memory as discussed by the OP. This may replace the need for you to use DASK in alot of projects since you can work with 3.5x the RAM now.
- MrPowers 4y agoDask sidesteps the problem by splitting up the data into smaller pandas DataFrames and processing them individually, rather than all at once. But the fundamental issue of pandas storing strings as objects in DataFrames (which is really inefficient) is also problematic with Dask (because a Dask DataFrame is composed of a lot of smaller pandas DataFrames). The technologies that really address this problem are Polars/PySpark - technologies that are a lot more memory efficient.
- 0cf8612b2e1e 4y agoI am always tripping over my feet when I use Dask. Definitely due to inexperience (I only reach for it when my usual Pandas tricks fail me), but I never have a good time there. Pandas idiom X will still blow up memory, you have to know Dask idiom Y, etc. I am glad Dask exists, but it is not a seamless solution where you can just swap in a Dask dataframe.