3 ms·

I’ve had a lot of success with Dask lately. It’s comparable to spark in some ways [0]. Being written in python and built on top of pandas/numpy it allows much m

I’ve had a lot of success with Dask lately. It’s comparable to spark in some ways [0]. Being written in python and built on top of pandas/numpy it allows much more flexibility. It’s also easier to get adoption from data scientists who are usually more comfortable with python than scala. It also has great tools built on top of kubernetes making deployment quick and easy [1]. [0]https://docs.dask.org/en/latest/spark.html https://docs.dask.org/en/latest/spark.html [1]https://github.com/dask/dask-gateway https://github.com/dask/dask-gateway

MrPowers 6y ago
My feeling is that Spark is better for huge ETL jobs and Dask is better for certain types of model building (because of easy access to Python libraries), but don't have any benchmarks to back this up. Would love some benchmark results comparing processing tens of terabytes on equal sized i3.xlarge EC2 clusters to get a better idea of benchmarks. Have you seen any good Spark vs. Dask benchmarks?
- peterthehacker 6y ago
  I would be interested in seeing some spark v dask benchmarks too. Haven’t seen any yet though. In my experience, Dask really shines when you implement custom numpy computations that could only be done in spark UDFs. We saw a decent performance difference there, but for common built-in computations I’d imagine that spark has better performance. Edit: after some googling I found this paper with benchmarks. https://arxiv.org/pdf/1907.13030.pdf https://arxiv.org/pdf/1907.13030.pdf > Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python’s GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer.
  - kornish 6y ago
    It's also worth noting that with Spark, you can perform arbitrary computation using the Dataset API and operating on case classes.
- hogu 6y ago
  The area that we've seen Dask shine the most is ML. This is because the Python ecosystem has more advantages when it comes to ML and there is better GPU support. https://towardsdatascience.com/supercharging-hyperparameter-tuning-with-dask-ab2c28788bcf https://towardsdatascience.com/supercharging-hyperparameter-... https://towardsdatascience.com/random-forest-on-gpus-2000x-faster-than-apache-spark-9561f13b00ae https://towardsdatascience.com/random-forest-on-gpus-2000x-f... Disclaimer: We produced those benchmark. I'm a founder of Saturn (https://www.saturncloud.io/ https://www.saturncloud.io/) and we focus on providing Databricks-like capabilities with Jupyter + Dask + Prefect, so I definitely have strong feelings in this area.
pletnes 6y ago
Just for clarification, dask lets you run any python function on any python datatype. Numpy/pandas is faster, but anything is doable with dask. This makes it eminently flexible, unlike many «big data» tools that only work on tables, or other arbitrary limitations.
- peterthehacker 6y ago
  Yes, you can use dask’s lower level APIs, like futures [0] or dask delayed, and it’ll just pickle the python objects. Dask provides a group of collections APIs, like dataframes [1] and arrays that use more efficient serialization methods. [0]https://docs.dask.org/en/latest/futures.html https://docs.dask.org/en/latest/futures.html [1]https://docs.dask.org/en/latest/dataframe.html https://docs.dask.org/en/latest/dataframe.html

MrPowers 6y ago

My feeling is that Spark is better for huge ETL jobs and Dask is better for certain types of model building (because of easy access to Python libraries), but don't have any benchmarks to back this up. Would love some benchmark results comparing processing tens of terabytes on equal sized i3.xlarge EC2 clusters to get a better idea of benchmarks. Have you seen any good Spark vs. Dask benchmarks?

peterthehacker 6y ago
I would be interested in seeing some spark v dask benchmarks too. Haven’t seen any yet though. In my experience, Dask really shines when you implement custom numpy computations that could only be done in spark UDFs. We saw a decent performance difference there, but for common built-in computations I’d imagine that spark has better performance. Edit: after some googling I found this paper with benchmarks. https://arxiv.org/pdf/1907.13030.pdf https://arxiv.org/pdf/1907.13030.pdf > Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python’s GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer.
- kornish 6y ago
  It's also worth noting that with Spark, you can perform arbitrary computation using the Dataset API and operating on case classes.
hogu 6y ago
The area that we've seen Dask shine the most is ML. This is because the Python ecosystem has more advantages when it comes to ML and there is better GPU support. https://towardsdatascience.com/supercharging-hyperparameter-tuning-with-dask-ab2c28788bcf https://towardsdatascience.com/supercharging-hyperparameter-... https://towardsdatascience.com/random-forest-on-gpus-2000x-faster-than-apache-spark-9561f13b00ae https://towardsdatascience.com/random-forest-on-gpus-2000x-f... Disclaimer: We produced those benchmark. I'm a founder of Saturn (https://www.saturncloud.io/ https://www.saturncloud.io/) and we focus on providing Databricks-like capabilities with Jupyter + Dask + Prefect, so I definitely have strong feelings in this area.

pletnes 6y ago

Just for clarification, dask lets you run any python function on any python datatype. Numpy/pandas is faster, but anything is doable with dask. This makes it eminently flexible, unlike many «big data» tools that only work on tables, or other arbitrary limitations.

peterthehacker 6y ago
Yes, you can use dask’s lower level APIs, like futures [0] or dask delayed, and it’ll just pickle the python objects. Dask provides a group of collections APIs, like dataframes [1] and arrays that use more efficient serialization methods. [0]https://docs.dask.org/en/latest/futures.html https://docs.dask.org/en/latest/futures.html [1]https://docs.dask.org/en/latest/dataframe.html https://docs.dask.org/en/latest/dataframe.html