4 ms·

Anecdote. Last year I had to work with a heavy analytics process. The whole thing was 4 or 5 large steps and was written with PySpark. It was really slow and me

Anecdote. Last year I had to work with a heavy analytics process. The whole thing was 4 or 5 large steps and was written with PySpark. It was really slow and memory on my system run quite low (on a 8Gb system with a generous swap), sometimes even stopping the whole processing of the pipeline. For one heavy step we tried out DuckDB and I was blown away how performant against PySpark was. It was not only fast as hell but its memory footprint extremely low as well, almost as if something was wrong and had to recheck several times that it was correct, and yes it did what it was supposed to do. Now this is a place where I do actually care about how fast and performant a thing can be and not the nanoseconds that each JS frontend framework of the day claims to win. KUDOS to the DuckDB team.

Marazan 2y ago
Spark is never going to be the right choice when running on a single system. Spark is for when you have a hundreds of machines worth of processing to do
- moooo99 2y ago
  > Spark is for when you have a hundreds of machines worth of processing to do Absolutely agree. However, most uses of Spark I've seen in my career are people thinking they have hundreds of machines worth of processing to do.
  - Marazan 2y ago
    And even when you jave quite a lot of machines worth of processing some single threaded streaming of data on a single machine can still beat out any distributed framework as the the overhead of distribution is large.
    - 0cf8612b2e1e 2y ago
      A favorite paper, “Scalability! But at what COST?”. Authors show a single machine implementation (even single threaded) can wipe the floor with the maximum parallel capable implementation. http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/ http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/
- liendolucas 2y ago
  We haven't developed the PySpark pipeline. It was given to us to be improved, which we did a whole rewrite to leave it more clean and understandable. We also tried a persistence switch to test if it was a better choice just in case a step failed we could resume from a prevoius one. I also had zero hands-on on PySpark and DuckDB. But yes, I was amazed at how far it was falling behind DuckDB. I wasn't expecting such a difference. Ah also this pipeline did indeed run on the cloud, but it was not posible to test it there, so the only choice was to run it locally.
- aradox66 2y ago
  Motherduck has an excellent article about this: https://motherduck.com/blog/the-simple-joys-of-scaling-up/ https://motherduck.com/blog/the-simple-joys-of-scaling-up/
mritchie712 2y ago
simple example along these lines running on AWS (specifically paired with Iceberg): https://www.definite.app/blog/cloud-iceberg-duckdb-aws https://www.definite.app/blog/cloud-iceberg-duckdb-aws

Marazan 2y ago

Spark is never going to be the right choice when running on a single system. Spark is for when you have a hundreds of machines worth of processing to do

moooo99 2y ago
> Spark is for when you have a hundreds of machines worth of processing to do Absolutely agree. However, most uses of Spark I've seen in my career are people thinking they have hundreds of machines worth of processing to do.
- Marazan 2y ago
  And even when you jave quite a lot of machines worth of processing some single threaded streaming of data on a single machine can still beat out any distributed framework as the the overhead of distribution is large.
  - 0cf8612b2e1e 2y ago
    A favorite paper, “Scalability! But at what COST?”. Authors show a single machine implementation (even single threaded) can wipe the floor with the maximum parallel capable implementation. http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/ http://dsrg.pdos.csail.mit.edu/2016/06/26/scalability-cost/
liendolucas 2y ago
We haven't developed the PySpark pipeline. It was given to us to be improved, which we did a whole rewrite to leave it more clean and understandable. We also tried a persistence switch to test if it was a better choice just in case a step failed we could resume from a prevoius one. I also had zero hands-on on PySpark and DuckDB. But yes, I was amazed at how far it was falling behind DuckDB. I wasn't expecting such a difference. Ah also this pipeline did indeed run on the cloud, but it was not posible to test it there, so the only choice was to run it locally.
aradox66 2y ago
Motherduck has an excellent article about this: https://motherduck.com/blog/the-simple-joys-of-scaling-up/ https://motherduck.com/blog/the-simple-joys-of-scaling-up/

mritchie712 2y ago

simple example along these lines running on AWS (specifically paired with Iceberg): https://www.definite.app/blog/cloud-iceberg-duckdb-aws https://www.definite.app/blog/cloud-iceberg-duckdb-aws