4 ms·

Dask and especially the dashboard is a very nice piece of technology. But "naïve programming" (the art of writing a dumb algorithm with no regards to performan

by linkdd 5y ago

Dask and especially the dashboard is a very nice piece of technology. But "naïve programming" (the art of writing a dumb algorithm with no regards to performance or optimizations) can bite you hard: https://docs.dask.org/en/latest/best-practices.html#avoid-very-large-graphs https://docs.dask.org/en/latest/best-practices.html#avoid-ve... I implemented an algorithm that was slower than the non-parallelized version this way :D

tubby12345 5y ago
>But "naïve programming" (the art of writing a dumb algorithm with no regards to performance or optimizations) can bite you hard amen. people should learn to write fast code instead of "spraying and praying" https://users.ece.cmu.edu/~franzf/papers/gttse07.pdf https://users.ece.cmu.edu/~franzf/papers/gttse07.pdf
otherme123 5y ago
An anecdote that cannot be extrapolated: I did a course on using Dask a few years ago (Dask was fresh then). They teach us how Dask would improve our calculations, to the point of almost promissing a time improvement of "if you chop your dataframe in 8 partitions, your times will be divided by 8", no questions asked. Then they gave us a sample of a dataframe, to analyze firstly with Pandas, and then with Dask, and compare. The times not only didn't improve, or even equal Pandas: they got worse by more than 10x. The teachers looked the code with a mix of disbelief and horror, and only after that they started to talk about tradeoffs, overheads, data fitting in memory, etc.
- chaxor 5y ago
  Yeah it took my computer 10 minutes to save a 40 MB file on Spark today. Instantaneous on pandas.
  - disgruntledphd2 5y ago
    So, the trouble with Spark (as I suspect you're probably aware) is that a) it's distributed and b) it's lazy. This means that nothing happens until it needs to. So I have experienced .head() taking hours if there's a lot of upstream processing. Another reason for this could be that your file is distributed across many nodes, and thus you need lots of network requests to get the file. I'm not disagreeing with you that Spark can often have problems, but it's a really useful tool that can do things that are really hard to do elsewhere (like distributed ML).
    - chaxor 5y ago
      I'm pretty sure the problem was that it was fragmented across workers and such. I had already computed the filters and done a df.show() and count prior to trying save it. It's just a really unfriendly system for creating packages for everyone to run on a single node unfortunately.
      - disgruntledphd2 5y ago
        Oh yeah, there's stuff you can do, but again it takes tuning and ideally it shouldn't be so annoying. I definitely get the frustration, but waiting for sklearn for hours to fit a fraction of the observations that spark can do is also frustrating, in a different way. > run on a single node yeah, Spark is pretty crap here, but it's not really one of their main use cases, to be fair.

by linkdd 5y ago

tubby12345 5y ago

>But "naïve programming" (the art of writing a dumb algorithm with no regards to performance or optimizations) can bite you hard amen. people should learn to write fast code instead of "spraying and praying" https://users.ece.cmu.edu/~franzf/papers/gttse07.pdf https://users.ece.cmu.edu/~franzf/papers/gttse07.pdf

otherme123 5y ago

An anecdote that cannot be extrapolated: I did a course on using Dask a few years ago (Dask was fresh then). They teach us how Dask would improve our calculations, to the point of almost promissing a time improvement of "if you chop your dataframe in 8 partitions, your times will be divided by 8", no questions asked. Then they gave us a sample of a dataframe, to analyze firstly with Pandas, and then with Dask, and compare. The times not only didn't improve, or even equal Pandas: they got worse by more than 10x. The teachers looked the code with a mix of disbelief and horror, and only after that they started to talk about tradeoffs, overheads, data fitting in memory, etc.

chaxor 5y ago
Yeah it took my computer 10 minutes to save a 40 MB file on Spark today. Instantaneous on pandas.
- disgruntledphd2 5y ago
  So, the trouble with Spark (as I suspect you're probably aware) is that a) it's distributed and b) it's lazy. This means that nothing happens until it needs to. So I have experienced .head() taking hours if there's a lot of upstream processing. Another reason for this could be that your file is distributed across many nodes, and thus you need lots of network requests to get the file. I'm not disagreeing with you that Spark can often have problems, but it's a really useful tool that can do things that are really hard to do elsewhere (like distributed ML).
  - chaxor 5y ago
    I'm pretty sure the problem was that it was fragmented across workers and such. I had already computed the filters and done a df.show() and count prior to trying save it. It's just a really unfriendly system for creating packages for everyone to run on a single node unfortunately.
    - disgruntledphd2 5y ago
      Oh yeah, there's stuff you can do, but again it takes tuning and ideally it shouldn't be so annoying. I definitely get the frustration, but waiting for sklearn for hours to fit a fraction of the observations that spark can do is also frustrating, in a different way. > run on a single node yeah, Spark is pretty crap here, but it's not really one of their main use cases, to be fair.