3 ms·

Apologies for being pessimistic. I've never had a reason to use dask. I like Pandas only for very small sample datasets. Nothing more than a few MBs. I deal wit

by jp0d 5y ago

Apologies for being pessimistic. I've never had a reason to use dask. I like Pandas only for very small sample datasets. Nothing more than a few MBs. I deal with large amount of alpha-numeric data that can no way fit on a single computer. Even with multi-processing abilities, it's never really that useful and pandas does the job. I've to use large clusters and spark for any real datasets before I even think about using pandas. Spark has its quirks but I'd argue that it has a lot more features when it comes to distributed computing.

nerdponx 5y ago
Data in the 1-15 GB range is more common than you'd think. Spark is overkill and comparatively a nightmare to configure. You can fit 15 GB in memory nowadays on a single core, but you probably want things to move a little quicker.
- jp0d 5y ago
  You're definitely correct about Spark being a nightmare to configure. Luckily we've got data platform teams who take care of that. Analysts like I still do configure parameters depending on the scenario. You might be correct about data in the 1-15 GB being more common. I'm an analyst so most of the time, I've to aggregate data before I do any analysis. But data scientists in my team typically work on smaller datasets and use Sagemaker instances.
- physicsguy 5y ago
  I work at a University HPC centre and our new 72 core Icelake nodes have 512gb of RAM, and even our older 40 core Cascade Lake nodes had 256gb as standard, so I always scratch my head at who these libraries serve. That's pretty standard hardware for a HPC cluster, which is where people doing this stuff should be working anyway.
  - jp0d 5y ago
    Off the top of my head, I can't remember what size nodes we use. I think it's m5x large AWS nodes with 16 cores and 128 GB RAM. Depending on the size of the data I use anywhere between 20 to 80 of these nodes. Sometimes even higher. That's why I've never had a need for these libraries.
  - nerdponx 5y ago
    I wasn't thinking about HPC at all. I more had in mind "single-node" use cases like a workstation PC. But even in a small cluster type of use case, I think the main users are scientists or other practitioners who, for whatever reason, don't have the support of a platform team or data engineering team that can keep a Spark (or other) cluster fed and cared for.
  - higeorge13 5y ago
    I have seen european companies (claiming) doing big data on 16gb, 4-cpu, 15 node clusters and claiming that their spark jobd were not failing due to 90s hardware. I will send them your configuration. :)
cowsandmilk 5y ago
I don't think that's being pessimistic. I think it reflects dask's documentation. Some items I looked up because I remember from the past: > If your data fits comfortably in RAM and you are not performance bound, then using NumPy might be the right choice. Dask adds another layer of complexity which may get in the way. from https://docs.dask.org/en/latest/array-best-practices.html#use-numpy https://docs.dask.org/en/latest/array-best-practices.html#us... > In many workloads it is common to use Dask to read in a large amount of data, reduce it down, and then iterate on a much smaller amount of data. For this latter stage on smaller data it may make sense to stop using Dask, and start using normal Python again. https://docs.dask.org/en/latest/best-practices.html#stop-using-dask-when-no-longer-needed https://docs.dask.org/en/latest/best-practices.html#stop-usi... So, dask even says not to use it for small data sets. And in their comparison to spark, they say spark has a lot more features.
- jp0d 5y ago
  ohh yeah. Thanks for pointing that out. I do remember reading this a few months ago.
JPKab 5y ago
Spark is massive overkill for the vast majority of needs. Can't fit the data on a computer, like your use case? Yep, good for that. Anything else, and it's not worth the overhead. Dask doesn't require a JVM cluster with executors and name nodes and hdfs and everything else that keeps databricks in business.
- jp0d 5y ago
  yeah, like others have pointed out, you might be correct about it being a massive overkill for most users. It takes me at least 10 to 15 minutes to spin up a cluster and configure it before I can start anything.

by jp0d 5y ago

nerdponx 5y ago

Data in the 1-15 GB range is more common than you'd think. Spark is overkill and comparatively a nightmare to configure. You can fit 15 GB in memory nowadays on a single core, but you probably want things to move a little quicker.

jp0d 5y ago
You're definitely correct about Spark being a nightmare to configure. Luckily we've got data platform teams who take care of that. Analysts like I still do configure parameters depending on the scenario. You might be correct about data in the 1-15 GB being more common. I'm an analyst so most of the time, I've to aggregate data before I do any analysis. But data scientists in my team typically work on smaller datasets and use Sagemaker instances.
physicsguy 5y ago
I work at a University HPC centre and our new 72 core Icelake nodes have 512gb of RAM, and even our older 40 core Cascade Lake nodes had 256gb as standard, so I always scratch my head at who these libraries serve. That's pretty standard hardware for a HPC cluster, which is where people doing this stuff should be working anyway.
- jp0d 5y ago
  Off the top of my head, I can't remember what size nodes we use. I think it's m5x large AWS nodes with 16 cores and 128 GB RAM. Depending on the size of the data I use anywhere between 20 to 80 of these nodes. Sometimes even higher. That's why I've never had a need for these libraries.
- nerdponx 5y ago
  I wasn't thinking about HPC at all. I more had in mind "single-node" use cases like a workstation PC. But even in a small cluster type of use case, I think the main users are scientists or other practitioners who, for whatever reason, don't have the support of a platform team or data engineering team that can keep a Spark (or other) cluster fed and cared for.
- higeorge13 5y ago
  I have seen european companies (claiming) doing big data on 16gb, 4-cpu, 15 node clusters and claiming that their spark jobd were not failing due to 90s hardware. I will send them your configuration. :)

cowsandmilk 5y ago

I don't think that's being pessimistic. I think it reflects dask's documentation. Some items I looked up because I remember from the past: > If your data fits comfortably in RAM and you are not performance bound, then using NumPy might be the right choice. Dask adds another layer of complexity which may get in the way. from https://docs.dask.org/en/latest/array-best-practices.html#use-numpy https://docs.dask.org/en/latest/array-best-practices.html#us... > In many workloads it is common to use Dask to read in a large amount of data, reduce it down, and then iterate on a much smaller amount of data. For this latter stage on smaller data it may make sense to stop using Dask, and start using normal Python again. https://docs.dask.org/en/latest/best-practices.html#stop-using-dask-when-no-longer-needed https://docs.dask.org/en/latest/best-practices.html#stop-usi... So, dask even says not to use it for small data sets. And in their comparison to spark, they say spark has a lot more features.

jp0d 5y ago
ohh yeah. Thanks for pointing that out. I do remember reading this a few months ago.

JPKab 5y ago

Spark is massive overkill for the vast majority of needs. Can't fit the data on a computer, like your use case? Yep, good for that. Anything else, and it's not worth the overhead. Dask doesn't require a JVM cluster with executors and name nodes and hdfs and everything else that keeps databricks in business.

jp0d 5y ago
yeah, like others have pointed out, you might be correct about it being a massive overkill for most users. It takes me at least 10 to 15 minutes to spin up a cluster and configure it before I can start anything.