3 ms·
Spark is massive overkill for the vast majority of needs. Can't fit the data on a computer, like your use case? Yep, good for that. Anything else, and it's n
by JPKab 5y ago
Spark is massive overkill for the vast majority of needs. Can't fit the data on a computer, like your use case? Yep, good for that.
Anything else, and it's not worth the overhead. Dask doesn't require a JVM cluster with executors and name nodes and hdfs and everything else that keeps databricks in business.
- jp0d 5y agoyeah, like others have pointed out, you might be correct about it being a massive overkill for most users. It takes me at least 10 to 15 minutes to spin up a cluster and configure it before I can start anything.