2 ms·
I use Spark at work, it's really good when you need to do some large scale analysis with your own custom code. Everything else it does is just ok. It certainly
by Eridrus 10y ago
I use Spark at work, it's really good when you need to do some large scale analysis with your own custom code. Everything else it does is just ok. It certainly doesn't subsume ML tools.
- thr0waway1239 10y agoGreat to hear from someone who uses Spark at work. I see what you mean - subsumed is a bad word choice and I should not have tried to paraphrase. What do you feel about the phrase that the presenter uses - "One tool that fits everything"? (I am new to data science)
- Eridrus 10y agoI think it's marketing nonsense. Databricks in particular has good "proximity to data" IMO, in that once you've figured out how to use it with your data sources, you can just fire up a web browser and connect to them, and connecting those data sources to your code is easy. The problem with seeing Spark as your one tool for everything is that's only true if it's trivial to integrate your code with Spark. Viz tools like Plotly/Bokeh don't integrate well with Databricks' notebook, Deep Learning tools are not really supported yet unless you're running your own clusters and running special libraries to wire thing together. I think Spark is a good workhorse for big data; it can do repetitive things well at large scale, it's less good when you want to use more niche tools since most of the Data Science community is not focussed on Spark. PySpark exists and will probably be good enough, but only if your data fits into memory in a single machine anyway. And if you're not dealing with big data, Spark is overkill. It's usually simpler to just get a box with more RAM.
- nl 10y agoSpark's MLLib is pretty weak compared to other options.