4 ms·

After doing a significant deep dive into Databricks as a third party solution for my team at work, we decided Databricks and a deep commitment to Spark was a ve

by mlthoughts2018 8y ago

After doing a significant deep dive into Databricks as a third party solution for my team at work, we decided Databricks and a deep commitment to Spark was a very poor choice for machine learning, though Spark seems generally fine as an interface for map reduce or scheduled cluster compute tasks. Spark is in 2019 what Hadoop was in ~2014. In 5-6 years Spark will be the cure-all basket that a bunch of people put all their eggs into not realizing the deep-seated limitations. This is especially true for machine learning.

cheriot 8y ago
What are the deep-seated limitations you see?
- mlthoughts2018 8y ago
  Rather than type it out again, I’ll link to a previous thread where I discussed a small bit of it, https://news.ycombinator.com/item?id=19321372 https://news.ycombinator.com/item?id=19321372
  - FridgeSeal 8y ago
    I cannot agree more with what you wrote in your linked comment. People dramatically underestimate how far you can get with even a single machine and some slightly better engineering. Best ever example of this I've seen is the super impressive work done by Frank McSherry: http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...
ganeshkrishnan 8y ago
Spark is big data processor. Use dl4j or tensor flow along with it for machine learning
- mlthoughts2018 8y ago
  Use of those tools (along with MLlib and virtually anything relying on a py4j bridge) was precisely the setup I tested and found to have unacceptably poor performance when taken across our range of both large and small workloads, in addition to many problems with deep inflexibility in controlling the runtime environment on a per-project basis (our most critical requirement). See my other comment below with a link to a previous discussion.

cheriot 8y ago

What are the deep-seated limitations you see?

mlthoughts2018 8y ago
Rather than type it out again, I’ll link to a previous thread where I discussed a small bit of it, https://news.ycombinator.com/item?id=19321372 https://news.ycombinator.com/item?id=19321372
- FridgeSeal 8y ago
  I cannot agree more with what you wrote in your linked comment. People dramatically underestimate how far you can get with even a single machine and some slightly better engineering. Best ever example of this I've seen is the super impressive work done by Frank McSherry: http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...

ganeshkrishnan 8y ago

Spark is big data processor. Use dl4j or tensor flow along with it for machine learning

mlthoughts2018 8y ago
Use of those tools (along with MLlib and virtually anything relying on a py4j bridge) was precisely the setup I tested and found to have unacceptably poor performance when taken across our range of both large and small workloads, in addition to many problems with deep inflexibility in controlling the runtime environment on a per-project basis (our most critical requirement). See my other comment below with a link to a previous discussion.