5 ms·

> not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) As someone who's

> not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) As someone who's in the midst of a plan to migrate off of Apache Spark, because we've discovered that, for our particular purposes, a single-machine implementation in a language with better crunching and multithreading chops than Java, can easily achieve comparable performance to a Spark cluster, while being good deal easier to develop and maintain, this comment strikes me as being very poignant. I have come to suspect that the big data ecosystem is a castle that was largely built on a nice, thick technical foundation that is largely composed of pointer chasing.

solidasparagus 6y ago
I think the problem is that people don't really appreciate how big your data has to be to be Big Data.
- JulianMorrison 6y ago
  To be fair that threshold has risen substantially since the paper out of Google that kicked the whole thing off in 2003.
- mumblemumble 6y ago
  Well, there is that. But, rewind 10 or 15 years to when Big Data was really taking off, and the world was also different in ways that meant that your data didn't need to be nearly so big, even after adjusting for hardware advances, to qualify as Big. Most the programming platforms and libraries that one might choose instead of Hadoop & friends didn't exist back then, or weren't or weren't particularly robust. So there weren't a lot of games other than Java in town. And Java (idiomatic Java, not benchmark-friendly Java) tends to reduce the volume of work you can squeeze out of a given hardware setup by introducing a hefty does of memory overhead.
JulianMorrison 6y ago
Big Data was initially a solution to a problem that basically nobody had except Google. It has remained such.
- shusson 6y ago
  > It has remained such. You should take a look at genomic data. The volume of data is one of the main challenges[1]. [1] https://spectrum.ieee.org/computing/software/the-desperate-quest-for-genomic-compression-algorithms https://spectrum.ieee.org/computing/software/the-desperate-q...
xiaodai 6y ago
Omg you are so right! See diskframe.com a single machine focused framework
nautilus12 6y ago
I often find data scientist (from python persuasion) come into engineering organizations with this mindset and undo alot of progress. They end up realizing that engineering really does have value and isn't just engineering for engineering's sake and end up layering on a thousand hacks that replicate what was already there. Simplicity for simplicities sake is just as dangerous as complexity for complexities sake. There is a reason major enterprise organizations (LinkedIn, Facebook, etc) have java backends. Also it's not a matter of misappropriation. Some of what they do does apply outside the enterprise. You see a ton of python libraries make this mistake, like dask by saying "it's just pandas that runs on multiple machines". "It's just" is usually a sign that there is some complexity they are ignoring or don't have (that most others will)
smabie 6y ago
Yeah, just look at kdb+/q performance compared to Spark. The former is orders of magnitude more efficient and you realize that a cluster is only necessary because the performance and memory usage of the JVM is very poor on numerical workloads. Same goes for Hadoop. The amount of money and engineering resources thrown at a problem that actually has a pretty simple solution (integrate your db and language, make your program and language runtime fit in L1 i-cache, optimize the hell out of your language) is a little disheartening. But hey, it keeps people employed managing and debugging an unholy mess of DBs, K8 clusters, load balancers, KV stores, and message brokers..
- AzzieElbab 6y ago
  Sure, but there are 12 people in the world who know how to use k and q efficiently, and there is no fail recovery or any way to deal with data that actually does not fit in ram
  - snicker7 6y ago
    That is not true. RAM is cheap (terabytes per node is not uncommon). Moreover, a common pattern for kdb+ databases is to have two parts: a real-time database (rdb) that is typically memory backed and a historical database (hdb) that is serialized to disk (mmap'd for near-RAM speed). In practice, most data fetching is sequential (cache friendly), so you get good performance regardless. kdb+ is commonly used in the financial industry, where data volumes are large. The language itself is extremely minimal, so the learning curve is actually much lower than most languages (it forces you to write in a "vectorized" style, which is not too different than how numpy/R/matlab works). I was able to get up to speed in a couple of days. Source: Used to work with kdb+ (still do sometimes). Not a shill (I don't even like k/j/q/APL).
    - AzzieElbab 6y ago
      I am kdb user, I am not arguing against it. More power to you if you can afford to endlessly scale kdb vertically. Rest of the world have to deal with distributed systems, because people do not want to pay for kdb or the machines to run it on
lmm 6y ago
Spark runs pretty nicely in single-machine mode, and it forces you to structure your logic as a nice clean map-reduce. So I think it's fine to use it for "small" data.
FridgeSeal 6y ago
Have you seen Frank McSherry's COST paper? (and follow ups) http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... Really fun reading on exactly this issue - definitely changed my view on Spark and compute clusters. People definitely don't realise just how powerful a modern computer is, and how far you can get on a single one.
Abishek_Muthian 6y ago
Distributed computing for data science not only benefits when we go big, but also when we need go small with independent nodes. e.g. Cluster of scalable ARM SBC nodes or even multi-architectural nodes could be highly efficient. I've been exploring this kind of setup for a while now, frameworks like Dask, Ray[1], Modin can do this with Python to some extent. But they are still finicky and Dask seemed more stable than other frameworks for this setup. I wanted to try out language level distributed computing setup, but Julia required same setup (OS/Arch/Path) replicated on all their nodes last time I visited it. I feel, distributed computing as such hasn't got much love as it deserves in the consumer market, especially since many have several computing units in their houses now(PC, tablet, smartphone, Watch, TV, Game console). If interoperable distributed computing layer was fundamentally baked in with all modern operating systems minimising the latency with Network, Storage, Memory; Then we could leverage huge compute power on demand, which not possible without investing lot money in single compute unit nowadays. Then again, Compute power is a major strategic advantage for the manufacturers. Why would Apple share its A13X for compute with a Snapdragon or Intel? [1]https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b4b8cc4 https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b...
mark_l_watson 6y ago
Good point. I used Spark and DataFrames at my last job just because the data I needed was conveniently available in that environment. Using a high-memory single server seems better on general principles, until your data won't fit.
oxinabox 6y ago
I call this the: "large-but-not-really-all-that-big data" For if you can fit it in RAM on a high-end machine, and on a single harddrive in a low-end machine. and its practically a more useful case. Since if data is truely big many algorthms can't be used (since if it is O(n^2) its not practical).

solidasparagus 6y ago

I think the problem is that people don't really appreciate how big your data has to be to be Big Data.

JulianMorrison 6y ago
To be fair that threshold has risen substantially since the paper out of Google that kicked the whole thing off in 2003.
mumblemumble 6y ago
Well, there is that. But, rewind 10 or 15 years to when Big Data was really taking off, and the world was also different in ways that meant that your data didn't need to be nearly so big, even after adjusting for hardware advances, to qualify as Big. Most the programming platforms and libraries that one might choose instead of Hadoop & friends didn't exist back then, or weren't or weren't particularly robust. So there weren't a lot of games other than Java in town. And Java (idiomatic Java, not benchmark-friendly Java) tends to reduce the volume of work you can squeeze out of a given hardware setup by introducing a hefty does of memory overhead.

JulianMorrison 6y ago

Big Data was initially a solution to a problem that basically nobody had except Google. It has remained such.

shusson 6y ago
> It has remained such. You should take a look at genomic data. The volume of data is one of the main challenges[1]. [1] https://spectrum.ieee.org/computing/software/the-desperate-quest-for-genomic-compression-algorithms https://spectrum.ieee.org/computing/software/the-desperate-q...

xiaodai 6y ago

Omg you are so right! See diskframe.com a single machine focused framework

nautilus12 6y ago

I often find data scientist (from python persuasion) come into engineering organizations with this mindset and undo alot of progress. They end up realizing that engineering really does have value and isn't just engineering for engineering's sake and end up layering on a thousand hacks that replicate what was already there. Simplicity for simplicities sake is just as dangerous as complexity for complexities sake. There is a reason major enterprise organizations (LinkedIn, Facebook, etc) have java backends. Also it's not a matter of misappropriation. Some of what they do does apply outside the enterprise. You see a ton of python libraries make this mistake, like dask by saying "it's just pandas that runs on multiple machines". "It's just" is usually a sign that there is some complexity they are ignoring or don't have (that most others will)

smabie 6y ago

Yeah, just look at kdb+/q performance compared to Spark. The former is orders of magnitude more efficient and you realize that a cluster is only necessary because the performance and memory usage of the JVM is very poor on numerical workloads. Same goes for Hadoop. The amount of money and engineering resources thrown at a problem that actually has a pretty simple solution (integrate your db and language, make your program and language runtime fit in L1 i-cache, optimize the hell out of your language) is a little disheartening. But hey, it keeps people employed managing and debugging an unholy mess of DBs, K8 clusters, load balancers, KV stores, and message brokers..

AzzieElbab 6y ago
Sure, but there are 12 people in the world who know how to use k and q efficiently, and there is no fail recovery or any way to deal with data that actually does not fit in ram
- snicker7 6y ago
  That is not true. RAM is cheap (terabytes per node is not uncommon). Moreover, a common pattern for kdb+ databases is to have two parts: a real-time database (rdb) that is typically memory backed and a historical database (hdb) that is serialized to disk (mmap'd for near-RAM speed). In practice, most data fetching is sequential (cache friendly), so you get good performance regardless. kdb+ is commonly used in the financial industry, where data volumes are large. The language itself is extremely minimal, so the learning curve is actually much lower than most languages (it forces you to write in a "vectorized" style, which is not too different than how numpy/R/matlab works). I was able to get up to speed in a couple of days. Source: Used to work with kdb+ (still do sometimes). Not a shill (I don't even like k/j/q/APL).
  - AzzieElbab 6y ago
    I am kdb user, I am not arguing against it. More power to you if you can afford to endlessly scale kdb vertically. Rest of the world have to deal with distributed systems, because people do not want to pay for kdb or the machines to run it on

lmm 6y ago

Spark runs pretty nicely in single-machine mode, and it forces you to structure your logic as a nice clean map-reduce. So I think it's fine to use it for "small" data.

FridgeSeal 6y ago

Have you seen Frank McSherry's COST paper? (and follow ups) http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... Really fun reading on exactly this issue - definitely changed my view on Spark and compute clusters. People definitely don't realise just how powerful a modern computer is, and how far you can get on a single one.

Abishek_Muthian 6y ago

Distributed computing for data science not only benefits when we go big, but also when we need go small with independent nodes. e.g. Cluster of scalable ARM SBC nodes or even multi-architectural nodes could be highly efficient. I've been exploring this kind of setup for a while now, frameworks like Dask, Ray[1], Modin can do this with Python to some extent. But they are still finicky and Dask seemed more stable than other frameworks for this setup. I wanted to try out language level distributed computing setup, but Julia required same setup (OS/Arch/Path) replicated on all their nodes last time I visited it. I feel, distributed computing as such hasn't got much love as it deserves in the consumer market, especially since many have several computing units in their houses now(PC, tablet, smartphone, Watch, TV, Game console). If interoperable distributed computing layer was fundamentally baked in with all modern operating systems minimising the latency with Network, Storage, Memory; Then we could leverage huge compute power on demand, which not possible without investing lot money in single compute unit nowadays. Then again, Compute power is a major strategic advantage for the manufacturers. Why would Apple share its A13X for compute with a Snapdragon or Intel? [1]https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b4b8cc4 https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b...

mark_l_watson 6y ago

Good point. I used Spark and DataFrames at my last job just because the data I needed was conveniently available in that environment. Using a high-memory single server seems better on general principles, until your data won't fit.

oxinabox 6y ago

I call this the: "large-but-not-really-all-that-big data" For if you can fit it in RAM on a high-end machine, and on a single harddrive in a low-end machine. and its practically a more useful case. Since if data is truely big many algorthms can't be used (since if it is O(n^2) its not practical).