4 ms·

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smart about memory consumption. It would have taken an enormous and expensive Spark cluster.

manigandham 8y ago
We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.
- riku_iki 8y ago
  And spark also has some other goodies: integration with storages, monitoring, fault tolerance, common algorithms implemented.
bunderbunder 8y ago
This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1] There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided. [1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)
- lenticular 8y ago
  That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.
- mikhailfranco 8y ago
  See McSherry et al, "Scalability! But at what COST?" https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
deepsun 8y ago
I've seen the opposite in our company. We did some data science, on a data that wasn't that big, but big enough to not fit into one machine. Everyone was reluctant to move to Spark, so data-scientists computed their models over subset of data: one day instead of one week of data. And after the project finished, and passed to client, they realized that subset was not representative enough to meet the criteria. Of course, it's kinda their fault to assume that subset is representative, but if you start checking every assumption you make in data science, then you won't get far. Had they used Spark from the beginning, they would not have that last-moment surprise, because they would work on proper sized dataset from the beginning.
- deleted 8y ago
  [deleted]
- bunderbunder 8y ago
  Well, two things there - First, the case you describe is one that's covered by using Spark for batch processing. Parent was criticizing using it and other tools for stream processing. The two are very different use cases. Second, just gotta call out that 2nd paragraph. A qualified data scientist should have a solid training in statistics. And someone who has a solid training in statistics should rarely if ever make the assumption that one day is representative of the whole week in the first place, regardless of whether they subsequently check that assumption. They would probably start with the presumption that one day is not representative of the whole week, because that is self-evidently going to be near-universally the case for any data that measures something about human behavior.
  - deepsun 8y ago
    Right, for batch models and training evaluation. Re. assumptions -- it all depends on the data. If your data, say, collection of bird songs on Galapagos, then assuming there's correlation between day-of-week is counter productive. If a data scientist says me that they need to spend a day or two checking that assumption is correct, then I would look for a more productive scientist. Time to market is critical. And there are many of such assumptions in every project, just try to do something, anything with data science checking every assumption you make. Basically as I said above, you won't get far. By the way, majority of statistics theorems start with "take N independently distributed variables". In real world nothing is independent, but we still make useful predictions.

manigandham 8y ago

We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.

riku_iki 8y ago
And spark also has some other goodies: integration with storages, monitoring, fault tolerance, common algorithms implemented.

bunderbunder 8y ago

This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1] There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided. [1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)

lenticular 8y ago
That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.
mikhailfranco 8y ago
See McSherry et al, "Scalability! But at what COST?" https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...

deepsun 8y ago

I've seen the opposite in our company. We did some data science, on a data that wasn't that big, but big enough to not fit into one machine. Everyone was reluctant to move to Spark, so data-scientists computed their models over subset of data: one day instead of one week of data. And after the project finished, and passed to client, they realized that subset was not representative enough to meet the criteria. Of course, it's kinda their fault to assume that subset is representative, but if you start checking every assumption you make in data science, then you won't get far. Had they used Spark from the beginning, they would not have that last-moment surprise, because they would work on proper sized dataset from the beginning.

deleted 8y ago
[deleted]
bunderbunder 8y ago
Well, two things there - First, the case you describe is one that's covered by using Spark for batch processing. Parent was criticizing using it and other tools for stream processing. The two are very different use cases. Second, just gotta call out that 2nd paragraph. A qualified data scientist should have a solid training in statistics. And someone who has a solid training in statistics should rarely if ever make the assumption that one day is representative of the whole week in the first place, regardless of whether they subsequently check that assumption. They would probably start with the presumption that one day is not representative of the whole week, because that is self-evidently going to be near-universally the case for any data that measures something about human behavior.
- deepsun 8y ago
  Right, for batch models and training evaluation. Re. assumptions -- it all depends on the data. If your data, say, collection of bird songs on Galapagos, then assuming there's correlation between day-of-week is counter productive. If a data scientist says me that they need to spend a day or two checking that assumption is correct, then I would look for a more productive scientist. Time to market is critical. And there are many of such assumptions in every project, just try to do something, anything with data science checking every assumption you make. Basically as I said above, you won't get far. By the way, majority of statistics theorems start with "take N independently distributed variables". In real world nothing is independent, but we still make useful predictions.