4 ms·
> In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill. I like that you said this and I've seen
by jeremy_k 7y ago
> In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill.
I like that you said this and I've seen a similar sentiment before. I was first exposed to it watching an ElixirConf keynote from José Valim[1] about the Flow framework they built in Elixir where he summarized it as "For between 40-80% of the jobs submitted to MapReduce systems, you'd be better off running them on a single machine." which referenced the paper Musketeer: all for one, one for all in data processing systems [2].
While I'm no Data Engineer myself, I do often wonder if distributing the workload is always better? The anecdote above indicates that a powerful single multicore machine may be right solution for many.
Now that isn't to discount what Prophecy is trying to do, my company just went through a huge re-platforming moving from on on premises to the cloud and it is not easy; any company trying to tackle that space is on the right track. But I just wonder if its overkill for most use cases?
[1] - https://www.youtube.com/watch?v=srtMWzyqdp8 https://www.youtube.com/watch?v=srtMWzyqdp8
[2] - http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketeer.pdf http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketee...