3 ms·
Hadoop and then Yarn+Mapreduce gave us quite a lot of value even until 2017. I honestly don't think it was a bad technology choice. Should have moved off it fas
by renewiltord 3y ago
Hadoop and then Yarn+Mapreduce gave us quite a lot of value even until 2017. I honestly don't think it was a bad technology choice. Should have moved off it faster but we had cheap commodity hardware running cheap software and early Spark was massively memory-finicky (cliff of performance). I wouldn't use that tech today but back in 2008-2014 it let us run things over a few petabytes relatively cheaply with relatively slow interconnect.
We had an impl of the Pregel paper on top of the Yarn manager.
The API was painful and easy to err on but it did provide quite a bit of functionality.
Now, of course, that stuff is all out of date. Where I am now we have custom job engine and it's way better. I imagine others have something like this too.
Things have just changed. Interconnect is now cheap and fast: 40 Gbps is commodity.