3 ms·
The picture has now gotten a little fuzzier as this blog post conflates map reduce and YARN and calls them both hadoop. The scala pseudo code is just about exac
by placeybordeaux 10y ago
The picture has now gotten a little fuzzier as this blog post conflates map reduce and YARN and calls them both hadoop. The scala pseudo code is just about exactly what you'd use with spark which runs on YARN.
- edw 10y agoI think his point is that bloated, over-engineered Big Data systems—whether batch or streaming—are overkill for the vast majority of problems.
- placeybordeaux 10y agoThere are just many points that don't really apply to stuff like spark or tez that runs on YARN: ex: Hadoop << SQL, Python Scripts I completely agree with Mapreduce << SQL, Python Scripts I do a lot of my processing on sparkSQL and through RDD transformations as opposed to Mapreduce limiting, slow KV style processing.