3 ms·
You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the ch
by llarsson 7y ago
You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice.
Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.
- threeseed 7y agoMapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc. Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.
- wbl 7y agoBack in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!
- jbergens 7y agoSome places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.
- threeseed 7y agoVery curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago. You've seen people writing pages and pages of Java code to do ETL ?
- doteka 7y agoI worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions. The rest of the world does not particularly care about SV fashion trends.
- artimaeis 7y agoI'm on a team of a Fortune500 who uses a combination of Java and C# to do ETL. AMA. Edit: We're also a fairly greenfield team, with a product in beta that's less than 4 years old. So like, the leaders of the team knowingly crafted us in this particular direction.
- vonmoltke 7y agoI'm very curious to know where you got this strange notion that MapReduce is "a technology/paradigm that stopped being used a decade ago". I barely knew what MapReduce was in 2010, and didn't touch my first Hadoop cluster until late 2012. Every place I have worked since has had a Hadoop cluster or two.
- commandlinefan 7y ago> in favour of Hive, Spark Well, Hive and Spark create MapReduce jobs to satisfy your queries: it's still in the background, but you don't have to think about it (as much).