4 ms·

Out of interest, is everyone using Spark/Hadoop/etc because you need to and/or you chose to or another reason? IMO, legit use cases seem to me relatively sparse

• syllogism 10y ago
  Let's say you're developing your analysis script, and it takes 45 minutes to run. That doesn't sound like much of a problem, but it means you only really get to try 10 things a day. You can kick something off and then start trying something different, but that's actually quite a burden. It's cognitively easier to run a series of parallel tasks instead of parallel serial tasks. Even if a job is taking 15-20 minutes, that can really wear you down, and give you a pretty unproductive week. It's hard to keep your place. That said: I think often these 45 minute tasks should be much faster, and folks would actually benefit from optimising their code a bit more. If I have to do a word count in 30gb of text, I'd rather implement carefully or use Hyper Log Log. I find it weird that people would rather use Spark, which I always find quite painful. I guess it comes down to familiarity.
• eshvk 10y ago
  I run models on large scale user data (for recommendations). A couple of hundred gigs is what "cleaned" data looks like. This might need to be joined with metadata. This cleaning, joining is easier to do on Map Reduce than wait a few days on a beefy computational machine.