3 ms·
28GB is really too small for Hadoop to get out of bed for, in general. Though I would wonder why it was _that_ slow with Spark (or Hadoop, for that matter).
by rsynnott 9y ago
28GB is really too small for Hadoop to get out of bed for, in general. Though I would wonder why it was _that_ slow with Spark (or Hadoop, for that matter).
- sfifs 9y agoBecause it was running on a laptop instead of on a cluster :-)
- mr_toad 9y agoSpark is going to try and ingest all the data, and it won’t fit in RAM. Wrong tool for the job basically.
- rsynnott 9y agoDepends on exactly how you do it, I suppose, but it shouldn't necessarily. Most Hadoop-y work can also be accomplished in Spark without much fuss.