4 ms·

This is just a map/reduce problem. Use Hadoop. It's Java, isn't it?

• deleted 3y ago
  [deleted]
• kriskrunch 3y ago
  > No external dependencies may be used
• deleted 3y ago
  [deleted]
• baq 3y ago
  Why would I use Hadoop for such a small number of rows…?
  • jalino23 3y ago
    1 billion is small for hadoop?
    • rapsey 3y ago
      If it fits on one computer it's not a hadoop problem.
      • quickthrower2 3y ago
        It fits on a dusty ten year old USB stick
      • badgersnake 3y ago
        Sounds like an awk problem tbh.
      • 8organicbits 3y ago
        A Hadoop submission may help people realize that. But since you only have one machine to work with it should be obvious that you're not going to get any speed-up via divide and conquer.
    • brokensegue 3y ago
      ~14GB file? it's on the small side for hadoop
    • chmod775 3y ago
      Anything that fits in RAM on one machine is easily too small for Hadoop. In those cases, the overhead of Hadoop is going to make it get destroyed by a single beefy machine. The only times where this might not be the case is when you're doing a crazy amount of computation relative to the data you have. Note that you can easily reach 1TB of RAM on (enterprise) commodity hardware now, and SSDs are pretty fast too. Old but gold post from 2014: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
      • makapuf 3y ago
        Also from 2013: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html