4 ms·
Curious why you went to Hadoop/Hive from Redshift? I usually hear of people going the other way. I would recommend staying away from Hadoop -- if you're a Jav
by mattbillenstein 9y ago
Curious why you went to Hadoop/Hive from Redshift? I usually hear of people going the other way.
I would recommend staying away from Hadoop -- if you're a Java shop, it may make some sense if you're writing your own map/reduce jobs, but it seems particularly brittle and hard to debug although my info is a bit dated at this point.
And generally if you must have Hadoop or Hive, the most elegant thing I've seen is EMR backed on S3 by .json.gz. You can spin up a cluster for a short time to run some jobs, and then spin it down when you're done -- hundreds of small instances seem to work well at this; but most of your interactive stuff should probably be in another system.
- vlahmot 9y agoWe do the EMR backed by s3 setup, only with snappy over gz as gz can't be split.
- mattbillenstein 9y agoAh, word, do you roll up the data by day? Or hour? I think in a situation where you roll it up by hour and you have a lot of files, it can be spread out pretty evenly on a large cluster.