3 ms·
I am a consultant working on Hadoop installations across the globe. As an average I usually able to save 70% disk usage and 30% overall cost by changing the def
by StreamBright 8y ago
I am a consultant working on Hadoop installations across the globe. As an average I usually able to save 70% disk usage and 30% overall cost by changing the defaults to a reasonable value as well as migrating companies out of HDFS to something like S3. I have spent majority of my career (10 years) working on Hadoop and I can tell you that it is a terrible piece of software with insane ineffciency all over the place. If you would switch over all the Hadoop installations on Earth at once to something more reasonable it would be visible on the global CO2 production chart quite a bit. What is good for the bottom 90% of companies is a financial question not a technological question. It is a bit unfurtunate that Hadoop is so popular and nobody cares about efficiency, not even the Hadoop vendors (maybe with the exception of MapR, which is not opensource).
- snaky 8y ago> This find | xargs mawk | mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- StreamBright 8y agoWell this is great until you need more nodes. :) I am talking about the same scalability while maintaining a much lower ecological and financial footprint.
- Zariel 8y agoUsing hadoop/spark for <2gb of data seems like a terrible idea. When all you have is a hammer everything starts to look like a nail.
- alexhutcheson 8y agoIs there a good open source alternative that meets the HDFS use-case (i.e. file or blob storage, rather than a KV store designed for point lookups)? Or is tuning the HDFS defaults the best you can do without migrating onto someone's cloud platform?