3 ms·
Indeed. A rule of the thumb that I've inferred from many installations is: Just introducing Hadoop makes everything 10 times slower AND more expensive than an
by beagle3 13y ago
Indeed.
A rule of the thumb that I've inferred from many installations is: Just introducing Hadoop makes everything 10 times slower AND more expensive than an efficient tool set (e.g. pandas).
So it only makes sense to start hadooping when you are getting close to the limit of what you can pandas - everything you do before that is a horrible waste of resources.
And when you do get there - often, a slightly smarter distribution among servers and staying with e.g. pandas, will let you keep scaling up without introducing the /10 factor in productivity. Although, it might be unavoidable at some point.
- yummyfajitas 13y agoThere is a huge gap between Pandas and Hadoop. That gap is well served by a $2-3k Postgres box with 32-64GB ram and 5-10TB of disk. The main reason to switch to Hadoop at the point when Pandas fails is because you expect to scale past that gap fairly quickly.