4 ms·
The author is experimenting with 1.75Gigs of data. At that scale sure, a local machine will be faster. Hadoop's real use-case though is when your data doesn't f
by thecleaner 7y ago
The author is experimenting with 1.75Gigs of data. At that scale sure, a local machine will be faster. Hadoop's real use-case though is when your data doesn't fit in memory and even this is kind of debatable. It makes sense to measure the performance with some prototypes and then make a final design rather than just use whatever AWS offers. Besides packaged services in AWS are also a bit more costly than basic services like EC2 instances and network goodies.
- reallydontask 7y agoThese days you can get servers with Terabytes of RAM, so a lot of people (most?) could fit their data in memory. I just took a gander to HPE's website and you can get proliant servers with up to 12 TB of RAM (you might be able to get them with more RAM, did not check in detail)
- thecleaner 7y agoHPE ?
- arnolox 7y agoHewlett-Packard Enterprise
- keanzu 7y agoHow much does it matter that the data doesn't fit in memory when consumer grade (Samsung 970 Pro) SSDs can deliver 3GB/s sequential read? On inexpensive hardware you can process a ~TB every 5 minutes per SSD.
- gambiting 7y agoWell, I know someone in the business where the average database is multiple TB, and they still have to somehow run queries against it, 1TB/minute is just far too slow.
- sp332 7y agoHadoop is useful at the scale where your data is already distributed, maybe even in different datacenters. At that point, it's faster to push the computation out to where the data is already stored. It doesn't make sense to me when the data starts out all in one place.