5 ms·
Too many companies do 'big data', as it's currently a hype. In most cases hadoop is total overkill -- developers don't understand how to reduce problems, might
by lessthunk 11y ago
Too many companies do 'big data', as it's currently a hype.
In most cases hadoop is total overkill -- developers don't understand how to reduce problems, might have never heard of sampling, good data-structures, know your problem, etc.
For sure, there are valid use scenarios for map reduce/hadoop, etc. but in many cases it's a big waste of money.
- rdtsc 11y agoExactly. "We don't do big data, we do little data" -- said no dev team ever, since Big Data has appeared. Loading a 100MB Excel file -- big data. Dumping a few GB to SQLite -- big data. And programmers are now of course "Data Scientists". Same with services. When microservices became cool, all the other services have disappeared and everyone is doing microservices.
- threeseed 11y agoDo you work in big data ? I do and have never actually seen a company do "big data" without needing to. At enterprise level it is often a pretty sizeable investment e.g. buying dedicated appliances and hiring expensive teams and so generally quite a number of architects and engineers are involved in the decision. And I would disagree that Hadoop is a big waste of money. There isn't anything else really that comes close to its cost effectiveness once you reach a certain data size.