3 ms·
I've heard people say that a huge portion of the "big" data is garbage, throwing more garbage at something won't make it better, so you'd get much more out of i
by dkersten 7y ago
I've heard people say that a huge portion of the "big" data is garbage, throwing more garbage at something won't make it better, so you'd get much more out of it by cutting it down to a more manageable amount of good data, which you can then easily process in postgres or whatever.
But even if you "collect everything and sort it out later", in my own personal experience and in what I've read here on HN, you can go a long long way before you need to reach for the power tools. What most companies call "big data" is typically not that much (in quantity and in velocity). Most companies don't have tens or hundreds of terabytes of data. For example, I'm currently processing timeseries data in postgres using the timescaledb extension, which makes it perform very well. Still too early to state numbers, but its looking promising so far and if their claims are true, then I won't need anything else. We will see :)