4 ms·
> “Big Data” data tends to be cold data My experience is the opposite: big data tends to be a lot of rapid streaming data, near-real-time, with inputs from man
by jph 13y ago
> “Big Data” data tends to be cold data
My experience is the opposite: big data tends to be a lot of rapid streaming data, near-real-time, with inputs from many sources and sensors. This overwhelms traditional databases.
For us, big data means we want stream filtering, heuristic sampling, map reduce, and the like.
- oijaf888 13y agoHow often do you go over the raw data, in a manner that aggregates couldn't help you with, for data from 6+ months ago?
- jph 13y agoNever. "Big Data" for this use case is entirely about velocity, not volume. I suppose a better catchphrase would be "Fast Data". :)
- glesica 13y agoYes, I think "streaming data" and "big data" really are different things, and I like your term "Fast Data". From a lower-level perspective they involve very different challenges. If you need to store the data (Twitter-like data) then storage becomes your primary concern. Analysis can be done later and in less-than-real time. But if your data are coming in so fast that keeping up is the problem, then storage isn't even a consideration (you generally just don't) and analysis becomes the challenging part, you need to aggregate do be able to update your metrics on the fly. So they really aren't even overlapping problems outside of the fact that they both deal with lots of data.
- eitally 13y agomanufacturing systems are one area where you really care about historical data for many years. for operational reasons as well as for related data retention adherence.
- eitally 13y agomanufacturing systems are one area where you really care about historical data for many years. for operational reasons as well as for related data retention adherence.