4 ms·

How often do you go over the raw data, in a manner that aggregates couldn't help you with, for data from 6+ months ago?

• jph 13y ago
  Never. "Big Data" for this use case is entirely about velocity, not volume. I suppose a better catchphrase would be "Fast Data". :)
  • glesica 13y ago
    Yes, I think "streaming data" and "big data" really are different things, and I like your term "Fast Data". From a lower-level perspective they involve very different challenges. If you need to store the data (Twitter-like data) then storage becomes your primary concern. Analysis can be done later and in less-than-real time. But if your data are coming in so fast that keeping up is the problem, then storage isn't even a consideration (you generally just don't) and analysis becomes the challenging part, you need to aggregate do be able to update your metrics on the fly. So they really aren't even overlapping problems outside of the fact that they both deal with lots of data.
  • eitally 13y ago
    manufacturing systems are one area where you really care about historical data for many years. for operational reasons as well as for related data retention adherence.
  • eitally 13y ago
    manufacturing systems are one area where you really care about historical data for many years. for operational reasons as well as for related data retention adherence.