3 ms·

Could just use a search db (e.g., elasticsearch) depending on the structure of your data and the aggregations you need.

• bmh100 12y ago
  For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.
  • pentium10 12y ago
    I would go to with elasticsearch, or if you reach terabytes/petabytes of data then look into Google BigQuery there you would get a constant 5-10 seconds response time for those high volume data. BigQuery is a WORM technology, so an append-only system might help you, especially in event-driven systems.
    • bmh100 12y ago
      BigQuery is interesting. Thank you for the link. I am trying to determine if large quantities of data like this can be analyzed with sub-second aggregation time, targeting a 2-3 second total response time to the user, including network latency. The hypothetical data set would not necessarily be event driven, but of course it is useful to have lower data latency. The aggregation speed is the biggest performance concern for me.