4 ms·

Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second

by bmh100 12y ago

Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second response time?

ddorian43 12y ago
From a glance, you have to define the queries before. Are you using anything currently? What is your insert/update/delete rate? If low, maybe look at postgresql in-memory column-store https://github.com/knizhnik/imcs/ https://github.com/knizhnik/imcs/ Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app. Also curious.
- bmh100 12y ago
  I specialize in an in-memory analytics (business intelligence) platform called "QlikView" or "Qlik". It operates on top of RDBMSs/operational databases, so it batch processes data on a daily or hourly basis usually.
  - pentium10 12y ago
    If you are already familiar with QlikView, then check out Google BigQuery, you already have a connector for it.
syncerr 12y ago
Could just use a search db (e.g., elasticsearch) depending on the structure of your data and the aggregations you need.
- bmh100 12y ago
  For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.
  - pentium10 12y ago
    I would go to with elasticsearch, or if you reach terabytes/petabytes of data then look into Google BigQuery there you would get a constant 5-10 seconds response time for those high volume data. BigQuery is a WORM technology, so an append-only system might help you, especially in event-driven systems.
    - bmh100 12y ago
      BigQuery is interesting. Thank you for the link. I am trying to determine if large quantities of data like this can be analyzed with sub-second aggregation time, targeting a 2-3 second total response time to the user, including network latency. The hypothetical data set would not necessarily be event driven, but of course it is useful to have lower data latency. The aggregation speed is the biggest performance concern for me.
gianm 12y ago
The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).

by bmh100 12y ago

Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second response time?

ddorian43 12y ago

From a glance, you have to define the queries before. Are you using anything currently? What is your insert/update/delete rate? If low, maybe look at postgresql in-memory column-store https://github.com/knizhnik/imcs/ https://github.com/knizhnik/imcs/ Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app. Also curious.

bmh100 12y ago
I specialize in an in-memory analytics (business intelligence) platform called "QlikView" or "Qlik". It operates on top of RDBMSs/operational databases, so it batch processes data on a daily or hourly basis usually.
- pentium10 12y ago
  If you are already familiar with QlikView, then check out Google BigQuery, you already have a connector for it.

syncerr 12y ago

Could just use a search db (e.g., elasticsearch) depending on the structure of your data and the aggregations you need.

bmh100 12y ago
For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.
- pentium10 12y ago
  I would go to with elasticsearch, or if you reach terabytes/petabytes of data then look into Google BigQuery there you would get a constant 5-10 seconds response time for those high volume data. BigQuery is a WORM technology, so an append-only system might help you, especially in event-driven systems.
  - bmh100 12y ago
    BigQuery is interesting. Thank you for the link. I am trying to determine if large quantities of data like this can be analyzed with sub-second aggregation time, targeting a 2-3 second total response time to the user, including network latency. The hypothetical data set would not necessarily be event driven, but of course it is useful to have lower data latency. The aggregation speed is the biggest performance concern for me.

gianm 12y ago

The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).