6 ms·

Am I completely alone in thinking ElasticSearch is out of place here? Am I misunderstanding something about ES? It's a NoSQL doc store but is there something ab

Am I completely alone in thinking ElasticSearch is out of place here? Am I misunderstanding something about ES? It's a NoSQL doc store but is there something about its indexing or storage tech (Lucene, right?) that specifically makes it optimal for time-series data?

gillh 10y ago
There are a bunch of companies that use ElasticSearch for storing timeseries data for monitoring use-cases. Reference: 1. https://www.opsclarity.com/riak-elasticsearch-scaled-monitoring-backend-opsclarity/ https://www.opsclarity.com/riak-elasticsearch-scaled-monitor... 2. https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store https://www.elastic.co/blog/elasticsearch-as-a-time-series-d...
lobster_johnson 10y ago
ES is quite good at time-series data, actually. Lucene is effectively a sparse column store. Its storage format is quite compact, and they do some clever things to balance between speed and memory usage. Lucene uses append-only index files (segments) that are compacted (merged) after a while; not too dissimilar from how Cassandra or LevelDB work. The aggregation API in ES is quite extensive, too. Date histograms, top-K, etc., all composable and extremely fast, particularly when parallelized over multiple shards (though you have to prepare for how sharded aggregation can skew some aggregation calculations). ES is also impressively fast at bulk indexing. There are some downsides. ES has a global field cache of some sort (I forget the details) that makes it increasingly bloated as you add more unique fields, which can be a problem for some applications, particularly if you federate many different apps in one cluster. In general, ES -- being on the JVM -- is RAM-hungry, and it's a little too easy to cause it to run out of memory, even with a huge heap and all caches etc. configured with hard limits. ES's schema system (the "mappings") also leaves a lot to be desired, but this problem is less applicable to an analytics app. Lastly: If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?), ES doesn't give you any tools to do it. You will have to write the logic yourself.
- iampims 10y ago
  > If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?) I believe this is called a roll-up.
laichzeit0 10y ago
There's a comparison between InfluxDB and Elastic (albeit written by Influx themselves, so take it with a grain of salt), but it would appear there's a strong case to be made that there are better alternatives than Elastic if want to store time-series metrics. https://www.influxdata.com/influxdb-markedly-elasticsearch-in-time-series-data-metrics-benchmark/ https://www.influxdata.com/influxdb-markedly-elasticsearch-i...

gillh 10y ago

There are a bunch of companies that use ElasticSearch for storing timeseries data for monitoring use-cases. Reference: 1. https://www.opsclarity.com/riak-elasticsearch-scaled-monitoring-backend-opsclarity/ https://www.opsclarity.com/riak-elasticsearch-scaled-monitor... 2. https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store https://www.elastic.co/blog/elasticsearch-as-a-time-series-d...

lobster_johnson 10y ago

ES is quite good at time-series data, actually. Lucene is effectively a sparse column store. Its storage format is quite compact, and they do some clever things to balance between speed and memory usage. Lucene uses append-only index files (segments) that are compacted (merged) after a while; not too dissimilar from how Cassandra or LevelDB work. The aggregation API in ES is quite extensive, too. Date histograms, top-K, etc., all composable and extremely fast, particularly when parallelized over multiple shards (though you have to prepare for how sharded aggregation can skew some aggregation calculations). ES is also impressively fast at bulk indexing. There are some downsides. ES has a global field cache of some sort (I forget the details) that makes it increasingly bloated as you add more unique fields, which can be a problem for some applications, particularly if you federate many different apps in one cluster. In general, ES -- being on the JVM -- is RAM-hungry, and it's a little too easy to cause it to run out of memory, even with a huge heap and all caches etc. configured with hard limits. ES's schema system (the "mappings") also leaves a lot to be desired, but this problem is less applicable to an analytics app. Lastly: If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?), ES doesn't give you any tools to do it. You will have to write the logic yourself.

iampims 10y ago
> If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?) I believe this is called a roll-up.

laichzeit0 10y ago

There's a comparison between InfluxDB and Elastic (albeit written by Influx themselves, so take it with a grain of salt), but it would appear there's a strong case to be made that there are better alternatives than Elastic if want to store time-series metrics. https://www.influxdata.com/influxdb-markedly-elasticsearch-in-time-series-data-metrics-benchmark/ https://www.influxdata.com/influxdb-markedly-elasticsearch-i...