3 ms·
ES is quite good at time-series data, actually. Lucene is effectively a sparse column store. Its storage format is quite compact, and they do some clever thing
by lobster_johnson 10y ago
ES is quite good at time-series data, actually.
Lucene is effectively a sparse column store. Its storage format is quite compact, and they do some clever things to balance between speed and memory usage. Lucene uses append-only index files (segments) that are compacted (merged) after a while; not too dissimilar from how Cassandra or LevelDB work.
The aggregation API in ES is quite extensive, too. Date histograms, top-K, etc., all composable and extremely fast, particularly when parallelized over multiple shards (though you have to prepare for how sharded aggregation can skew some aggregation calculations).
ES is also impressively fast at bulk indexing.
There are some downsides. ES has a global field cache of some sort (I forget the details) that makes it increasingly bloated as you add more unique fields, which can be a problem for some applications, particularly if you federate many different apps in one cluster. In general, ES -- being on the JVM -- is RAM-hungry, and it's a little too easy to cause it to run out of memory, even with a huge heap and all caches etc. configured with hard limits.
ES's schema system (the "mappings") also leaves a lot to be desired, but this problem is less applicable to an analytics app.
Lastly: If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?), ES doesn't give you any tools to do it. You will have to write the logic yourself.
- iampims 10y ago> If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?) I believe this is called a roll-up.