4 ms·
We are using influxDB to calculate SLO performance. Currently, among other things, we process about 30M ELB logs entries into influxdb per day; it handles this
by Rapzid 10y ago
We are using influxDB to calculate SLO performance. Currently, among other things, we process about 30M ELB logs entries into influxdb per day; it handles this easily of course. Here are some musings for those interested based on 0.9:
Wins:
* The new storage engine is very, very cool. Would love to work on this thing. It's fast and space efficient.
* Built in support for time bucketing GROUP's
* Grafana integration is pretty good
* Writes come back after the data is stored; makes it easy to create durable, idempotent processing pipelines.
Woahs
* Unable to combine measurements in the same query; needs ETL with continuous queries or external tools
* No support for subqueries; more ETL
* Stream processing is a little lacking -> can't group on values and high cardinality tags make the DB explode; high cardinality is being worked on but IDK how high it will go, plus I mean the storage engine serves up streams of time-sorted data so samza that stuff up.
* Random crashes but the DB gets along fine when it comes back up
* Compactions use LOTS of RAM. Supposedly this can be tweaked and has been improved for 1.0
* Backfill queries with lots of points seem to use a crazy amount of RAM when bucketing on narrow time windows
Overall it's chugging along quite well. Most of the query limitations we are able to solve with a combination of continuous queries and AWS lambda functions kicked off by CloudWatch Events.