8 ms·

There still isn't a leading time series database. Think the use case definitely deserves it though. I've seen so many projects spend months in building custom d

There still isn't a leading time series database. Think the use case definitely deserves it though. I've seen so many projects spend months in building custom data rollup solutions, it's a waste of time. A good time series database would be pretty amazing.

invertedohm 10y ago
While it's a relatively young product, I've been running InfluxDB at scale for over a year in production and it's been a joy to use. Hundreds of hosts reporting stats every 10 seconds and the query language is straightforward and powerful once you grok the data structures a bit. It fits the bill, I think.
- pwernersbach 10y ago
  Can you describe your workload and your experience with InfluxDB? We evaluated InfluxDB in 2015 and found that it fell apart due to the large amount of data we produce. I'd be interested in hearing your experiences to see if we need to reevaluate.
  - invertedohm 10y ago
    Workload is mostly writing incoming metrics from many dynamic instances (AWS). In 2015 Influx was at pre-1.0 and definitely had a tendency to "fall apart" under stress - for one, there were no memory limitations implemented, so an accidental large query would eat up all your memory and kill the database. To be fair, no one recommended putting it in production at that point. It's just that I needed something to run a Graphite-like metrics platform at scale and it had to be future-proof. Bit of a gamble but it turned out well! :)
- lsh 10y ago
  +1 for InfluxDB and the TICK stack. We evaluated it for our monitoring needs, decided not to go with it but were damned impressed with how easy it was to setup and use, the features (including the query language) and the overall quality of the stack. Less impressed with the Influx org and their SaaS, but I definitely want to find a usecase for getting stuck into Influx for time series collection
infinite8s 10y ago
There is, it just costs as much as a fully loaded engineer (about $150k/yr). It's called Kdb.
- snark42 10y ago
  Or more, and means you have to learn the dreaded Q language.
  - cleum 10y ago
    Personally I find Q fine. It's a useful filter. If the language is too hard for someone to pick up, the person would struggle with the complexities of correctly designing or performing analytics on a high frequency distributed custom time series db anyway.
bbrazil 10y ago
There's several, depending on use case. Each make different tradeoffs, so you have to decide what's important for you. For example Prometheus (which I work on) is great at reliable monitoring and powerful processing of metrics at high volumes, but it'd be unwise to use it for event logging or customer billing. If you're doing IoT or event logging then InfluxDB might be a good choice for you, though if you're doing more text-based logging then Elasticsearch is nearer to what you're looking for. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit#gid=0 https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE... is one comparison of the various open source options.
- pwernersbach 10y ago
  In addition, there is influx-mysql: https://github.com/philip-wernersbach/influx-mysql https://github.com/philip-wernersbach/influx-mysql , if you're looking for a MySQL compatible solution.
- lobster_johnson 10y ago
  We're using Elasticsearch for event logging -- where "event" means analytics event, e.g. a page view -- and it's fantastic. The aggregation support is superb. We initially used Influx, but it could not perform well at the time (0.8). Our events are also heavily label-based. Basically, we do ETL at the time of write, collecting multiple documents into one mega-event, which is a complex, nested JSON document. It may have perhaps 150-200 fields. A single event may be something like "clicked button X". By storing the original document, we can aggregate based on any field value, including text and scalar fields, without having to think about a schema or about planning ahead of time what fields should be indexed or not. ES handles the rest pretty well. To do the same thing with Influx or Prometheus I suspect we'd have to reverse this and store the document as the labels, along with a single count (1) as the "metric". I don't know how well Influx etc. scale with number of unique label values, though I'd love to find out. The last time I read about this, I think they recommended not going overboard with them. What's different with business analytics is that the end product is typically multidimensional rollup reports over large time windows (number of page views per customer per web property per month, comparing by 2015 vs 2016, for example), and it's almost all "group by count", sometimes "count distinct" or averages. Whereas "rate per second"-type metrics aren't used anywhere in our app, for example.
siculars 10y ago
Riak TS 1.5 just dropped this week [0]. Designed to do the horizontal scale thing over temporally sorted data. /disclaimer/ I work for Basho [0] http://docs.basho.com/riak/ts/1.5.0/releasenotes/ http://docs.basho.com/riak/ts/1.5.0/releasenotes/
- Rapzid 10y ago
  Have you guys considered using influxdb as a backend? Influxdb has written at length about the difficulties of existing storage engines(including leveldb). My guess though is that Basho would opt to write their own backend off of influxdb's notes.. On a side note I find influxdb's use of nanosecond time precision spot on and Riak TS's millisecond resolution disappointing.
ironman1478 10y ago
To some extent there is http://www.osisoft.com http://www.osisoft.com (I work for this company so I'm a bit biased) The thing is, yes we have a time series database, but a lot of value comes from giving people the tools to analyze, distribute, and act on the data stored (why store data if you can't do anything useful with it?)
linuxhansl 10y ago
We use OpenTSDB with HBase with good success. Writing over 2m events/s in some clusters. We're also working on something with Apache Phoenix over HBase to store TS so that we can query the data in more flexible ways.
chaotic-good 10y ago
I'm working on my own solution for a while (about 3 years) - https://github.com/akumuli/Akumuli https://github.com/akumuli/Akumuli. One of the greatest things about it is that there is no need for preconfigured rollups. The storage is based on B+Tree and some aggregates are stored inside inner nodes of the tree.

invertedohm 10y ago

While it's a relatively young product, I've been running InfluxDB at scale for over a year in production and it's been a joy to use. Hundreds of hosts reporting stats every 10 seconds and the query language is straightforward and powerful once you grok the data structures a bit. It fits the bill, I think.

pwernersbach 10y ago
Can you describe your workload and your experience with InfluxDB? We evaluated InfluxDB in 2015 and found that it fell apart due to the large amount of data we produce. I'd be interested in hearing your experiences to see if we need to reevaluate.
- invertedohm 10y ago
  Workload is mostly writing incoming metrics from many dynamic instances (AWS). In 2015 Influx was at pre-1.0 and definitely had a tendency to "fall apart" under stress - for one, there were no memory limitations implemented, so an accidental large query would eat up all your memory and kill the database. To be fair, no one recommended putting it in production at that point. It's just that I needed something to run a Graphite-like metrics platform at scale and it had to be future-proof. Bit of a gamble but it turned out well! :)
lsh 10y ago
+1 for InfluxDB and the TICK stack. We evaluated it for our monitoring needs, decided not to go with it but were damned impressed with how easy it was to setup and use, the features (including the query language) and the overall quality of the stack. Less impressed with the Influx org and their SaaS, but I definitely want to find a usecase for getting stuck into Influx for time series collection

infinite8s 10y ago

There is, it just costs as much as a fully loaded engineer (about $150k/yr). It's called Kdb.

snark42 10y ago
Or more, and means you have to learn the dreaded Q language.
- cleum 10y ago
  Personally I find Q fine. It's a useful filter. If the language is too hard for someone to pick up, the person would struggle with the complexities of correctly designing or performing analytics on a high frequency distributed custom time series db anyway.

bbrazil 10y ago

There's several, depending on use case. Each make different tradeoffs, so you have to decide what's important for you. For example Prometheus (which I work on) is great at reliable monitoring and powerful processing of metrics at high volumes, but it'd be unwise to use it for event logging or customer billing. If you're doing IoT or event logging then InfluxDB might be a good choice for you, though if you're doing more text-based logging then Elasticsearch is nearer to what you're looking for. https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit#gid=0 https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE... is one comparison of the various open source options.

pwernersbach 10y ago
In addition, there is influx-mysql: https://github.com/philip-wernersbach/influx-mysql https://github.com/philip-wernersbach/influx-mysql , if you're looking for a MySQL compatible solution.
lobster_johnson 10y ago
We're using Elasticsearch for event logging -- where "event" means analytics event, e.g. a page view -- and it's fantastic. The aggregation support is superb. We initially used Influx, but it could not perform well at the time (0.8). Our events are also heavily label-based. Basically, we do ETL at the time of write, collecting multiple documents into one mega-event, which is a complex, nested JSON document. It may have perhaps 150-200 fields. A single event may be something like "clicked button X". By storing the original document, we can aggregate based on any field value, including text and scalar fields, without having to think about a schema or about planning ahead of time what fields should be indexed or not. ES handles the rest pretty well. To do the same thing with Influx or Prometheus I suspect we'd have to reverse this and store the document as the labels, along with a single count (1) as the "metric". I don't know how well Influx etc. scale with number of unique label values, though I'd love to find out. The last time I read about this, I think they recommended not going overboard with them. What's different with business analytics is that the end product is typically multidimensional rollup reports over large time windows (number of page views per customer per web property per month, comparing by 2015 vs 2016, for example), and it's almost all "group by count", sometimes "count distinct" or averages. Whereas "rate per second"-type metrics aren't used anywhere in our app, for example.

siculars 10y ago

Riak TS 1.5 just dropped this week [0]. Designed to do the horizontal scale thing over temporally sorted data. /disclaimer/ I work for Basho [0] http://docs.basho.com/riak/ts/1.5.0/releasenotes/ http://docs.basho.com/riak/ts/1.5.0/releasenotes/

Rapzid 10y ago
Have you guys considered using influxdb as a backend? Influxdb has written at length about the difficulties of existing storage engines(including leveldb). My guess though is that Basho would opt to write their own backend off of influxdb's notes.. On a side note I find influxdb's use of nanosecond time precision spot on and Riak TS's millisecond resolution disappointing.

ironman1478 10y ago

To some extent there is http://www.osisoft.com http://www.osisoft.com (I work for this company so I'm a bit biased) The thing is, yes we have a time series database, but a lot of value comes from giving people the tools to analyze, distribute, and act on the data stored (why store data if you can't do anything useful with it?)

linuxhansl 10y ago

We use OpenTSDB with HBase with good success. Writing over 2m events/s in some clusters. We're also working on something with Apache Phoenix over HBase to store TS so that we can query the data in more flexible ways.

chaotic-good 10y ago

I'm working on my own solution for a while (about 3 years) - https://github.com/akumuli/Akumuli https://github.com/akumuli/Akumuli. One of the greatest things about it is that there is no need for preconfigured rollups. The storage is based on B+Tree and some aggregates are stored inside inner nodes of the tree.