4 ms·

Is like everyone creating a time series database from scratch?

rodionos 9y ago
The fact that many companies (FB, Uber, Google, Netflix, SO) roll their own TSDBs for metrics collection suggests that there is a real need. Or maybe there is not. It could a way to make boring system monitoring jobs fun and fancy again.
- metaobject 9y ago
  Perhaps these companies have such varied requirements that none of the existing TSDB systems fully support? We built our own custom time series db, along with a suite of tools for accessing, slicing, plotting, etc bc (at the time, at least) there was no support for bitpacking data, and storing/calculating certain spatial data operations.
- weq 9y ago
  My first job was in 2005 writing a TSDB abstraction over sql server used for BI with defense. we merged ~30Gb of data per night in about 8hrs on a 32core machine which was severly limited by a platter SAN. We would get about 1 disk failure /6months. So do u need an actual TSDB? Or are other companies doing what we did?
Rapzid 9y ago
I was under the impression that influxdb's storage engine was going to be a viable solution for Prometheus at some point. Now they are writing their own; not sure where that interest went.
- sagichmal 9y ago
  That was never true AFAIK. I recall one of the core devs did a comparison of storage engines years ago and found Influx unsuitable; that work led to the current-gen storage engine.
  - bbrazil 9y ago
    That was the 1st InfluxDB storage engine, things have evolved in the intervening years. The latest InfluxDB design is actually quite similar to the latest Prometheus design, what's different is our approaches to reliability and clustering. It's presently looking like Influx might once again be an option for long term storage for Prometheus.
    - Rapzid 9y ago
      That was my impression after the influxdb storage engine rewrite; I'm guessing the similarities are not a coincidence :) I'm not sure how interested influxdb would be to the idea, given their shrewd moves towards monetization, but it would be nice if the storage engine could be developed as a component of influxdb and adopted into Prometheus(ala rocks, level, etc).
- pas 9y ago
  That was about remote storage. And that was around the time when Influx itself used pluggable backends. Now influx has its own storage engine, and prometheus too.

The fact that many companies (FB, Uber, Google, Netflix, SO) roll their own TSDBs for metrics collection suggests that there is a real need. Or maybe there is not. It could a way to make boring system monitoring jobs fun and fancy again.

metaobject 9y ago
Perhaps these companies have such varied requirements that none of the existing TSDB systems fully support? We built our own custom time series db, along with a suite of tools for accessing, slicing, plotting, etc bc (at the time, at least) there was no support for bitpacking data, and storing/calculating certain spatial data operations.
weq 9y ago
My first job was in 2005 writing a TSDB abstraction over sql server used for BI with defense. we merged ~30Gb of data per night in about 8hrs on a 32core machine which was severly limited by a platter SAN. We would get about 1 disk failure /6months. So do u need an actual TSDB? Or are other companies doing what we did?

Rapzid 9y ago

I was under the impression that influxdb's storage engine was going to be a viable solution for Prometheus at some point. Now they are writing their own; not sure where that interest went.

sagichmal 9y ago
That was never true AFAIK. I recall one of the core devs did a comparison of storage engines years ago and found Influx unsuitable; that work led to the current-gen storage engine.
- bbrazil 9y ago
  That was the 1st InfluxDB storage engine, things have evolved in the intervening years. The latest InfluxDB design is actually quite similar to the latest Prometheus design, what's different is our approaches to reliability and clustering. It's presently looking like Influx might once again be an option for long term storage for Prometheus.
  - Rapzid 9y ago
    That was my impression after the influxdb storage engine rewrite; I'm guessing the similarities are not a coincidence :) I'm not sure how interested influxdb would be to the idea, given their shrewd moves towards monetization, but it would be nice if the storage engine could be developed as a component of influxdb and adopted into Prometheus(ala rocks, level, etc).
pas 9y ago
That was about remote storage. And that was around the time when Influx itself used pluggable backends. Now influx has its own storage engine, and prometheus too.