4 ms·

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in t

by beorn7 12y ago

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

falcolas 12y ago
So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks. Still in wait-and-see mode.
- jrv 12y ago
  So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten by new ones at predictable periods. It's great because you can just derive the file position of a sample based on its timestamp, but in Prometheus, samples can have arbitrary time stamps and gaps between them, and time series can also grow arbitrarily large (depending on currently configured retention period) and our data format needs to support that. InnoDB: not sure how this works internally, but given it's usually used in MySQL, does it work well for time series data? I.e. millions of time series that each get frequent appends? KairosDB: depends on Cassandra, AFAICS. One main design goal was to not depend on complex distributed storage for immediate fault detection, etc. InfluxDB: looks great, but has an incompatible data model. See http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu... I guess a central question that touches on your iops one is: you always end up having a two-dimensional layout on disk: timeseries X time. I don't really see a way to both store and retrieve data in such a way that you can arbitrarily select a time range and time series without incurring a lot of iops either on read or write.
  - falcolas 12y ago
    > does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.
    - jrv 12y ago
      An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values. > Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data. Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?
      - ithkuil 12y ago
        Did your previous leveldb approach perform way worse on reads, writes or both?
        bbrazil 12y ago
        Both. When I switched our production Prometheis over, the consoles rendered more than twice as fast for a simple test case.
  - pauldix 12y ago
    InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.
- beorn7 12y ago
  The checkpointing is just a way to minimize sample losses from unfinished chunks upon a crash. If you don't care or if you don't crash, you can disable checkpointing. Normal operation will be the same. Invalidation of old data is super easy with the chunked files as you can simply "truncate from the beginning", which is supported by various file systems (e.g. XFS). However, benchmarks so far showed that the relative amount of IO for purging old chunks is so small compared to overall IO that we didn't bother to implement it yet. Could be done if it turns out to be critical.

by beorn7 12y ago

falcolas 12y ago

So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks. Still in wait-and-see mode.

jrv 12y ago
So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten by new ones at predictable periods. It's great because you can just derive the file position of a sample based on its timestamp, but in Prometheus, samples can have arbitrary time stamps and gaps between them, and time series can also grow arbitrarily large (depending on currently configured retention period) and our data format needs to support that. InnoDB: not sure how this works internally, but given it's usually used in MySQL, does it work well for time series data? I.e. millions of time series that each get frequent appends? KairosDB: depends on Cassandra, AFAICS. One main design goal was to not depend on complex distributed storage for immediate fault detection, etc. InfluxDB: looks great, but has an incompatible data model. See http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu... I guess a central question that touches on your iops one is: you always end up having a two-dimensional layout on disk: timeseries X time. I don't really see a way to both store and retrieve data in such a way that you can arbitrarily select a time range and time series without incurring a lot of iops either on read or write.
- falcolas 12y ago
  > does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.
  - jrv 12y ago
    An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values. > Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data. Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?
    - ithkuil 12y ago
      Did your previous leveldb approach perform way worse on reads, writes or both?
      - bbrazil 12y ago
        Both. When I switched our production Prometheis over, the consoles rendered more than twice as fast for a simple test case.
- pauldix 12y ago
  InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.
beorn7 12y ago
The checkpointing is just a way to minimize sample losses from unfinished chunks upon a crash. If you don't care or if you don't crash, you can disable checkpointing. Normal operation will be the same. Invalidation of old data is super easy with the chunked files as you can simply "truncate from the beginning", which is supported by various file systems (e.g. XFS). However, benchmarks so far showed that the relative amount of IO for purging old chunks is so small compared to overall IO that we didn't bother to implement it yet. Could be done if it turns out to be critical.