4 ms·

From the storage system docs: > which organizes sample data in chunks of constant size (1024 bytes payload). These chunks are then stored on disk in one file p

by mbell 12y ago

From the storage system docs: > which organizes sample data in chunks of constant size (1024 bytes payload). These chunks are then stored on disk in one file per time series. That is concerning, is this going to have the same problem with disk IO that graphite does? i.e. Every metric update requires a disk IO due to this one file per metric structure.

falcolas 12y ago
This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data). Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.
- beorn7 12y ago
  We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.
  - falcolas 12y ago
    So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks. Still in wait-and-see mode.
    - jrv 12y ago
      So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten by new ones at predictable periods. It's great because you can just derive the file position of a sample based on its timestamp, but in Prometheus, samples can have arbitrary time stamps and gaps between them, and time series can also grow arbitrarily large (depending on currently configured retention period) and our data format needs to support that. InnoDB: not sure how this works internally, but given it's usually used in MySQL, does it work well for time series data? I.e. millions of time series that each get frequent appends? KairosDB: depends on Cassandra, AFAICS. One main design goal was to not depend on complex distributed storage for immediate fault detection, etc. InfluxDB: looks great, but has an incompatible data model. See http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu... I guess a central question that touches on your iops one is: you always end up having a two-dimensional layout on disk: timeseries X time. I don't really see a way to both store and retrieve data in such a way that you can arbitrarily select a time range and time series without incurring a lot of iops either on read or write.
      - falcolas 12y ago
        > does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.
        jrv 12y ago
        An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values. > Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data. Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?
beorn7 12y ago
(Disclaimer: I'm a Prometheus author.) Chunks are only written to that one file per time series once they are complete. Depending on their compression behavior, they will contain at least 64 samples, but usually a couple of hundreds. Even then, chunks are just queued for disk persistence. The storage layer operates completely from RAM, only has to swap in chunks if it was evicted from memory previously. Obviously, if you consistently create more sample data than your disk can write, the persist queue will back up at some point and Prometheus will throttle ingestion.

by mbell 12y ago

falcolas 12y ago

This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data). Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.

beorn7 12y ago
We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.
- falcolas 12y ago
  So, the open source timeseries DBs (RRD Tool, Influx, and KairosDB), and others like sqlite or even InnoDB didn't make the cut? That surprises me. > file systems are way better in managing the data Except they're not managing data, they're just separating tables, to extend the DB metaphor. And you still run the chance of running out of inodes on a "modern" file system like ext4. After having briefly dug into the code, I'm particularly worried about the fact that instead of minimizing iops by only writing the relevant changes to the same file, Prometheus is constantly copying data from one file to the next, both to perform your checkpoints and to invalidate old data. That's a lot of iops for such basic (and frequently repeated) tasks. Still in wait-and-see mode.
  - jrv 12y ago
    So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten by new ones at predictable periods. It's great because you can just derive the file position of a sample based on its timestamp, but in Prometheus, samples can have arbitrary time stamps and gaps between them, and time series can also grow arbitrarily large (depending on currently configured retention period) and our data format needs to support that. InnoDB: not sure how this works internally, but given it's usually used in MySQL, does it work well for time series data? I.e. millions of time series that each get frequent appends? KairosDB: depends on Cassandra, AFAICS. One main design goal was to not depend on complex distributed storage for immediate fault detection, etc. InfluxDB: looks great, but has an incompatible data model. See http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu... I guess a central question that touches on your iops one is: you always end up having a two-dimensional layout on disk: timeseries X time. I don't really see a way to both store and retrieve data in such a way that you can arbitrarily select a time range and time series without incurring a lot of iops either on read or write.
    - falcolas 12y ago
      > does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.
      - jrv 12y ago
        An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values. > Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data. Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?

beorn7 12y ago

(Disclaimer: I'm a Prometheus author.) Chunks are only written to that one file per time series once they are complete. Depending on their compression behavior, they will contain at least 64 samples, but usually a couple of hundreds. Even then, chunks are just queued for disk persistence. The storage layer operates completely from RAM, only has to swap in chunks if it was evicted from memory previously. Obviously, if you consistently create more sample data than your disk can write, the persist queue will back up at some point and Prometheus will throttle ingestion.