5 ms·

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be per

by jrv 12y ago

So we're admittedly not hardcore storage experts, but we did a lot of experiments and iterations until we arrived at the current storage, and it seems to be performing quite well for our needs, and much better than previous iterations. We're happy to learn better ways of data storage/access for time series data though. RRD Tool: expects samples to come in at regular intervals and expects old samples to be overwritten by new ones at predictable periods. It's great because you can just derive the file position of a sample based on its timestamp, but in Prometheus, samples can have arbitrary time stamps and gaps between them, and time series can also grow arbitrarily large (depending on currently configured retention period) and our data format needs to support that. InnoDB: not sure how this works internally, but given it's usually used in MySQL, does it work well for time series data? I.e. millions of time series that each get frequent appends? KairosDB: depends on Cassandra, AFAICS. One main design goal was to not depend on complex distributed storage for immediate fault detection, etc. InfluxDB: looks great, but has an incompatible data model. See http://prometheus.io/docs/introduction/comparison/#prometheus-vs.-influxdb http://prometheus.io/docs/introduction/comparison/#prometheu... I guess a central question that touches on your iops one is: you always end up having a two-dimensional layout on disk: timeseries X time. I don't really see a way to both store and retrieve data in such a way that you can arbitrarily select a time range and time series without incurring a lot of iops either on read or write.

falcolas 12y ago
> does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.
- jrv 12y ago
  An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values. > Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data. Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?
  - ithkuil 12y ago
    Did your previous leveldb approach perform way worse on reads, writes or both?
    - bbrazil 12y ago
      Both. When I switched our production Prometheis over, the consoles rendered more than twice as fast for a simple test case.
pauldix 12y ago
InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.
- jrv 12y ago
  Hey Paul, it'd be great to work together! Although long-term storage isn't a SoundCloud priority right now (meaning it's going to be more of a free-time project with less time allocated), it's definitely important for many others and it would be great to have. I think we should start with Prometheus->InfluxDB write support first, and then approach the more complicated problem of reading back from InfluxDB through Prometheus. What's the best place to get informed about the new 0.9.0 data model?
  - jrv 12y ago
    Ok, I found http://influxdb.com/blog/2014/12/08/clustering_tags_and_enhancements_in_0_9_0.html http://influxdb.com/blog/2014/12/08/clustering_tags_and_enha.... That all sounds very promising!
  - jrv 12y ago
    After reading http://influxdb.com/blog/2014/12/08/clustering_tags_and_enhancements_in_0_9_0.html http://influxdb.com/blog/2014/12/08/clustering_tags_and_enha..., I'm not completely clear on what "tags" mean. Are they single-word names, or are they conceptually key=value pairs like labels in Prometheus or tags in OpenTSDB? EDIT: found https://github.com/influxdb/influxdb/pull/1059 https://github.com/influxdb/influxdb/pull/1059. Ok, does seem like tags mean key=value pairs.

by jrv 12y ago

falcolas 12y ago

> does it work well for time series data It's a key/value store at its heart, with all the ACID magic and memory buffering built in. Almost any KV store would preform relatively well at time series data by simply by updates to overwrite old data instead of constantly deleting old data (assuming the KV store is efficient in its updates). Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data.

jrv 12y ago
An earlier iteration of our storage was actually based on LevelDB (key-value store), with this kind of key->value layout: [time series fingerprint : time range] -> [chunk of ts/value samples]] At least this scheme performed way worse than what we currently have. You could say that file systems also come with pretty good memory buffering and can act as key-value stores (with the file name being the key and the contents the value), except that they also allow efficient appends to values. > Issuing updates instead of deletes is possible because you know the storage duration and interval, and can thus easily identify an index at which to store the data. Do you mean you would actually append/update an existing value in the KV-store (which most KV stores don't allow without reading/writing the whole key)?
- ithkuil 12y ago
  Did your previous leveldb approach perform way worse on reads, writes or both?
  - bbrazil 12y ago
    Both. When I switched our production Prometheis over, the consoles rendered more than twice as fast for a simple test case.

pauldix 12y ago

InfluxDB CEO here. The data model for 0.9.0 should map up nicely with what you're trying to do. It supports measurement names, and tags. It would be great to work together to have InfluxDB as another option (other than OpenTSDB) for long term storage of metrics data.

jrv 12y ago
Hey Paul, it'd be great to work together! Although long-term storage isn't a SoundCloud priority right now (meaning it's going to be more of a free-time project with less time allocated), it's definitely important for many others and it would be great to have. I think we should start with Prometheus->InfluxDB write support first, and then approach the more complicated problem of reading back from InfluxDB through Prometheus. What's the best place to get informed about the new 0.9.0 data model?
- jrv 12y ago
  Ok, I found http://influxdb.com/blog/2014/12/08/clustering_tags_and_enhancements_in_0_9_0.html http://influxdb.com/blog/2014/12/08/clustering_tags_and_enha.... That all sounds very promising!
- jrv 12y ago
  After reading http://influxdb.com/blog/2014/12/08/clustering_tags_and_enhancements_in_0_9_0.html http://influxdb.com/blog/2014/12/08/clustering_tags_and_enha..., I'm not completely clear on what "tags" mean. Are they single-word names, or are they conceptually key=value pairs like labels in Prometheus or tags in OpenTSDB? EDIT: found https://github.com/influxdb/influxdb/pull/1059 https://github.com/influxdb/influxdb/pull/1059. Ok, does seem like tags mean key=value pairs.