4 ms·

Since you've done your research, would you mind to post a short list of alternatives and reasons why have they been rejected? Thanks!

cube2222 7y ago
This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this resulted in gigabytes/s of network traffic where it definitely wasn't necessary, and fairly long query times. (I used it with ceph) Though I know the maintainers were planning to add some kind of caching so this may be fixed. By using the native prometheus data format you also don't get storage space savings over it. Cortex: Didn't spend any time on it, as I expected similar problems as with Thanos, so left it out for the end (which didn't came after all). I know it does contain a caching element. Victoria Metrics: As far as I know it's very well engineered and performs great. But I see only one active maintainer so am afraid to use it. M3DB: Requires a non-trivial amount of memory (I have 3 machines, each 128GB RAM to handle 70k writes/s each (though 1 was able to handle 120 and be stable)). However, with all machines on bunches of raid 0 ssd's, querying is quite snappy. You can set it up with different storage resolutions, so you get detailed data for recent queries, but also fast long range queries. It also uses a magnitude of storage space less than raw prometheus. The documentation is lacking in my opinion in terms of performance tuning, however, the code is well written, so I've just spent a while reading it and it exports very good metrics for itself. Network traffic between the m3coordinator (prometheus remote write gateway) and m3db nodes is kinda huge (5-10x the traffic prometheus->gateway) but that wasn't an issue. Another bonus is that it handles statsd metrics, though I haven’t yet tried that. For anybody afraid of it operationally, I’ve had no problems. It mostly worked as is.
- hagen1778 7y ago
  Thanks for details! Really appreciate it. > It also uses a magnitude of storage space less than raw prometheus AFAIK, Prometheus compression is about 1.2-3 bytes per datapoint. A magnitude less is 0.12-0.3 bytes - are these numbers correct?
  - cube2222 7y ago
    Here you have the specifics: https://m3db.github.io/m3/m3db/architecture/engine/ https://m3db.github.io/m3/m3db/architecture/engine/ I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.
    - sciurus 7y ago
      Downsampling might not save as much space as you think depending on how m3db works. https://github.com/thanos-io/thanos/issues/813 https://github.com/thanos-io/thanos/issues/813 goes in to why for a similar project.
      - roskilli 7y ago
        Whether it saves space or not, looking at metrics over period of months or years when the data is raw is far more slow/expensive than looking at downsampled data. If you still want to be able to quickly graph and view old data, downsampling is the only way to keep your queries interactive. Take for instance 30s data vs 10min data. 20x more computation, network exchange and everything else of that nature needs to happen. Also if you want to keep only a subset of your data for a very long time, you need to have retention policies - otherwise you end up storing all that extra data forever. At large numbers (terabytes to petabytes) this stuff is impactful, at smaller numbers (gigabytes) my points here are far less relevant.
      - cube2222 7y ago
        I know about the Thanos problem, and it's one of the things I didn't really like about it using the prometheus storage format. It does work in m3db. Following are the on-disk sizes of one replica: 2 weeks at 15s res: 90G 2 months at 1m res: 160G 3 months at 5m res: 60G EDIT: The only thing is that m3db doesn't really downsample. You just create one namespace (table) for each resolution, and set the m3coordinator up so that it writes to each at the wanted interval, then set up a different retention for it. (this way you have duplicates in recent data) M3 namespaces aren't set up for a specific resolution. The writer decides and you can write various series in different resolutions to one namespace in theory.
- sagichmal 7y ago
  > Victoria Metrics: As far as I know it's very well engineered and performs great As someone who has auditioned it, briefly, let me assure you that it is certainly not the former, and only appears to be the latter due to a lot of cut corners and spec-violating implementations.
  - cube2222 7y ago
    Could you elaborate on the specifics? I'd be very interested.
- cube2222 7y ago
  Another comment on m3db: m3query did have some inconsistencies compared to prometheus in how queries using intervals evaluated. (and sometimes didn't return any data because of it). Having stepped through the code I don't remember the reason why that was, but I ended up using prometheus instances using remote_read from the m3coordinator (gateway), working like a charm.
- bboreham 7y ago
  Cortex has a design much more like M3DB than Thanos; I wouldn’t expect the same problems as Thanos at all. (I am a Cortex maintainer)

cube2222 7y ago

This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this resulted in gigabytes/s of network traffic where it definitely wasn't necessary, and fairly long query times. (I used it with ceph) Though I know the maintainers were planning to add some kind of caching so this may be fixed. By using the native prometheus data format you also don't get storage space savings over it. Cortex: Didn't spend any time on it, as I expected similar problems as with Thanos, so left it out for the end (which didn't came after all). I know it does contain a caching element. Victoria Metrics: As far as I know it's very well engineered and performs great. But I see only one active maintainer so am afraid to use it. M3DB: Requires a non-trivial amount of memory (I have 3 machines, each 128GB RAM to handle 70k writes/s each (though 1 was able to handle 120 and be stable)). However, with all machines on bunches of raid 0 ssd's, querying is quite snappy. You can set it up with different storage resolutions, so you get detailed data for recent queries, but also fast long range queries. It also uses a magnitude of storage space less than raw prometheus. The documentation is lacking in my opinion in terms of performance tuning, however, the code is well written, so I've just spent a while reading it and it exports very good metrics for itself. Network traffic between the m3coordinator (prometheus remote write gateway) and m3db nodes is kinda huge (5-10x the traffic prometheus->gateway) but that wasn't an issue. Another bonus is that it handles statsd metrics, though I haven’t yet tried that. For anybody afraid of it operationally, I’ve had no problems. It mostly worked as is.

hagen1778 7y ago
Thanks for details! Really appreciate it. > It also uses a magnitude of storage space less than raw prometheus AFAIK, Prometheus compression is about 1.2-3 bytes per datapoint. A magnitude less is 0.12-0.3 bytes - are these numbers correct?
- cube2222 7y ago
  Here you have the specifics: https://m3db.github.io/m3/m3db/architecture/engine/ https://m3db.github.io/m3/m3db/architecture/engine/ I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.
  - sciurus 7y ago
    Downsampling might not save as much space as you think depending on how m3db works. https://github.com/thanos-io/thanos/issues/813 https://github.com/thanos-io/thanos/issues/813 goes in to why for a similar project.
    - roskilli 7y ago
      Whether it saves space or not, looking at metrics over period of months or years when the data is raw is far more slow/expensive than looking at downsampled data. If you still want to be able to quickly graph and view old data, downsampling is the only way to keep your queries interactive. Take for instance 30s data vs 10min data. 20x more computation, network exchange and everything else of that nature needs to happen. Also if you want to keep only a subset of your data for a very long time, you need to have retention policies - otherwise you end up storing all that extra data forever. At large numbers (terabytes to petabytes) this stuff is impactful, at smaller numbers (gigabytes) my points here are far less relevant.
    - cube2222 7y ago
      I know about the Thanos problem, and it's one of the things I didn't really like about it using the prometheus storage format. It does work in m3db. Following are the on-disk sizes of one replica: 2 weeks at 15s res: 90G 2 months at 1m res: 160G 3 months at 5m res: 60G EDIT: The only thing is that m3db doesn't really downsample. You just create one namespace (table) for each resolution, and set the m3coordinator up so that it writes to each at the wanted interval, then set up a different retention for it. (this way you have duplicates in recent data) M3 namespaces aren't set up for a specific resolution. The writer decides and you can write various series in different resolutions to one namespace in theory.
sagichmal 7y ago
> Victoria Metrics: As far as I know it's very well engineered and performs great As someone who has auditioned it, briefly, let me assure you that it is certainly not the former, and only appears to be the latter due to a lot of cut corners and spec-violating implementations.
- cube2222 7y ago
  Could you elaborate on the specifics? I'd be very interested.
cube2222 7y ago
Another comment on m3db: m3query did have some inconsistencies compared to prometheus in how queries using intervals evaluated. (and sometimes didn't return any data because of it). Having stepped through the code I don't remember the reason why that was, but I ended up using prometheus instances using remote_read from the m3coordinator (gateway), working like a charm.
bboreham 7y ago
Cortex has a design much more like M3DB than Thanos; I wouldn’t expect the same problems as Thanos at all. (I am a Cortex maintainer)