4 ms·

Alas, we’ve found the first thing I _dont_ like about fly. They’ve sadly bought into the TSDB model that everything is a counter instead of the more modern mode

Alas, we’ve found the first thing I _dont_ like about fly. They’ve sadly bought into the TSDB model that everything is a counter instead of the more modern model that everything is a histogram. Google has long since abandoned the Borgmon data model for histograms with monarch. The closest non google implementation is probably circonus. Sadly neither is available as open source software. I can’t really blame fly for not individually building an open source modern metric db. But it’s sort of sad that the infra team I’m most impressed with has to use metric systems from 15 years ago when the rest of their stack is so cutting edge.

tptacek 5y ago
We do plenty of histograms! Vicky is good at them! Counters are just easier to write about.
- kasey_junk 5y ago
  Vicky is about the best case scenario for a counter based metrics db. I am routinely impressed by the things it accomplishes but monarch & irondb from circonus use histograms as their base data structure which means they avoid all the hacks that counter based tsdb have to deal with. The obvious one in the Prometheus stack being having to pre-declare your histogram bounds.
  - tptacek 5y ago
    I'd also take Monarch's query language over PromQL any day.
  - valyala 5y ago
    There is no need in pre-declaring histogram bounds for VictoriaMetrics histograms - https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350 https://valyala.medium.com/improving-histogram-usability-for...
polskibus 5y ago
What would you recommend today as a self-hosted solution today?
- kasey_junk 5y ago
  Circonus’ Irondb is the only histogram db available outside of google that I know of. They have a self hosted option but it’s not free or open source.
darkwater 5y ago
So, most shops (at least the ones that have been existing for a few years) are just starting their move to Prometheus (or in the middle of it) and we already have a newer, better implementation to migrate to? I love the tech world! (I'm half ironic and half serious with my last sentence)
- Spivak 5y ago
  Well yeah because Prometheus was inspired by Google's Borgmon and soon there will be another product inspired by Google's Monarch. Google's SRE stack is lightyears ahead of anything else out there simply because they're at the scale where they can afford to hire dedicated developers to just write internal ops software where most other SREs are understaffed and overworked just on operational projects.
StreamBright 5y ago
For those who are curious and not familiar with Monarch: https://www.youtube.com/watch?v=2mw12B7W7RI https://www.youtube.com/watch?v=2mw12B7W7RI https://www.vldb.org/pvldb/vol13/p3181-adams.pdf https://www.vldb.org/pvldb/vol13/p3181-adams.pdf
bboreham 5y ago
Note there is work under way in Prometheus to make histograms more flexible: https://static.sched.com/hosted_files/promcononline2021/33/2021-05-03%20PromCon%20online.pdf https://static.sched.com/hosted_files/promcononline2021/33/2...

tptacek 5y ago

We do plenty of histograms! Vicky is good at them! Counters are just easier to write about.

kasey_junk 5y ago
Vicky is about the best case scenario for a counter based metrics db. I am routinely impressed by the things it accomplishes but monarch & irondb from circonus use histograms as their base data structure which means they avoid all the hacks that counter based tsdb have to deal with. The obvious one in the Prometheus stack being having to pre-declare your histogram bounds.
- tptacek 5y ago
  I'd also take Monarch's query language over PromQL any day.
- valyala 5y ago
  There is no need in pre-declaring histogram bounds for VictoriaMetrics histograms - https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350 https://valyala.medium.com/improving-histogram-usability-for...

polskibus 5y ago

What would you recommend today as a self-hosted solution today?

kasey_junk 5y ago
Circonus’ Irondb is the only histogram db available outside of google that I know of. They have a self hosted option but it’s not free or open source.

darkwater 5y ago

So, most shops (at least the ones that have been existing for a few years) are just starting their move to Prometheus (or in the middle of it) and we already have a newer, better implementation to migrate to? I love the tech world! (I'm half ironic and half serious with my last sentence)

Spivak 5y ago
Well yeah because Prometheus was inspired by Google's Borgmon and soon there will be another product inspired by Google's Monarch. Google's SRE stack is lightyears ahead of anything else out there simply because they're at the scale where they can afford to hire dedicated developers to just write internal ops software where most other SREs are understaffed and overworked just on operational projects.

StreamBright 5y ago

For those who are curious and not familiar with Monarch: https://www.youtube.com/watch?v=2mw12B7W7RI https://www.youtube.com/watch?v=2mw12B7W7RI https://www.vldb.org/pvldb/vol13/p3181-adams.pdf https://www.vldb.org/pvldb/vol13/p3181-adams.pdf

bboreham 5y ago

Note there is work under way in Prometheus to make histograms more flexible: https://static.sched.com/hosted_files/promcononline2021/33/2021-05-03%20PromCon%20online.pdf https://static.sched.com/hosted_files/promcononline2021/33/2...