2 ms·

I don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.

by klysm 4y ago

dijit 4y ago
There’s a good talk on Monarch https://youtu.be/2mw12B7W7RI https://youtu.be/2mw12B7W7RI Why it exists is laid out quite plainly. The pain of it is we’re all jumping on Prometheus (borgmon) without considering why Monarch exists. Monarch doesn’t have a good corollary outside of google. Maybe some weird mix of timescale DB backed by cockroachdb with a Prometheus push gateway.
- AlphaSite 4y ago
  Wavefront is based on FoundationDB which I’ve always found pretty cool. [1] https://news.ycombinator.com/item?id=16879392 https://news.ycombinator.com/item?id=16879392 Disclaimer: I work at vmware on an unrelated thing.
- bbkane 4y ago
  They should open source it like they did Kubernetes. Otherwise the world will (continue to) converge on the Prometheus model and Google will be left with this weird system that may be technically better but is unfamiliar to incoming engineers
gttalbot 4y ago
The key trade off if low-dependency, with everything up to and including alert notification delivery pushed down to the region. For alerting queries to happen there is little infrastructure Monarch depends upon to keep running beyond networking and being scheduled on the machines. If you think about Bigtable, a key observation that the Monarch team made very early on is that, if you can support good materialized views (implemented as periodic standing queries) written back to the memtable, and the memtable can hold the whole data set needed to drive alerting, this can work even if much of Google's infrastructure is having problems. It also allows Monarch to monitor systems like Bigtable, Colossus, etc., as it doesn't use them as serving dependencies for alerting or recent dashboard data. It's a question of optimizing for graceful degradation in the presence of failure of the infrastructure around the monitoring system. The times the system will experience its heaviest and most unpredictable load will be when everyone's trying to figure out why their service isn't working.

I don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.

by klysm 4y ago

I don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.

There’s a good talk on Monarch https://youtu.be/2mw12B7W7RI https://youtu.be/2mw12B7W7RI Why it exists is laid out quite plainly. The pain of it is we’re all jumping on Prometheus (borgmon) without considering why Monarch exists. Monarch doesn’t have a good corollary outside of google. Maybe some weird mix of timescale DB backed by cockroachdb with a Prometheus push gateway.

AlphaSite 4y ago
Wavefront is based on FoundationDB which I’ve always found pretty cool. [1] https://news.ycombinator.com/item?id=16879392 https://news.ycombinator.com/item?id=16879392 Disclaimer: I work at vmware on an unrelated thing.
bbkane 4y ago
They should open source it like they did Kubernetes. Otherwise the world will (continue to) converge on the Prometheus model and Google will be left with this weird system that may be technically better but is unfamiliar to incoming engineers

gttalbot 4y ago

The key trade off if low-dependency, with everything up to and including alert notification delivery pushed down to the region. For alerting queries to happen there is little infrastructure Monarch depends upon to keep running beyond networking and being scheduled on the machines. If you think about Bigtable, a key observation that the Monarch team made very early on is that, if you can support good materialized views (implemented as periodic standing queries) written back to the memtable, and the memtable can hold the whole data set needed to drive alerting, this can work even if much of Google's infrastructure is having problems. It also allows Monarch to monitor systems like Bigtable, Colossus, etc., as it doesn't use them as serving dependencies for alerting or recent dashboard data. It's a question of optimizing for graceful degradation in the presence of failure of the infrastructure around the monitoring system. The times the system will experience its heaviest and most unpredictable load will be when everyone's trying to figure out why their service isn't working.