29 ms·
Monarch: Google’s Planet-Scale In-Memory Time Series Database
- nickstinemates 4y agotoo small for me, i was looking more for the scale of the universe.
- yayr 4y agoin case this can be deployed single-handed it might be useful on a spaceship... would need some relativistic time accounting though.
- 8040 4y agoI broke this once several years ago. I even use the incident number in my random usernames to see if a Googler recognizes it.
- ajstiles 4y agoWow - that was a doozy.
- orf 4y agoHow did you break it?
- deleted 4y ago[deleted]
- zoover2020 4y agoThis is also why I love HN. So niche!
- ikiris 4y agoahahahahahaha
- foota 4y agoomg
- Xorlev 4y agoI was oncall for that incident. Good times.
- voldacar 4y agoCould someone elaborate on this
- 8040 4y agohttps://status.cloud.google.com/incident/google-stackdriver/18001 https://status.cloud.google.com/incident/google-stackdriver/... was one of the public incident reports around it.
- klysm 4y agoI don’t really grasp why this is a useful spot in the trade off space from a quick skim. Seems risky.
- dijit 4y agoThere’s a good talk on Monarch https://youtu.be/2mw12B7W7RI https://youtu.be/2mw12B7W7RI Why it exists is laid out quite plainly. The pain of it is we’re all jumping on Prometheus (borgmon) without considering why Monarch exists. Monarch doesn’t have a good corollary outside of google. Maybe some weird mix of timescale DB backed by cockroachdb with a Prometheus push gateway.
- AlphaSite 4y agoWavefront is based on FoundationDB which I’ve always found pretty cool. [1] https://news.ycombinator.com/item?id=16879392 https://news.ycombinator.com/item?id=16879392 Disclaimer: I work at vmware on an unrelated thing.
- bbkane 4y agoThey should open source it like they did Kubernetes. Otherwise the world will (continue to) converge on the Prometheus model and Google will be left with this weird system that may be technically better but is unfamiliar to incoming engineers
- gttalbot 4y agoThe key trade off if low-dependency, with everything up to and including alert notification delivery pushed down to the region. For alerting queries to happen there is little infrastructure Monarch depends upon to keep running beyond networking and being scheduled on the machines. If you think about Bigtable, a key observation that the Monarch team made very early on is that, if you can support good materialized views (implemented as periodic standing queries) written back to the memtable, and the memtable can hold the whole data set needed to drive alerting, this can work even if much of Google's infrastructure is having problems. It also allows Monarch to monitor systems like Bigtable, Colossus, etc., as it doesn't use them as serving dependencies for alerting or recent dashboard data. It's a question of optimizing for graceful degradation in the presence of failure of the infrastructure around the monitoring system. The times the system will experience its heaviest and most unpredictable load will be when everyone's trying to figure out why their service isn't working.
- pm90 4y agoA lot of Google projects seem to rely on other Google projects. In this case Monarch relies on spanner. I guess its nice to publish at least the conceptual design so that others can implement it in “rest of the world” case. Working with OSS can be painful, slow and time consuming so this seems like a reasonable middle ground (although selfishly I do wish all of this was source available).
- joshuamorton 4y agoI don't think there's any spanner necessity and iirc monarch existed pre-spanner.
- gttalbot 4y agoCorrect. Spanner is used to hold configuration state, but is not in the serving path.
- praptak 4y agoSpanner may be hard to set up even with source code available. It relies on atomic clocks for reliable ordering of events.
- deleted 4y ago[deleted]
- wbl 4y agoAtomic clocks aren't that exotic, and a GPS disciplined ovenized quartz oscillator will do just fine outside of a disruption. The hard part is getting the right sampling semantics, requiring end to end error analysis.
- dan-robertson 4y agoIt’s pretty hard to get well synchronised clocks between servers in a datacentre.
- candiddevmike 4y agoInteresting that Google replaced a pull based metric system similar to Prometheus with a push based system... I thought one of the selling points of Prometheus and the pull based dance was how scalable it was?
- jeffbee 4y agoPrometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
- buro9 4y agoThat's what Mimir solves
- deepsun 4y agoHow does it compare to VictoriaMetrics?
- buro9 4y ago100% Prometheus compatible, proven to scale to 1B active series. It's not about comparisons, every tool has it's own place and feature set that may be right for you depending on what you're doing. But if you've reached the end of the road with Prometheus due to scale and you need massive scale and perfect compatibility... Then Mimir stands out.
- halfmatthalfcat 4y agoCan you elaborate? I’ve ran Prometheus at some scale and it’s performed fine.
- lokar 4y agoYou pretty quickly exceed what one instance can handle for memory, cpu or both. At that point you don't have any real good options to scale while maintaining a flat namespace (you need to partition).
- dijit 4y agoDiscussion from 2020: https://news.ycombinator.com/item?id=24303422 https://news.ycombinator.com/item?id=24303422
- yegle 4y agoGoogle Cloud Monitoring's time series database is backed by Monarch. The query language is mql which closely resembles the internal Python based query language: https://cloud.google.com/monitoring/mql https://cloud.google.com/monitoring/mql
- sleepydog 4y agoMQL is an improvement over the internal language, IMO. There are some missing features around literal tables, but otherwise the language is more consistent and flexible.
- kasey_junk 4y agoA huge difference between monarch and other tsdb that isn’t outlined in this overview, is that a storage primitive for schema values is a histogram. Most (maybe all besides Circonus) tsdb try to create histograms at query time using counter primitives. All of those query time histogram aggregations are making pretty subtle trade offs that make analysis fraught.
- hn_go_brrrrr 4y agoIn my experience, Monarch storing histograms and being unable to rebucket on the fly is a big problem. A percentile line on a histogram will be incredibly misleading, because it's trying to figure out what the p50 of a bunch of buckets is. You'll see monitoring artifacts like large jumps and artificial plateaus as a result of how requests fall into buckets. The bucketer on the default RPC latency metric might not be well tuned for your service. I've seen countless experienced oncallers tripped up by this, because "my graphs are lying to me" is not their first thought.
- jrockway 4y agoI definitely remember a lot of time spent tweaking histogram buckets for performance vs. accuracy. The default bucketing algorithm at the time was powers of 4 or something very unusual like that.
- shadowgovt 4y agoIt's because powers of four was great for the original application of statistics on high traffic services where the primary thing the user was interested in was deviations from the norm, and with a high traffic system the signal for what the norm is would be very strong. I tried applying it to a service with much lower traffic and found the bucketing to be extremely fussy.
- deleted 4y ago[deleted]
- heinrichhartman 4y ago
- sydthrowaway 4y agoStop overhyping software with buzzwords
- gttalbot 4y agoWhat? Planet scale? Well. You can literally issue a query that fans out to every continent on Earth, and returns the result right to your dashboard. Not exaggerating. ;^) (OK maybe not Antarctica but I'm not sure...)
- codethief 4y agoThe first time I heard about Monarch was in discussions about the hilarious "I just want to serve 5 terabytes" video[0]. [0]: https://m.youtube.com/watch?v=3t6L-FlfeaI https://m.youtube.com/watch?v=3t6L-FlfeaI
- holly76 4y ago[flagged]
- hintymad 4y agoI don't quite get the benefit of pull model by default either. A pull model by default means that it's not easy for a library to publish its metrics. For instance, every god damn application is expected to implement a `/metrics` endpoint for a freaking agent to publish the application's metrics to Prometheus. With Monarch, any library or application can simply publish metrics to Monarch's API. Similarly in Netflix, publishing to its Altas system is totally transparent to library authors, with the help of their metric library. Sometimes I feel many open source systems do not give a shit about productivity.
- dan-robertson 4y agoThe general theory is that if a push-based system is getting overloaded you drop metric submissions but if a pull-based system is overloaded it will query less frequently and you’ll just get less resolution.
- hintymad 4y agoThanks. It's a valid thought on such trade-off. I do think we can still favor productivity without losing resolution, though, for the following two reasons: 1. A pull-based system can pull less when the system is overload, which means a pulled service needs to keep historical stats. For instance, the endpoint `/metric` needs to keep previous gauge values or the accumulated counters. That said, a push-based metric library can keep history too. Indeed, it is exactly what the micrometers library does. 2. Don't let the metric system overload. This sounds like a hyperbole, but it is what companies do in practice: telemetry system is so foundational and critical to an internet company that it should always run smoothly.
- cientifico 4y agoOfftopic: Could The web owner allow to zoom in, to see the content of the pictures?