9 ms·

Monarch: Google’s Planet-Scale In-Memory Time Series Database [pdf]

175 points by ngaut 6y ago

polskibus 6y ago
I wonder if Google will ever open source it.
- kyrra 6y ago
  Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed by a former Google SRE that has since returned). https://prometheus.io/ https://prometheus.io/
  - gregschlom 6y ago
    > Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released Monarch might be an exception to that, though, because - as noted in the paper -, it has to be built without many dependencies, since almost everything else depends on it for monitoring and alerting. That being said, I'm sure it would be extremely difficult to open source now since it was never built with open sourcing in mind.
    - cperciva 6y ago
      It might not depend on many other services, but that doesn't preclude it depending on other code.
  - fizixer 6y ago
    But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.
    - jeffbee 6y ago
      You could definitely build Monarch from this paper. It is very detailed. But keep in mind that Monarch is already ten years old.
    - RcouF1uZ4gsC 6y ago
      > But when you publish a paper on Monarch, aren't you giving away the core idea? The thing is Monarch is not the secret sauce that makes Google money. Every other company in the world could be running monarch, and Google revenue and market position would not be affected. Publicizing this is a win-win situation. The idea gets exposure and potentially other devs can be benefit, and Google gets some good PR as a place where devs get to work on cool stuff.
    - joshuamorton 6y ago
      [Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).
      - jldugger 6y ago
        The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
        jeffbee 6y ago
- justicezyx 6y ago
  Trust me, it's not that impressive if you want to use it in your dozens of machines cluster...
- tanilama 6y ago
  I doubt As impressive a the Planet-Scale sounds like, monitoring system has been essential in a lot of companies, big or small, and there are tons of companies dedicated to this problem domain. So it is not an untapped territory for many. Google solved a problem that is relevant to Google and its scale. The solution is cool, but it is only so because the problem exists in the first place. Without Google's problem, the solution will be seen as incredibly over-engineered and unnecessary.
jeffbee 6y ago
Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-years consumed per day, and a footnote in a paper that mentions how much CPU time Google Exacycle gives away to scientific computing every day. All of these calibration points are eye-opening.
- saddlerustle 6y ago
  Yeah the "supercomputer" ranking is a bit of a joke. Every mid-sized google dc would count as a top 10 supercomputer.
  - jankeymeulen 6y ago
    I work at one such "medium sized" Google DCs. Supercomputers are typically much more interconnected, whilst we have a much more traditional topology.
  - mhh__ 6y ago
    Isn't the whole point of a supercomputer that it isn't just a datacentre with an LED display on the front?
  - Aunche 6y ago
    Supercomputers are more about the network topology than raw processing power.
  - dekhn 6y ago
    supercomputers, by tradition, are tightly coupled in a way that Google datacenter servers aren't. The closest thing Google has to supercomputers are GPUs linked by high performance networks, and TPUs (which have their own custom toroidal mesh).
- vii 6y ago
  they also estimate close to a petabyte of RAM; if all the RAM is spent in leaves that is about 6-7GB per leaf. I don't know that we can say it would be unreasonable to have 1 core per leaf replica; presumably some leaves have low utilisation so it might make sense to share the core with another workload. From a capacity planning standpoint, I think they leave this open though they do indicate that they are sometimes CPU bound so don't try to compress beyond delta compression. That might suggest multiple cores per leaf. A single core design allows a very simple concurrency model, without having to worry about cache pingponging, false sharing, or myriad other issues. The parallelism is applied at higher layers, as there are multiple replicas for each leaf and obviously they can use many cores effectively overall. I don't see that the paper gives enough information to help us prune the design space here.
m0zg 6y ago
Question for googlers (and ex-googlers). Is there anything out there that's as convenient as /streamz, but in opensource form?
- yegle 6y ago
  If you are looking for /varz, https://prometheus.io https://prometheus.io If you really want /streamz with simple aggregation for distribution, description for metrics etc in an HTML page, I haven't encountered one yet.
  - m0zg 6y ago
    No, looking for streamz specifically. I'd like to be able to see how each node is doing and have automatic, zero-config aggregation in k8s a-la Borg/Monarch. The service I built was one of the first large users of Monarch at Google many years ago. All because I couldn't be bothered to learn Borgmon. :-)
- dvirsky 6y ago
  6-7 years ago I used to use Statsd+Graphite. Statsd is kinda like streamz (although it works in a completely different way, but from the app perspective it's similar), and Graphite is like Monarch. It had a pretty ugly UI, and I think today people use Grafana as the front-end instead.
- kccqzy 6y ago
  You should look into Riemann: http://riemann.io/ http://riemann.io/
- jrockway 6y ago
  Prometheus is as convenient as /streamz in the sense that it has service discovery, so you don't have to do any work to have your endpoints discovered and scraped. I haven't found anything as good as Monarch and the internal dashboard, though. Monarch always made sense to me, by not doing anything clever. For example, if you want to aggregate across different streams, you have to "align" them first, and the alignment is the operation that defines how to make up missing points. Other systems don't have that alignment step, and it causes me to struggle with everything else I want to do. As far as I can tell, Prometheus and InfluxDB have implicit alignment, but are never clear on what the rules are. At my last job, I collected some of the exact same metrics as I did at Google, and while I knew exactly how to get the charts I wanted with Monarch, I could never figure out how to get them with InfluxDB (I later found out that it simply didn't support my use case). (The exact use case was collecting counters from network devices opportunistically, and then generating a bandwidth chart for a section of the network. At Google it was easy; I could do the differencing to turn the packet counters into "bytes sent over the last X seconds", then align the streams so that data was available at each time, then aggregate over topology. With InfluxDB... the query language lets you express that, but it doesn't yield correct results. I complained about it on HN and the author of did a lot of handwaving about how what I want to do is wrong, or something, and so I just wrote my own thing instead.) The other thing I miss from Monarch is the default visualization for histograms. I have to manually reproduce that in Grafana with a series of manual queries like histogram_quantile(0.99, ...), histogram_quantile(0.90, ...), histogram_quantile(0.85, ...), ... where that was just the default visualization. Again, the open source world has some defaults, but I can't make heads or tails of what they are (try a default histogram visualization in Grafana)... Monarch just did the right thing by default. I miss it.
kccqzy 6y ago
As someone interested in PL theory, I've long found the query language exposed by Monarch surprisingly interesting (briefly discussed in section 5.1 but the description doesn't quite do it justice). It's a functional language, a breath of fresh air compared to "real programming languages" in use at Google like C++, Java, or Go. The most interesting idea is that its native data types are time series of integers, doubles, distributions of doubles, booleans, and tuples of the above. This means that the data you operate on intrinsically consist of many timestamped data points. It's easy to apply an operation to each point of the data, and it's also easy to apply operations on a rolling window, or on successive points. This makes the language have the feel of an array-based language, but even better because the elements are timestamped and the array can be sparse. Furthermore the presence of fields in each data point adds more dimensions to the level of aggregation (not just the inherent time-based). Now the language has the feel of a native multi-dimensional array language. It feels amazing to program in it. You can easily do sophisticated queries like figuring out how many standard deviations each task's RPC latency for a specific call is above or below all tasks' mean, for outlier detection.
- ethbro 6y ago
  Haven't read the paper yet, but could you expand on the tuple use? It seems like the odd person out in that list of primatives.
  - xyse53 6y ago
    They are used when joining timeseries.
    - jeffbee 6y ago
      A join creates a tuple but that's not the only way to use them. You can also just produce a tuple with an expression in the query such as (val(), 5, "dog") if you like. The whole language is documented here: https://cloud.google.com/monitoring/mql/reference https://cloud.google.com/monitoring/mql/reference
- joshuamorton 6y ago
ur-whale 6y ago
Given the size of the beast, I wonder what monitors it?
- ithkuil 6y ago
  A borgmon
- adammck 6y ago
  Monarch is monitored by a second instance of itself known as meta-monarch. It’s a fraction of the size, of course, but still mind-bogglingly large. Meta is in turn monitored by the main Monarch instance.

175 points by ngaut 6y ago

polskibus 6y ago

I wonder if Google will ever open source it.

kyrra 6y ago
Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed by a former Google SRE that has since returned). https://prometheus.io/ https://prometheus.io/
- gregschlom 6y ago
  > Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released Monarch might be an exception to that, though, because - as noted in the paper -, it has to be built without many dependencies, since almost everything else depends on it for monitoring and alerting. That being said, I'm sure it would be extremely difficult to open source now since it was never built with open sourcing in mind.
  - cperciva 6y ago
    It might not depend on many other services, but that doesn't preclude it depending on other code.
- fizixer 6y ago
  But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.
  - jeffbee 6y ago
    You could definitely build Monarch from this paper. It is very detailed. But keep in mind that Monarch is already ten years old.
  - RcouF1uZ4gsC 6y ago
    > But when you publish a paper on Monarch, aren't you giving away the core idea? The thing is Monarch is not the secret sauce that makes Google money. Every other company in the world could be running monarch, and Google revenue and market position would not be affected. Publicizing this is a win-win situation. The idea gets exposure and potentially other devs can be benefit, and Google gets some good PR as a place where devs get to work on cool stuff.
  - joshuamorton 6y ago
    [Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).
    - jldugger 6y ago
      The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
      - jeffbee 6y ago
justicezyx 6y ago
Trust me, it's not that impressive if you want to use it in your dozens of machines cluster...
tanilama 6y ago
I doubt As impressive a the Planet-Scale sounds like, monitoring system has been essential in a lot of companies, big or small, and there are tons of companies dedicated to this problem domain. So it is not an untapped territory for many. Google solved a problem that is relevant to Google and its scale. The solution is cool, but it is only so because the problem exists in the first place. Without Google's problem, the solution will be seen as incredibly over-engineered and unnecessary.

jeffbee 6y ago

Interesting that this paper contains hard numbers hinting at Google's absolute scale. They say Monarch has 144000 leaves. Even if each leaf is assigned only 1 CPU core -- which is probably a low estimate because who would do that? -- that makes Google's monitoring stack a Top 100 supercomputer. The only other places I've seen Google give out hard numbers were a presentation by Jeff Dean mentioning map-reduce core-years consumed per day, and a footnote in a paper that mentions how much CPU time Google Exacycle gives away to scientific computing every day. All of these calibration points are eye-opening.

saddlerustle 6y ago
Yeah the "supercomputer" ranking is a bit of a joke. Every mid-sized google dc would count as a top 10 supercomputer.
- jankeymeulen 6y ago
  I work at one such "medium sized" Google DCs. Supercomputers are typically much more interconnected, whilst we have a much more traditional topology.
- mhh__ 6y ago
  Isn't the whole point of a supercomputer that it isn't just a datacentre with an LED display on the front?
- Aunche 6y ago
  Supercomputers are more about the network topology than raw processing power.
- dekhn 6y ago
  supercomputers, by tradition, are tightly coupled in a way that Google datacenter servers aren't. The closest thing Google has to supercomputers are GPUs linked by high performance networks, and TPUs (which have their own custom toroidal mesh).
vii 6y ago
they also estimate close to a petabyte of RAM; if all the RAM is spent in leaves that is about 6-7GB per leaf. I don't know that we can say it would be unreasonable to have 1 core per leaf replica; presumably some leaves have low utilisation so it might make sense to share the core with another workload. From a capacity planning standpoint, I think they leave this open though they do indicate that they are sometimes CPU bound so don't try to compress beyond delta compression. That might suggest multiple cores per leaf. A single core design allows a very simple concurrency model, without having to worry about cache pingponging, false sharing, or myriad other issues. The parallelism is applied at higher layers, as there are multiple replicas for each leaf and obviously they can use many cores effectively overall. I don't see that the paper gives enough information to help us prune the design space here.

m0zg 6y ago

Question for googlers (and ex-googlers). Is there anything out there that's as convenient as /streamz, but in opensource form?

yegle 6y ago
If you are looking for /varz, https://prometheus.io https://prometheus.io If you really want /streamz with simple aggregation for distribution, description for metrics etc in an HTML page, I haven't encountered one yet.
- m0zg 6y ago
  No, looking for streamz specifically. I'd like to be able to see how each node is doing and have automatic, zero-config aggregation in k8s a-la Borg/Monarch. The service I built was one of the first large users of Monarch at Google many years ago. All because I couldn't be bothered to learn Borgmon. :-)
dvirsky 6y ago
6-7 years ago I used to use Statsd+Graphite. Statsd is kinda like streamz (although it works in a completely different way, but from the app perspective it's similar), and Graphite is like Monarch. It had a pretty ugly UI, and I think today people use Grafana as the front-end instead.
kccqzy 6y ago
You should look into Riemann: http://riemann.io/ http://riemann.io/
jrockway 6y ago
Prometheus is as convenient as /streamz in the sense that it has service discovery, so you don't have to do any work to have your endpoints discovered and scraped. I haven't found anything as good as Monarch and the internal dashboard, though. Monarch always made sense to me, by not doing anything clever. For example, if you want to aggregate across different streams, you have to "align" them first, and the alignment is the operation that defines how to make up missing points. Other systems don't have that alignment step, and it causes me to struggle with everything else I want to do. As far as I can tell, Prometheus and InfluxDB have implicit alignment, but are never clear on what the rules are. At my last job, I collected some of the exact same metrics as I did at Google, and while I knew exactly how to get the charts I wanted with Monarch, I could never figure out how to get them with InfluxDB (I later found out that it simply didn't support my use case). (The exact use case was collecting counters from network devices opportunistically, and then generating a bandwidth chart for a section of the network. At Google it was easy; I could do the differencing to turn the packet counters into "bytes sent over the last X seconds", then align the streams so that data was available at each time, then aggregate over topology. With InfluxDB... the query language lets you express that, but it doesn't yield correct results. I complained about it on HN and the author of did a lot of handwaving about how what I want to do is wrong, or something, and so I just wrote my own thing instead.) The other thing I miss from Monarch is the default visualization for histograms. I have to manually reproduce that in Grafana with a series of manual queries like histogram_quantile(0.99, ...), histogram_quantile(0.90, ...), histogram_quantile(0.85, ...), ... where that was just the default visualization. Again, the open source world has some defaults, but I can't make heads or tails of what they are (try a default histogram visualization in Grafana)... Monarch just did the right thing by default. I miss it.

kccqzy 6y ago

As someone interested in PL theory, I've long found the query language exposed by Monarch surprisingly interesting (briefly discussed in section 5.1 but the description doesn't quite do it justice). It's a functional language, a breath of fresh air compared to "real programming languages" in use at Google like C++, Java, or Go. The most interesting idea is that its native data types are time series of integers, doubles, distributions of doubles, booleans, and tuples of the above. This means that the data you operate on intrinsically consist of many timestamped data points. It's easy to apply an operation to each point of the data, and it's also easy to apply operations on a rolling window, or on successive points. This makes the language have the feel of an array-based language, but even better because the elements are timestamped and the array can be sparse. Furthermore the presence of fields in each data point adds more dimensions to the level of aggregation (not just the inherent time-based). Now the language has the feel of a native multi-dimensional array language. It feels amazing to program in it. You can easily do sophisticated queries like figuring out how many standard deviations each task's RPC latency for a specific call is above or below all tasks' mean, for outlier detection.

ethbro 6y ago
Haven't read the paper yet, but could you expand on the tuple use? It seems like the odd person out in that list of primatives.
- xyse53 6y ago
  They are used when joining timeseries.
  - jeffbee 6y ago
    A join creates a tuple but that's not the only way to use them. You can also just produce a tuple with an expression in the query such as (val(), 5, "dog") if you like. The whole language is documented here: https://cloud.google.com/monitoring/mql/reference https://cloud.google.com/monitoring/mql/reference
joshuamorton 6y ago

ur-whale 6y ago

Given the size of the beast, I wonder what monitors it?

ithkuil 6y ago
A borgmon
adammck 6y ago
Monarch is monitored by a second instance of itself known as meta-monarch. It’s a fraction of the size, of course, but still mind-bogglingly large. Meta is in turn monitored by the main Monarch instance.