4 ms·

I wonder if Google will ever open source it.

kyrra 6y ago
Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed by a former Google SRE that has since returned). https://prometheus.io/ https://prometheus.io/
- gregschlom 6y ago
  > Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released Monarch might be an exception to that, though, because - as noted in the paper -, it has to be built without many dependencies, since almost everything else depends on it for monitoring and alerting. That being said, I'm sure it would be extremely difficult to open source now since it was never built with open sourcing in mind.
  - cperciva 6y ago
    It might not depend on many other services, but that doesn't preclude it depending on other code.
- fizixer 6y ago
  But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.
  - jeffbee 6y ago
    You could definitely build Monarch from this paper. It is very detailed. But keep in mind that Monarch is already ten years old.
  - RcouF1uZ4gsC 6y ago
    > But when you publish a paper on Monarch, aren't you giving away the core idea? The thing is Monarch is not the secret sauce that makes Google money. Every other company in the world could be running monarch, and Google revenue and market position would not be affected. Publicizing this is a win-win situation. The idea gets exposure and potentially other devs can be benefit, and Google gets some good PR as a place where devs get to work on cool stuff.
  - joshuamorton 6y ago
    [Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).
    - jldugger 6y ago
      The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
      - jeffbee 6y ago
justicezyx 6y ago
Trust me, it's not that impressive if you want to use it in your dozens of machines cluster...
tanilama 6y ago
I doubt As impressive a the Planet-Scale sounds like, monitoring system has been essential in a lot of companies, big or small, and there are tons of companies dedicated to this problem domain. So it is not an untapped territory for many. Google solved a problem that is relevant to Google and its scale. The solution is cool, but it is only so because the problem exists in the first place. Without Google's problem, the solution will be seen as incredibly over-engineered and unnecessary.

Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed by a former Google SRE that has since returned). https://prometheus.io/ https://prometheus.io/

gregschlom 6y ago
> Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released Monarch might be an exception to that, though, because - as noted in the paper -, it has to be built without many dependencies, since almost everything else depends on it for monitoring and alerting. That being said, I'm sure it would be extremely difficult to open source now since it was never built with open sourcing in mind.
- cperciva 6y ago
  It might not depend on many other services, but that doesn't preclude it depending on other code.
fizixer 6y ago
But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.
- jeffbee 6y ago
  You could definitely build Monarch from this paper. It is very detailed. But keep in mind that Monarch is already ten years old.
- RcouF1uZ4gsC 6y ago
  > But when you publish a paper on Monarch, aren't you giving away the core idea? The thing is Monarch is not the secret sauce that makes Google money. Every other company in the world could be running monarch, and Google revenue and market position would not be affected. Publicizing this is a win-win situation. The idea gets exposure and potentially other devs can be benefit, and Google gets some good PR as a place where devs get to work on cool stuff.
- joshuamorton 6y ago
  [Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).
  - jldugger 6y ago
    The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
    - jeffbee 6y ago

justicezyx 6y ago

Trust me, it's not that impressive if you want to use it in your dozens of machines cluster...

tanilama 6y ago

I doubt As impressive a the Planet-Scale sounds like, monitoring system has been essential in a lot of companies, big or small, and there are tons of companies dedicated to this problem domain. So it is not an untapped territory for many. Google solved a problem that is relevant to Google and its scale. The solution is cool, but it is only so because the problem exists in the first place. Without Google's problem, the solution will be seen as incredibly over-engineered and unnecessary.