6 ms·

Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technolog

by kyrra 6y ago

Googler, opinions are my own. Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released. It tends to take a great amount of effort to open source anything at Google because of all the internal dependencies you have. It should be noted that Prometheus is an open source recreation of the precursor to monarch (developed by a former Google SRE that has since returned). https://prometheus.io/ https://prometheus.io/

gregschlom 6y ago
> Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released Monarch might be an exception to that, though, because - as noted in the paper -, it has to be built without many dependencies, since almost everything else depends on it for monitoring and alerting. That being said, I'm sure it would be extremely difficult to open source now since it was never built with open sourcing in mind.
- cperciva 6y ago
  It might not depend on many other services, but that doesn't preclude it depending on other code.
fizixer 6y ago
But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.
- jeffbee 6y ago
  You could definitely build Monarch from this paper. It is very detailed. But keep in mind that Monarch is already ten years old.
- RcouF1uZ4gsC 6y ago
  > But when you publish a paper on Monarch, aren't you giving away the core idea? The thing is Monarch is not the secret sauce that makes Google money. Every other company in the world could be running monarch, and Google revenue and market position would not be affected. Publicizing this is a win-win situation. The idea gets exposure and potentially other devs can be benefit, and Google gets some good PR as a place where devs get to work on cool stuff.
- joshuamorton 6y ago
  [Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).
  - jldugger 6y ago
    The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
    - jeffbee 6y ago
      Stackdriver/Google Cloud monitoring is backed by Monarch, so if you want the flavor of a Monarch distribution-valued metric, see the docs: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TypedValue#Distribution https://cloud.google.com/monitoring/api/ref_v3/rest/v3/Typed... Since the distribution is represented by a CDF of buckets, there's no guarantee that you'll get an accurate representation of the median or any other quantile. On the other hand you'll get an exact average.
    - scottlamb 6y ago
      Monarch has distributions with predetermined bucket boundaries. These are indeed very useful. Pet peeve: it can calculate and graph something it calls a quantile. But if the value is in the middle of a large bucket, it will just interpolate or something and the result will be terribly misleading. It'd be much better if it gave lower/upper bounds. I try to use distributions via questions of the form "what fraction of values are less than / greater than N [which I've verified is a bucket boundary]?". This gives you an answer you can trust.
- einpoklum 6y ago
  > aren't you giving away the core idea? The core idea is a distributed time-series DBMS... not much to give away there. It "gives away" some architectural novelty, but it's not the solution to P=NP. These papers typically describe engineering feats more than they do a revoluntionary idea.
- cortesoft 6y ago
  They aren’t saying it is difficult without giving away trade secrets, they are saying it is difficult because the software has dependencies on internal services, which themselves have dependencies on other internal services. Basically, in order to run it you need to also run the entirety of Google’s tech stack. This doesn’t work for an open source project. They aren’t worried about ‘giving away the idea’, they just don’t have an easy technical way to open source just the one component.

by kyrra 6y ago

gregschlom 6y ago

> Open sourcing anything at Google tends to be rather difficult because much of the technology we build is built on other technology that has not been released Monarch might be an exception to that, though, because - as noted in the paper -, it has to be built without many dependencies, since almost everything else depends on it for monitoring and alerting. That being said, I'm sure it would be extremely difficult to open source now since it was never built with open sourcing in mind.

cperciva 6y ago
It might not depend on many other services, but that doesn't preclude it depending on other code.

fizixer 6y ago

But when you publish a paper on Monarch, aren't you giving away the core idea? In the case of PageRank, at least it was published after Google had already dominated the search space, i.e., many years after the conception, implementation, and utilization. I wonder if Google papers are internally reviewed before publishing so as to make sure only partial information is revealed, and not the secret sauce.

jeffbee 6y ago
You could definitely build Monarch from this paper. It is very detailed. But keep in mind that Monarch is already ten years old.
RcouF1uZ4gsC 6y ago
> But when you publish a paper on Monarch, aren't you giving away the core idea? The thing is Monarch is not the secret sauce that makes Google money. Every other company in the world could be running monarch, and Google revenue and market position would not be affected. Publicizing this is a win-win situation. The idea gets exposure and potentially other devs can be benefit, and Google gets some good PR as a place where devs get to work on cool stuff.
joshuamorton 6y ago
[Also a Googler, opinons mine] In addition to what the other people are saying: there are some limitations to monarch (or really, the data upload path) that are quite annoying, so monarch isn't even necessarily the "best". It's just very good. There are ways to improve it. The issue is, even if you give away the secret sauce that doesn't really help with making the secret sauce scale or whatnot, nor does anyone that isn't a large cloud provider need a custom solution like monarch. Prometheus or Datadog work fine for everyone else. This might be interesting reading for those companies, but also it might not be, because those can't be as centralized as monarch is (consider if prometheus had an API and ran a centralized cluster of data-ingestion servers, and you made time series to that global, Prometheus-owned, cluster).
- jldugger 6y ago
  The one thing I really want (which apparently Monarch has) is histogram retention. I'm often called upon to summarize service latency as global p50 and p95, and at the sheer volume of data we have, we aggregate that metric. Thus I am left calculating an average of p95s, which isn't super useful. To the best of my knowledge, nothing else in the market does that.
  - jeffbee 6y ago
    Stackdriver/Google Cloud monitoring is backed by Monarch, so if you want the flavor of a Monarch distribution-valued metric, see the docs: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TypedValue#Distribution https://cloud.google.com/monitoring/api/ref_v3/rest/v3/Typed... Since the distribution is represented by a CDF of buckets, there's no guarantee that you'll get an accurate representation of the median or any other quantile. On the other hand you'll get an exact average.
  - scottlamb 6y ago
    Monarch has distributions with predetermined bucket boundaries. These are indeed very useful. Pet peeve: it can calculate and graph something it calls a quantile. But if the value is in the middle of a large bucket, it will just interpolate or something and the result will be terribly misleading. It'd be much better if it gave lower/upper bounds. I try to use distributions via questions of the form "what fraction of values are less than / greater than N [which I've verified is a bucket boundary]?". This gives you an answer you can trust.
einpoklum 6y ago
> aren't you giving away the core idea? The core idea is a distributed time-series DBMS... not much to give away there. It "gives away" some architectural novelty, but it's not the solution to P=NP. These papers typically describe engineering feats more than they do a revoluntionary idea.
cortesoft 6y ago
They aren’t saying it is difficult without giving away trade secrets, they are saying it is difficult because the software has dependencies on internal services, which themselves have dependencies on other internal services. Basically, in order to run it you need to also run the entirety of Google’s tech stack. This doesn’t work for an open source project. They aren’t worried about ‘giving away the idea’, they just don’t have an easy technical way to open source just the one component.