5 ms·

It is almost the other way around. Many of these open-source projects, have roots with engineers leaving these big companies and wanting to use the same tools,

It is almost the other way around. Many of these open-source projects, have roots with engineers leaving these big companies and wanting to use the same tools, so they create them. Or, by these big companies releasing papers and then folks creating open-source versions. Here's some examples of the latter: Dapper (w/ OpenTracing) [1], Spanner (w/ CockroachDB) [2], Borg (w/ Mesos) [3], Dremel (w/ Drill) [4]. I'm sure there are tons more examples but these are just off the top of my head. [1] https://ai.google/research/pubs/pub36356.pdf https://ai.google/research/pubs/pub36356.pdf [2] https://ai.google/research/pubs/pub39966.pdf https://ai.google/research/pubs/pub39966.pdf [3] https://ai.google/research/pubs/pub43438.pdf https://ai.google/research/pubs/pub43438.pdf [4] https://ai.google/research/pubs/pub36632.pdf https://ai.google/research/pubs/pub36632.pdf

kyrra 8y ago
Prometheus is borgmon. Which is a bit funny as it's used by k8s now. It's also funny as the author left Google, wrote it in 2 years for SoundCloud, then went back to Google. https://prometheus.io https://prometheus.io
- manojlds 8y ago
  Why is the part that k8s uses it funny? That would have been one of the goals right? K8s is like lot of these projects except that it was created as a open source version of borg from within Google.
  - shereadsthenews 8y ago
    It’s funny because inside google borgmon is deprecated for new projects and has been since nine years at least. Of course google has stretched “deprecated” to its limits.
    - QuercusMax 8y ago
      As long as you don't have to write promethus configs in the horrendous borgmon language....
      - boulos 8y ago
        Fwiw, I like Borgmon, but I also like APL derivatives :). My sadness is that Borgmon’s type system is quite poor, but like many “product of necessity” systems it actually worked incredibly well.
    - jaytaylor 8y ago
      What are they supposed to use instead of borgmon?
      - jhalstead 8y ago
        Monarch. https://youtu.be/LlvJdK1xsl4 https://youtu.be/LlvJdK1xsl4
    - wikibob 8y ago
      I keep seeing that Borgmon is deprecated but no references to what the replacement is like. Can someone please share even a high level summary? Is the replacement a high cardinality event store like Facebook’s Scuba [0]? [0] https://research.fb.com/wp-content/uploads/2016/11/scuba-diving-into-data-at-facebook.pdf https://research.fb.com/wp-content/uploads/2016/11/scuba-div...
      - londons_explore 8y ago
        The replacement (Monarch) is similar to borgmon except: * All metrics have an associated type. Eg. Response time (milliseconds). That's great because units for derived metrics can be dynamically computed. Eg. Bytes/second. * The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed (eg. 95 percent latency across 1000 tasks can be calculated in real-time). * The config system is a mess and nobody likes it. Borgmon uses a DSL which is obscure, but almost identical to Prometheus. Monarch has various different config frontends (mostly around the idea of running code to produce an expanded protobuffer config) which all suck. Luckily because there isn't a strong requirement for rules to aggregate data, you don't need much config for most services - just say "scrape everything and keep it for a year". * There are "levels" of storage at different speeds. In memory, on disk, etc. You have to configure where to put what data. You can also downsample (eg. Change scrape interval to 5 mins after a week). * Metric names follow a directory-like heirachy. Since tasks can easily have 10k exported metrics, that's pretty important. No need to scrape the ones that aren't relevant. * It has a shinier UI. * It has support for exemplars. So to answer the question "Give me an example of a request which saw this high request latency". With not much added code to the monitored service, a small number of exemplars are captured and aggregated in a way that median and outlier exemplars are available. They're super useful for finding out the cause of random slow performance. * It is run as a service. Rather than code that every team has to run, the new thing is a single instance for all teams in Google. That in turn means it can be more complex, have more dependencies, etc, without being a burden on the user.
        vthriller 8y ago
        > The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed I'm having hard time figuring out how query language might affect exectuion time. Granted, I've never seen borgmon in action, so the first question should probably be: how different its query language is from what Prometheus offers? If it's more on imperative side than declarative one then your point might actually make sense to me as it is. > You can also downsample (eg. Change scrape interval to 5 mins after a week). Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does?
    - endymi0n 8y ago
      A close source told me only half joking that at Google, when you start any new service, you‘ve got the choice between about a dozen backing services, of which half are deprecated and the other half are not officially supported yet...
      - JOnAgain 8y ago
        I don’t think he was joking at all
      - puzzle 8y ago
        The actual proverb, by an engineer (Paul C), is "there are two ways of doing things at Google: the deprecated one, and the one that doesn't work yet". Usually the deprecation occurred when a service outlived the original requirements and was no longer a great fit or not very usable. A good example were Babysitter and GWQ, which were eventually obsoleted by Borg.
  - auiya 8y ago
    I feel like much of the following conversation violates NDAs signed as a condition of employment there. Is this not the case?

kyrra 8y ago

Prometheus is borgmon. Which is a bit funny as it's used by k8s now. It's also funny as the author left Google, wrote it in 2 years for SoundCloud, then went back to Google. https://prometheus.io https://prometheus.io

manojlds 8y ago
Why is the part that k8s uses it funny? That would have been one of the goals right? K8s is like lot of these projects except that it was created as a open source version of borg from within Google.
- shereadsthenews 8y ago
  It’s funny because inside google borgmon is deprecated for new projects and has been since nine years at least. Of course google has stretched “deprecated” to its limits.
  - QuercusMax 8y ago
    As long as you don't have to write promethus configs in the horrendous borgmon language....
    - boulos 8y ago
      Fwiw, I like Borgmon, but I also like APL derivatives :). My sadness is that Borgmon’s type system is quite poor, but like many “product of necessity” systems it actually worked incredibly well.
  - jaytaylor 8y ago
    What are they supposed to use instead of borgmon?
    - jhalstead 8y ago
      Monarch. https://youtu.be/LlvJdK1xsl4 https://youtu.be/LlvJdK1xsl4
  - wikibob 8y ago
    I keep seeing that Borgmon is deprecated but no references to what the replacement is like. Can someone please share even a high level summary? Is the replacement a high cardinality event store like Facebook’s Scuba [0]? [0] https://research.fb.com/wp-content/uploads/2016/11/scuba-diving-into-data-at-facebook.pdf https://research.fb.com/wp-content/uploads/2016/11/scuba-div...
    - londons_explore 8y ago
      The replacement (Monarch) is similar to borgmon except: * All metrics have an associated type. Eg. Response time (milliseconds). That's great because units for derived metrics can be dynamically computed. Eg. Bytes/second. * The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed (eg. 95 percent latency across 1000 tasks can be calculated in real-time). * The config system is a mess and nobody likes it. Borgmon uses a DSL which is obscure, but almost identical to Prometheus. Monarch has various different config frontends (mostly around the idea of running code to produce an expanded protobuffer config) which all suck. Luckily because there isn't a strong requirement for rules to aggregate data, you don't need much config for most services - just say "scrape everything and keep it for a year". * There are "levels" of storage at different speeds. In memory, on disk, etc. You have to configure where to put what data. You can also downsample (eg. Change scrape interval to 5 mins after a week). * Metric names follow a directory-like heirachy. Since tasks can easily have 10k exported metrics, that's pretty important. No need to scrape the ones that aren't relevant. * It has a shinier UI. * It has support for exemplars. So to answer the question "Give me an example of a request which saw this high request latency". With not much added code to the monitored service, a small number of exemplars are captured and aggregated in a way that median and outlier exemplars are available. They're super useful for finding out the cause of random slow performance. * It is run as a service. Rather than code that every team has to run, the new thing is a single instance for all teams in Google. That in turn means it can be more complex, have more dependencies, etc, without being a burden on the user.
      - vthriller 8y ago
        > The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed I'm having hard time figuring out how query language might affect exectuion time. Granted, I've never seen borgmon in action, so the first question should probably be: how different its query language is from what Prometheus offers? If it's more on imperative side than declarative one then your point might actually make sense to me as it is. > You can also downsample (eg. Change scrape interval to 5 mins after a week). Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does?
  - endymi0n 8y ago
    A close source told me only half joking that at Google, when you start any new service, you‘ve got the choice between about a dozen backing services, of which half are deprecated and the other half are not officially supported yet...
    - JOnAgain 8y ago
      I don’t think he was joking at all
    - puzzle 8y ago
      The actual proverb, by an engineer (Paul C), is "there are two ways of doing things at Google: the deprecated one, and the one that doesn't work yet". Usually the deprecation occurred when a service outlived the original requirements and was no longer a great fit or not very usable. A good example were Babysitter and GWQ, which were eventually obsoleted by Borg.
- auiya 8y ago
  I feel like much of the following conversation violates NDAs signed as a condition of employment there. Is this not the case?