6 ms·

I keep seeing that Borgmon is deprecated but no references to what the replacement is like. Can someone please share even a high level summary? Is the replace

I keep seeing that Borgmon is deprecated but no references to what the replacement is like. Can someone please share even a high level summary? Is the replacement a high cardinality event store like Facebook’s Scuba [0]? [0] https://research.fb.com/wp-content/uploads/2016/11/scuba-diving-into-data-at-facebook.pdf https://research.fb.com/wp-content/uploads/2016/11/scuba-div...

londons_explore 8y ago
The replacement (Monarch) is similar to borgmon except: * All metrics have an associated type. Eg. Response time (milliseconds). That's great because units for derived metrics can be dynamically computed. Eg. Bytes/second. * The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed (eg. 95 percent latency across 1000 tasks can be calculated in real-time). * The config system is a mess and nobody likes it. Borgmon uses a DSL which is obscure, but almost identical to Prometheus. Monarch has various different config frontends (mostly around the idea of running code to produce an expanded protobuffer config) which all suck. Luckily because there isn't a strong requirement for rules to aggregate data, you don't need much config for most services - just say "scrape everything and keep it for a year". * There are "levels" of storage at different speeds. In memory, on disk, etc. You have to configure where to put what data. You can also downsample (eg. Change scrape interval to 5 mins after a week). * Metric names follow a directory-like heirachy. Since tasks can easily have 10k exported metrics, that's pretty important. No need to scrape the ones that aren't relevant. * It has a shinier UI. * It has support for exemplars. So to answer the question "Give me an example of a request which saw this high request latency". With not much added code to the monitored service, a small number of exemplars are captured and aggregated in a way that median and outlier exemplars are available. They're super useful for finding out the cause of random slow performance. * It is run as a service. Rather than code that every team has to run, the new thing is a single instance for all teams in Google. That in turn means it can be more complex, have more dependencies, etc, without being a burden on the user.
- vthriller 8y ago
  > The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed I'm having hard time figuring out how query language might affect exectuion time. Granted, I've never seen borgmon in action, so the first question should probably be: how different its query language is from what Prometheus offers? If it's more on imperative side than declarative one then your point might actually make sense to me as it is. > You can also downsample (eg. Change scrape interval to 5 mins after a week). Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does?
  - shereadsthenews 8y ago
    I think what he is saying is that it is easier to do something ad hoc in Monarch since it is a global service with unified storage and retrieval. A Googler can join the data of Gmail and Ads easily if they want, whereas combining the borgmon data of those services is either not possible or it involves downloading CSVs and performing the join externally. > Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does? All of these methods are available[1]. Stackdriver Monitioring is the public face of Monarch. 1: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies?_ga=2.5326451.-1585613562.1554906215#reducer https://cloud.google.com/monitoring/api/ref_v3/rest/v3/proje...
    - puzzle 8y ago
      No, I think it's strictly about where aggregations and other computations occur. You hint at that when you mention CSV files. Borgmon keeps X hours of data in memory and looks up anything older from TSDB. I never understood how to aggregate data from the same service, after the fact, in a reliable fashion. Or if it was possible at all. I believe it was only working when looking at data in memory. That's why people had rules to aggregate data at multiple levels as it was being ingested, then persisting that into TSDB. I don't remember the latter being much more sophisticated than a bunch of files in GFS with the timeseries and some metadata, either. Monarch, I think, is based on Bigtable. It has a richer API. It might even use BT coprocessors to perform computations closer to the where the raw data is stored, rather than in the Monarch frontends. (I haven't watched the public talk, so I'm just going from vague recollection.)

londons_explore 8y ago

The replacement (Monarch) is similar to borgmon except: * All metrics have an associated type. Eg. Response time (milliseconds). That's great because units for derived metrics can be dynamically computed. Eg. Bytes/second. * The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed (eg. 95 percent latency across 1000 tasks can be calculated in real-time). * The config system is a mess and nobody likes it. Borgmon uses a DSL which is obscure, but almost identical to Prometheus. Monarch has various different config frontends (mostly around the idea of running code to produce an expanded protobuffer config) which all suck. Luckily because there isn't a strong requirement for rules to aggregate data, you don't need much config for most services - just say "scrape everything and keep it for a year". * There are "levels" of storage at different speeds. In memory, on disk, etc. You have to configure where to put what data. You can also downsample (eg. Change scrape interval to 5 mins after a week). * Metric names follow a directory-like heirachy. Since tasks can easily have 10k exported metrics, that's pretty important. No need to scrape the ones that aren't relevant. * It has a shinier UI. * It has support for exemplars. So to answer the question "Give me an example of a request which saw this high request latency". With not much added code to the monitored service, a small number of exemplars are captured and aggregated in a way that median and outlier exemplars are available. They're super useful for finding out the cause of random slow performance. * It is run as a service. Rather than code that every team has to run, the new thing is a single instance for all teams in Google. That in turn means it can be more complex, have more dependencies, etc, without being a burden on the user.

vthriller 8y ago
> The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed I'm having hard time figuring out how query language might affect exectuion time. Granted, I've never seen borgmon in action, so the first question should probably be: how different its query language is from what Prometheus offers? If it's more on imperative side than declarative one then your point might actually make sense to me as it is. > You can also downsample (eg. Change scrape interval to 5 mins after a week). Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does?
- shereadsthenews 8y ago
  I think what he is saying is that it is easier to do something ad hoc in Monarch since it is a global service with unified storage and retrieval. A Googler can join the data of Gmail and Ads easily if they want, whereas combining the borgmon data of those services is either not possible or it involves downloading CSVs and performing the join externally. > Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does? All of these methods are available[1]. Stackdriver Monitioring is the public face of Monarch. 1: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies?_ga=2.5326451.-1585613562.1554906215#reducer https://cloud.google.com/monitoring/api/ref_v3/rest/v3/proje...
  - puzzle 8y ago
    No, I think it's strictly about where aggregations and other computations occur. You hint at that when you mention CSV files. Borgmon keeps X hours of data in memory and looks up anything older from TSDB. I never understood how to aggregate data from the same service, after the fact, in a reliable fashion. Or if it was possible at all. I believe it was only working when looking at data in memory. That's why people had rules to aggregate data at multiple levels as it was being ingested, then persisting that into TSDB. I don't remember the latter being much more sophisticated than a bunch of files in GFS with the timeseries and some metadata, either. Monarch, I think, is based on Bigtable. It has a richer API. It might even use BT coprocessors to perform computations closer to the where the raw data is stored, rather than in the Monarch frontends. (I haven't watched the public talk, so I'm just going from vague recollection.)