4 ms·

Thanks George, and apologies for missing this comment on my first scan through this page. Your Youtube talk is lined up for viewing later today. We're using p

by Jedd 4y ago

Thanks George, and apologies for missing this comment on my first scan through this page. Your Youtube talk is lined up for viewing later today. We're using prom + cortex/mimir. With ~30-60k hosts + at least that figure again for other endpoints (k8s, snmp, etc), so we can get away with semi-manual sharding (os, geo, env, etc). We're happy with 1m polling, which is still maybe 50 packets per query, but no persistent conns held open to agents. I'm guessing your TCP issues were exacerbated by a much high polling frequency requirement? You come back to persistent connections a lot, so this sounds like a bespoke agent, and/or the problem was not (mostly) a connection establish/tear-down performance issue? The external discovery service - I assume an in-house, and now long disappeared and not well publicly described system? ;) We're looking at NodeRED to fill that gap, so it also becomes a critical component, but the absence only bites at agent restart. We're pondering wrapping some code around the agents to be smarter about dealing with a non-responsive config service. (During a major incident we have to assume a lot of things will be absent and/or restarting.) The concerns around incoming conns to their apps, it sounds like those same teams you were dealing with ended up having to instrument their code with something from you anyway -- was it the DoS risk they were concerned about?

gttalbot 4y ago
It was more that they would rather send Monarch an RPC than be connected to. Not everyone wants e.g. an HTTP server in their process. For example maybe they are security sensitive, or have a limited memory envelope, or other reasons.
- gttalbot 4y ago
  And yeah, everyone used the same instrumentation library, which could be used in a lot of different contexts, sometimes surprising!
  - Jedd 4y ago
    Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint. Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom performance monitoring library that was integrated into in-house applications.
    - Too 4y ago
      The rest of us can only hope Opentelemetry becomes more widely adopted. They have put in a lot of effort in decoupling the application instrumentation from the monitoring solution, to allow more rich instrumentation than just a prom-alike endpoint.
      - gttalbot 4y ago
        Yeah I hope OpenTelemetry works out as well.
    - gttalbot 4y ago
      Yeah that's a tough one, and it was indeed a luxury to be able to have a uniform instrumentation library.
      - Jedd 4y ago
        Re-reading my comment, it may have been worded badly. The phrase 'you aren't Google' is true for 99.9% of us. We all get to fix the problems in front of us, was my point. And at that scale you've got unique problems, but also an architecture, imperative, and most importantly an ethos that lets you solve them in this fashion. I was more reflecting on the (actually pretty fine) tools available to SREs caring for off-the-shelf OS's and products, and a little on the whole 'we keep coming full circle' thing. Anyway, I very much appreciate the insights.

by Jedd 4y ago

gttalbot 4y ago

It was more that they would rather send Monarch an RPC than be connected to. Not everyone wants e.g. an HTTP server in their process. For example maybe they are security sensitive, or have a limited memory envelope, or other reasons.

gttalbot 4y ago
And yeah, everyone used the same instrumentation library, which could be used in a lot of different contexts, sometimes surprising!
- Jedd 4y ago
  Again, a highly envious feature of a large organisation with almost exclusively bespoke applications that can port & integrate custom libraries directly into applications. Us little people have to contend with mostly black box applications, or occasionally native instrumented with at best a prom-alike endpoint. Amusingly in the pre-web 1990's, at Telstra (Australia telco) we also developed & implemented a custom performance monitoring library that was integrated into in-house applications.
  - Too 4y ago
    The rest of us can only hope Opentelemetry becomes more widely adopted. They have put in a lot of effort in decoupling the application instrumentation from the monitoring solution, to allow more rich instrumentation than just a prom-alike endpoint.
    - gttalbot 4y ago
      Yeah I hope OpenTelemetry works out as well.
  - gttalbot 4y ago
    Yeah that's a tough one, and it was indeed a luxury to be able to have a uniform instrumentation library.
    - Jedd 4y ago
      Re-reading my comment, it may have been worded badly. The phrase 'you aren't Google' is true for 99.9% of us. We all get to fix the problems in front of us, was my point. And at that scale you've got unique problems, but also an architecture, imperative, and most importantly an ethos that lets you solve them in this fashion. I was more reflecting on the (actually pretty fine) tools available to SREs caring for off-the-shelf OS's and products, and a little on the whole 'we keep coming full circle' thing. Anyway, I very much appreciate the insights.