3 ms·
So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities. We're diving into OTEL, and the registration / discovery challenges don't
by Jedd 4y ago
So, much like a Zabbix agent, with both active (push) & passive (pull) capabilities.
We're diving into OTEL, and the registration / discovery challenges don't seem to have any kind of best-practice consensus out there. We're looking at NodeRED (telegraf agent can query from same at startup) but it brings its own challenges.
I haven't read the full paper, but do you know if the push model was revisited mostly for auto-registration / discovery, or performance bottlenecks at the server, or some other concern?
Typically for us, once we've got the hard part - an entity registered - we're happy with pull only. A no-response from a prod end-point is an automatic critical. I guess at their scale there's more nuance around one or more agents being non-responsive.
EDIT: Oh, there's not much in the paper on the subject, as it happens. And yes, it's vanilla discovery woes.
"Push-based data collection improves system robustness while simplifying system architecture. Early versions of Monarch discovered monitored entities and “pulled” monitoring data by querying the monitored entity.
"This required setting up discovery services and proxies, complicating system architecture and negatively impacting overall scalability. Push-based collection, where entities simply send their data to Monarch, eliminates these dependencies."
- gttalbot 4y agoAlso, I gave this talk a couple of years ago, though I'm not sure how deeply I talked about collection models. https://youtu.be/2mw12B7W7RI https://youtu.be/2mw12B7W7RI
- lokar 4y agoPush can help a bit, but you still have to know which endpoints you expect to hear from (if you want to detect they are missing).
- gttalbot 4y agoIf you can put these into the metrics system as metrics themselves, then join against them, that works. (IIRC handling all the various things that could be happening in a data center like drains, planned & unplanned downtime, etc., it's more complex than a simple join.)
- gttalbot 4y agoSee my comment below, on the challenges of pull based collection on Monarch. There were many. I can answer questions, if that's helpful.