3 ms·
Because it shifts all the complexity to the monitoring system, making the "agents" really, really dumb. There would have to be more to push than just a single I
by lobster_johnson 9y ago
Because it shifts all the complexity to the monitoring system, making the "agents" really, really dumb. There would have to be more to push than just a single IP:
* Many installations run multiple Prometheus servers for redundancy, so to start, it'd have to be multiple IPs.
* They would also need auth credentials.
* They'd need retry/failure logic with backoff to prevent dogpiling.
* Clients would have to be careful to resolve the name, not cache the DNS lookup, in order to always resolve Prometheus to the right IP.
* If Prometheus moves, every pusher has to be updated.
* Since Prometheus wouldn't know about pushers, it wouldn't know if a push has failed. As Prometheus is pull-based, you can detect actual failure, not just absence of data.
There's a lot to be said for Prometheus' principle of baking exporters into individual, completely self-encapsulated programs — as opposed to things like collectd, diamond, Munin, Nagios etc. that collect a lot of stuff into a single, possibly plugin-based, system.
Don't forget, a lot of exporters come with third-party software. You want those programs to have as little config as possible. If I release an open-source app (let's say, a search engine), I can include a /metrics handler, and users who deploy my app can just point their Prometheus at it. It's enticingly simple.
As for graceful shutdown: The default pull frequency is 15 seconds, and you can increase it if you want to avoid losing metrics. Prometheus is designed not to deal with extremely fine-grained metrics; losing a few requests due to a shutdown shouldn't matter in the big picture. But for metrics that are sensitive, it's easy enough to bake them into some stateful store anyway (Redis or etcd, for example), or computing them in real time from stateful data (e.g. SQL). For example, if you have some kind of e-commerce order system, it's better if the exporter produces the numbers by issuing a query against the transaction tables, rather than maintaining RAM counters of dollars and cents.
- boto3 9y agoHow do you handle aggregated metrics, e.g. request count? What does the instance (either server/container) expose at localhost:9001/metrics?
- jalk 9y agoThe same way push based systems do it. Prom. scrapes the individual metrics off each instance and provides aggr. functions in the query lang.
- foxylion 9y agoYou would expose a counter with the total request count. Summing those up across all nodes known by Prometheus will give you the total amount of requests currently visible to monitoring. With "rate()" you could calculate the requests/second. But yes it is possible to miss some requests if a node goes down without Prometheus collecting the latest stats. But as the parent said, if you need such totals it might be better to store them persistently. Also I do not know a scenario where the total number of requests will trigger an alert.
- bbrazil 9y ago> But yes it is possible to miss some requests if a node goes down without Prometheus collecting the latest stats. The rate() function allows for this, you'll get the right answer on average.