3 ms·
The #1 reason Prometheus went with pull versus push is that it's fundamentally impossible to overload a pull-based system. If you have a push-based system, ther
by andrewstuart2 10y ago
The #1 reason Prometheus went with pull versus push is that it's fundamentally impossible to overload a pull-based system. If you have a push-based system, there's a hard problem of knowing whether you're getting all the messages or if system load or outages at scale are causing you to drop some metrics on the floor (as was the case at SoundCloud which drove them to start Prometheus).
I'd recommend listening to their recent CNCF online meetup [1] for some more of the background on push/pull and why they made the choices they did.
[1] https://www.youtube.com/watch?v=gfpG2_zIOvw https://www.youtube.com/watch?v=gfpG2_zIOvw
- agentgt 10y agoI will for sure check it out but I have found the opposite to be true (well not the opposite but a different problem). With pull I loose invaluable data because either the pulling system is down or it just doesn't know about all the nodes. Of course the converse could be said... what happens if nodes just choose not to push. Consequently I made the nodes smarter. IF they can't push they crash. I guess it is architecture choice but honestly if our nodes can't push to the bus (because this bus is more than just metrics that can be dropped it is critical biz shit) it is a fatal state for us. I think if you have a good enough bus like Kafka or RabbitMQ you can create a big enough buffer to prevent absolute chaos of overload (and now you just monitor queue size). If you see overload happening (ie massive queue) you can selectively drop messages (particularly with RabbitMQ as it has routing). But you are absolutely right that bad stuff can happen.