5 ms·

I might be wrong but one reasoning I saw mentioned was that pulling would give you a better picture into failures. If one of the services you are monintoring go

by tzumby 9y ago

I might be wrong but one reasoning I saw mentioned was that pulling would give you a better picture into failures. If one of the services you are monintoring goes down it won't be able to push anything.

lima 9y ago
But where's the difference between not being able to pull vs. not receiving metrics for a period of time?
- sz4kerto 9y ago
  > But where's the difference between not being able to pull vs. not receiving metrics for a period of time? Let's say I have a service that's only used once per day, at a random point in the day. I need to know if the service is available all the time, even if it's not used. With pull, I'll get back an 'OK' e.g. every minute. You could say that we should implement heartbeats, but again, then this service needs to be configurable to pick up changes in the monitoring infrastructure, then I need to have a separate check for the presence of heartbeats and the values of the samples..
  - zaroth 9y ago
    Monitoring if the process is running, if the input queue is free, if it will take a new job, etc. seems to me the responsibility for all that should be with whatever supervisor has the ability to reset/restart the service. Then it can report stats on uptime and number of restarts required. What if the service spins up on demand when it's needed, does its job, and then exits. You want to track how often it starts, how long it runs, how long the job was in the queue, how much CPU/RAM/IO was consumed, etc. With push that's all part of the cleanup code in the process. With pull it seems like you could completely miss that the process was even ever run?
    - bbrazil 9y ago
      That's a short lived batch job, for which a different approach is needed - basically push is the sane option there as pull won't cut it. Prometheus has the Pushgateway for this. The push vs pull debate is relevant for longer running daemons, not batch jobs.
- bostik 9y ago
  Well, there's one thing going for pull over push - and generally I am strongly in favour of push. With push, you need to know where to send the data. So you depend on a fixed configuration. With pull, you can essentially do a periodic nmap sweep to discover and update the monitoring data sources. That's one less thing to go wrong.
  - the_evacuator 9y ago
    Another nice thing about pull is you only pull from the authoritative source. If you have a daemon that has gone rogue your load balancer or naming system can throw it out of the named group/job/whatever even if it can’t shut it off. With push you might have zombies pushing duplicate data for what should be a unique target.

by tzumby 9y ago

lima 9y ago

But where's the difference between not being able to pull vs. not receiving metrics for a period of time?

sz4kerto 9y ago
> But where's the difference between not being able to pull vs. not receiving metrics for a period of time? Let's say I have a service that's only used once per day, at a random point in the day. I need to know if the service is available all the time, even if it's not used. With pull, I'll get back an 'OK' e.g. every minute. You could say that we should implement heartbeats, but again, then this service needs to be configurable to pick up changes in the monitoring infrastructure, then I need to have a separate check for the presence of heartbeats and the values of the samples..
- zaroth 9y ago
  Monitoring if the process is running, if the input queue is free, if it will take a new job, etc. seems to me the responsibility for all that should be with whatever supervisor has the ability to reset/restart the service. Then it can report stats on uptime and number of restarts required. What if the service spins up on demand when it's needed, does its job, and then exits. You want to track how often it starts, how long it runs, how long the job was in the queue, how much CPU/RAM/IO was consumed, etc. With push that's all part of the cleanup code in the process. With pull it seems like you could completely miss that the process was even ever run?
  - bbrazil 9y ago
    That's a short lived batch job, for which a different approach is needed - basically push is the sane option there as pull won't cut it. Prometheus has the Pushgateway for this. The push vs pull debate is relevant for longer running daemons, not batch jobs.
bostik 9y ago
Well, there's one thing going for pull over push - and generally I am strongly in favour of push. With push, you need to know where to send the data. So you depend on a fixed configuration. With pull, you can essentially do a periodic nmap sweep to discover and update the monitoring data sources. That's one less thing to go wrong.
- the_evacuator 9y ago
  Another nice thing about pull is you only pull from the authoritative source. If you have a daemon that has gone rogue your load balancer or naming system can throw it out of the named group/job/whatever even if it can’t shut it off. With push you might have zombies pushing duplicate data for what should be a unique target.