4 ms·

I don't know a lot about monitoring, why do you say pulling is an anti pattern? My understanding is that with the push pattern you submit a metric when it's av

by pnt12 3y ago

I don't know a lot about monitoring, why do you say pulling is an anti pattern? My understanding is that with the push pattern you submit a metric when it's available, with the pull pattern you make them available via an interface. I've read the first is not as performant, as it leads to submitting lots of metrics. Although I can think of an alternative, which is storing them and pushing batches periodically?

marcosdumay 3y ago
Pulling implies you go storing the bare data, and run through the data when it's needed. Pushing implies that you run through the data as it appears, and sends the results to the log aggregator for storage as soon as they exist. Pulling has a lot of problems with storage and storage bandwidth.
- baq 3y ago
  Pushing has the Byzantine generals problem where you have no idea if your data actually ended up anywhere and still need to pull to backfill.
  - marcosdumay 3y ago
    I'm not sure we are talking about the same thing, because I see no need at all to backfill monitoring data. In fact, it's one of my guidelines to decide if something is monitoring or logging; logs can not have holes. You push as a best effort. It's up to the receiving party to react to a lack of data.
  - KaiserPro 3y ago
    > Pushing has the Byzantine generals not really. the general's problem is about trust and authentication. however, to your point about not being sure if the metrics ever get there, there is another way. If you just have metrics, then its a single point of failure. so you need another basic "alive" check. for web services most people already have some sort of /__health or other check to allow loadbalancers to work. marrying up the two sources of data allows you to work out if its the service, metrics or loadbalancer thats not working.
briffle 3y ago
Pulling metrics from your service is easy and simple. Pulling metrics from thousands and thousands of services, spread out over many clouds/regions/environments, puts a HUGE strain on the single pulling server. Especially if it then evaluates its list of rules for each pull, etc. But for pushing, if you don't get a message from service X in the last hour, is it down, or is it just not being used. So things like heartbeats intervals need to be configured, etc.
- llama052 3y ago
  > Pulling metrics from your service is easy and simple. Pulling metrics from thousands and thousands of services, spread out over many clouds/regions/environments, puts a HUGE strain on the single pulling server. Especially if it then evaluates its list of rules for each pull, etc. Oh god if you're trying to do that with a single pulling server you're doing something terribly wrong. > But for pushing, if you don't get a message from service X in the last hour, is it down, or is it just not being used. So things like heartbeats intervals need to be configured, etc. The inverse is true on this situation, if a server is attempting to accept traffic from thousands of services it's got the potential to get DOS'd pretty damn quick, and you can't even control that flow easily. At least from a pulling server you can dictate that on one host, or scale that out and aggregate in another layer. (Thanos/Cortex). I'll always prefer pulling versus pushing because you gain more control and prevent yourself from blasting a server into space with a thousand calls it can't refuse.

by pnt12 3y ago

marcosdumay 3y ago

Pulling implies you go storing the bare data, and run through the data when it's needed. Pushing implies that you run through the data as it appears, and sends the results to the log aggregator for storage as soon as they exist. Pulling has a lot of problems with storage and storage bandwidth.

baq 3y ago
Pushing has the Byzantine generals problem where you have no idea if your data actually ended up anywhere and still need to pull to backfill.
- marcosdumay 3y ago
  I'm not sure we are talking about the same thing, because I see no need at all to backfill monitoring data. In fact, it's one of my guidelines to decide if something is monitoring or logging; logs can not have holes. You push as a best effort. It's up to the receiving party to react to a lack of data.
- KaiserPro 3y ago
  > Pushing has the Byzantine generals not really. the general's problem is about trust and authentication. however, to your point about not being sure if the metrics ever get there, there is another way. If you just have metrics, then its a single point of failure. so you need another basic "alive" check. for web services most people already have some sort of /__health or other check to allow loadbalancers to work. marrying up the two sources of data allows you to work out if its the service, metrics or loadbalancer thats not working.

briffle 3y ago

Pulling metrics from your service is easy and simple. Pulling metrics from thousands and thousands of services, spread out over many clouds/regions/environments, puts a HUGE strain on the single pulling server. Especially if it then evaluates its list of rules for each pull, etc. But for pushing, if you don't get a message from service X in the last hour, is it down, or is it just not being used. So things like heartbeats intervals need to be configured, etc.

llama052 3y ago
> Pulling metrics from your service is easy and simple. Pulling metrics from thousands and thousands of services, spread out over many clouds/regions/environments, puts a HUGE strain on the single pulling server. Especially if it then evaluates its list of rules for each pull, etc. Oh god if you're trying to do that with a single pulling server you're doing something terribly wrong. > But for pushing, if you don't get a message from service X in the last hour, is it down, or is it just not being used. So things like heartbeats intervals need to be configured, etc. The inverse is true on this situation, if a server is attempting to accept traffic from thousands of services it's got the potential to get DOS'd pretty damn quick, and you can't even control that flow easily. At least from a pulling server you can dictate that on one host, or scale that out and aggregate in another layer. (Thanos/Cortex). I'll always prefer pulling versus pushing because you gain more control and prevent yourself from blasting a server into space with a thousand calls it can't refuse.