4 ms·

Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configura

Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.

bornfreddy 4y ago
How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network. This is my main gripe with Prometheus (didn't use it yet).
- whateveracct 4y ago
  Feels like some combination of VPCs and bastion edges would work
- luismedel 4y ago
  Exactly. I feel that monitoring solutions have to be the opposite of backup ones. For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups. But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
  - noja 4y ago
    For backups you can push with no security problems (unless you are doing a straight copy). If your backup client has rights to overwrite previous backups, that's a design problem.
staticassertion 4y ago
At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure). I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache. This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.
- evv555 4y ago
  Part of what parent is describing is pushing configurations down to the agents from a centrally managed admin panel. I think this works well for basic system metrics but not for anything off the beaten path requiring integrations and scripts. In response to your vector solution:. What I found highly adaptable is having all servers forward to a central vector/fluent-bit agent, have that agent forward to kinesis firehose, then attach a transform lambda to create the final output. You end up with 2 configuration points from which you can control and transform all your logs. Instead of having to push this down to the servers through config management.

bornfreddy 4y ago

How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network. This is my main gripe with Prometheus (didn't use it yet).

whateveracct 4y ago
Feels like some combination of VPCs and bastion edges would work
luismedel 4y ago
Exactly. I feel that monitoring solutions have to be the opposite of backup ones. For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups. But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
- noja 4y ago
  For backups you can push with no security problems (unless you are doing a straight copy). If your backup client has rights to overwrite previous backups, that's a design problem.

staticassertion 4y ago

At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure). I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache. This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.

evv555 4y ago
Part of what parent is describing is pushing configurations down to the agents from a centrally managed admin panel. I think this works well for basic system metrics but not for anything off the beaten path requiring integrations and scripts. In response to your vector solution:. What I found highly adaptable is having all servers forward to a central vector/fluent-bit agent, have that agent forward to kinesis firehose, then attach a transform lambda to create the final output. You end up with 2 configuration points from which you can control and transform all your logs. Instead of having to push this down to the servers through config management.