5 ms·

Explaining modern server monitoring stacks for self-hosting

118 points by zdw 4y ago

melvyn2 4y ago
For my single self-hosted server (also NixOS), I use netdata, because it has all the info I need, is efficient enough while keeping a day’s worth of history as well, and most importantly has built-in alerts for services like telegram or slack. Other stacks can do the same, but I haven’t found anything as simple to set up; netdata is just a simple service.
- sokoloff 4y ago
  My first exposure to netdata was part of the default FreeNAS install. It was super easy to use and effective to understand the basic server health and telemetry.
- Gigachad 4y ago
  Self hosting in general tends to be overkill. It’s mostly people practicing for enterprise tasks. If you wanted to be practical about things you very likely would just give up self hosting entirely.
  - woopwoop24 4y ago
    based on?
  - doubled112 4y ago
    I think there's a pretty large middle ground between what I'd consider a home server setup (or self hosting it wherever) and practicing for enterprise tasks. Much of the crowd around here and certain subreddits IS probably practicing for more, but I have a counterexample. You won't find him around the internet. Not interested. I have a friend who's recently learned just enough Linux and Docker to get through a Debian install with some prompting, and run a Matrix home server, Nextcloud and Jellyfin because he thought they were cool. Its an old (for him) repurposed desktop. He's a mechanic. He doesn't care about anything enterprise. He doesn't want monitoring because he'll know if it's down. He doesn't care if backups run because he's sure he's got the important stuff on an old disk somewhere, probably.
- PanosJee 4y ago
  Simple yes; simplistic no. It can provide you with thousands of metrics you never thought of monitoring. On top of that comes with pre-configured alerts, eBPF support, metric correlations and anomaly rates for every single metric it collects.
Sin2x 4y ago
Prometheus and other modern application monitoring oriented timeseries solution stacks seem to be an overkill for simple server infrastructure monitoring, Zabbix is easier to setup and use and has all the batteries included.
- heavyset_go 4y ago
  Agree with this take, depending on just how simple a home setup is. If you're going to be working with it on the job, Prometheus is relatively simple to provision, though, and can provide some experience if you use it at home.
- xwowsersx 4y ago
  I hadn't known about Zabbix (or netdata) and I'm currently evaluating our options for monitoring. I was going to explore in the direction of Prometheus (+ Grafana), so I'm glad to know about these other seemingly more straightforward and simple options. Thanks for mentioning it.
  - KronisLV 4y ago
    Currently using Zabbix for monitoring my own servers. It's... okay. The UI sometimes feels a bit dated and not everything is as straightforward as one might expect, but for my use case (monitoring a bunch of GNU/Linux hosts) it's sufficient. Some of the things that are good about it: - can be run in containers if you want to, use a familiar DB like MySQL/MariaDB/PostgreSQL - the agent installation on the hosts that you want to manage is also pretty simple - supports both active (monitored host sends data to Zabbix) and passive (Zabbix asks the host for data) configurations - depending on the template that you use, has lots of built in metrics out of the box - easy to integrate with something like e-mail or SMS messaging for alerting, other plugins exist AFAIK - also has built in alerts, such as when disk space is low, CPU load is high, memory usage is high, swap space is low, host is unreachable etc. - has the ability to build dashboards with graphs, network maps etc. Some of the less nice points about it: - last I checked (older version, now using Uptime Kuma for this use case) the web monitoring didn't send alerts by default - last I checked (since haven't bothered to set up again) maintenance windows straight up didn't work and still sent notifications about the agent going down - in general the UI can be a bit cumbersome and there is some legacy to be found, at least a while back there were legacy graphs and the "modern" ones, each of which had different sets of functionality available - sometimes certain parts of the OS template also decided not to work, e.g. currently have disk usage not showing up in like 1/6 graphs, even though the configuration is pretty much the same for all of the nodes in question - in general, it's just not as popular as other solutions and might not have as much tutorials around setting things up in it I'd probably compare Zabbix against something like Nagios or LibreNMS, rather than netdata or Prometheus/Grafana, but perhaps that's just me. Edit: That said, the docs of Zabbix are pretty decent. Here's how configuring something in it typically looks like in the UI: https://www.zabbix.com/documentation/current/en/manual/web_monitoring https://www.zabbix.com/documentation/current/en/manual/web_m... Here's an example of some of the dashboard functionality: https://www.zabbix.com/documentation/6.2/en/manual/web_interface/frontend_sections/monitoring/dashboard https://www.zabbix.com/documentation/6.2/en/manual/web_inter... And here's maps that you can embed in the dashboard: https://www.zabbix.com/documentation/6.2/en/manual/web_interface/frontend_sections/monitoring/maps https://www.zabbix.com/documentation/6.2/en/manual/web_inter...
mr-karan 4y ago
Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.
- bornfreddy 4y ago
  How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network. This is my main gripe with Prometheus (didn't use it yet).
  - whateveracct 4y ago
    Feels like some combination of VPCs and bastion edges would work
  - luismedel 4y ago
    Exactly. I feel that monitoring solutions have to be the opposite of backup ones. For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups. But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
    - noja 4y ago
      For backups you can push with no security problems (unless you are doing a straight copy). If your backup client has rights to overwrite previous backups, that's a design problem.
- staticassertion 4y ago
  At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure). I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache. This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.
bullen 4y ago
I made my own monitor in 500 lines of code. It uses a HTTP client to poll all instances every minute which return /proc/stat so that I can see how much CPU they use: http://monitor.rupy.se http://monitor.rupy.se
PanosJee 4y ago
I always default to Netdata as it is batteries-included and can rarely outgrow it.
sccxy 4y ago
I have tried multiple monitoring tools, but I have found Uptime Kuma most useful for me. Alerts work well and it is not too complicated for simple things. https://github.com/louislam/uptime-kuma https://github.com/louislam/uptime-kuma
- ssddanbrown 4y ago
  My monitoring needs are very simple and I've had good success with Uptime Kuma. For checking server metrics i built a simple go service [1] to expose metrics with simple rules applied so uptime kurma will notify me when issues arise (Since the go service will change http status upon alert scenarios). [1]: https://github.com/ssddanbrown/wirestat https://github.com/ssddanbrown/wirestat
viraptor 4y ago
Since this post was interested in resource usage, I'm using telegraf for pushing system information (similar idea as collectd) and it's at 24kB of res memory. It reads from / writes to pretty much anything you can think of https://docs.influxdata.com/telegraf/v1.23/plugins/ https://docs.influxdata.com/telegraf/v1.23/plugins/
- linsomniac 4y ago
  Also a telegraf user here, been using it for ~5 years across 100+ machines, and it's been rock solid.
lupimiguel 4y ago
I find it very difficult to understand how "modern server monitoring" requires 6 or 7 components, when you can just deploy netdata on all your nodes with a 1-line installer and get way higher fidelity monitoring.
- infocollector 4y ago
  Is anyone using netdata in production? Has anyone looked at their data sharing/selling policies? How does it compare with other tools (https://github.com/louislam/uptime-kuma https://github.com/louislam/uptime-kuma) ? Looks very impressive.
  - JustFinishedBSG 4y ago
    They aren't even in the same category though ( except the "monitoring" super-category ). Uptime-kuma monitors uptime only while netdata doesn't even (afaik) monitor uptime, it's mostly machine related metrics.
    - cakrit 4y ago
      It monitors uptime too. Netdata has over 200 integrations, including all Prometheus endpoints. So yeah, it's not in the same category.
countryclub 4y ago
[flagged]
AishwaryaVenkat 4y ago
For server monitoring, Have a look at Prometheus, Grafana, Zabbix

118 points by zdw 4y ago

For my single self-hosted server (also NixOS), I use netdata, because it has all the info I need, is efficient enough while keeping a day’s worth of history as well, and most importantly has built-in alerts for services like telegram or slack. Other stacks can do the same, but I haven’t found anything as simple to set up; netdata is just a simple service.

sokoloff 4y ago
My first exposure to netdata was part of the default FreeNAS install. It was super easy to use and effective to understand the basic server health and telemetry.
Gigachad 4y ago
Self hosting in general tends to be overkill. It’s mostly people practicing for enterprise tasks. If you wanted to be practical about things you very likely would just give up self hosting entirely.
- woopwoop24 4y ago
  based on?
- doubled112 4y ago
  I think there's a pretty large middle ground between what I'd consider a home server setup (or self hosting it wherever) and practicing for enterprise tasks. Much of the crowd around here and certain subreddits IS probably practicing for more, but I have a counterexample. You won't find him around the internet. Not interested. I have a friend who's recently learned just enough Linux and Docker to get through a Debian install with some prompting, and run a Matrix home server, Nextcloud and Jellyfin because he thought they were cool. Its an old (for him) repurposed desktop. He's a mechanic. He doesn't care about anything enterprise. He doesn't want monitoring because he'll know if it's down. He doesn't care if backups run because he's sure he's got the important stuff on an old disk somewhere, probably.
PanosJee 4y ago
Simple yes; simplistic no. It can provide you with thousands of metrics you never thought of monitoring. On top of that comes with pre-configured alerts, eBPF support, metric correlations and anomaly rates for every single metric it collects.

Sin2x 4y ago

Prometheus and other modern application monitoring oriented timeseries solution stacks seem to be an overkill for simple server infrastructure monitoring, Zabbix is easier to setup and use and has all the batteries included.

heavyset_go 4y ago
Agree with this take, depending on just how simple a home setup is. If you're going to be working with it on the job, Prometheus is relatively simple to provision, though, and can provide some experience if you use it at home.
xwowsersx 4y ago
I hadn't known about Zabbix (or netdata) and I'm currently evaluating our options for monitoring. I was going to explore in the direction of Prometheus (+ Grafana), so I'm glad to know about these other seemingly more straightforward and simple options. Thanks for mentioning it.
- KronisLV 4y ago
  Currently using Zabbix for monitoring my own servers. It's... okay. The UI sometimes feels a bit dated and not everything is as straightforward as one might expect, but for my use case (monitoring a bunch of GNU/Linux hosts) it's sufficient. Some of the things that are good about it: - can be run in containers if you want to, use a familiar DB like MySQL/MariaDB/PostgreSQL - the agent installation on the hosts that you want to manage is also pretty simple - supports both active (monitored host sends data to Zabbix) and passive (Zabbix asks the host for data) configurations - depending on the template that you use, has lots of built in metrics out of the box - easy to integrate with something like e-mail or SMS messaging for alerting, other plugins exist AFAIK - also has built in alerts, such as when disk space is low, CPU load is high, memory usage is high, swap space is low, host is unreachable etc. - has the ability to build dashboards with graphs, network maps etc. Some of the less nice points about it: - last I checked (older version, now using Uptime Kuma for this use case) the web monitoring didn't send alerts by default - last I checked (since haven't bothered to set up again) maintenance windows straight up didn't work and still sent notifications about the agent going down - in general the UI can be a bit cumbersome and there is some legacy to be found, at least a while back there were legacy graphs and the "modern" ones, each of which had different sets of functionality available - sometimes certain parts of the OS template also decided not to work, e.g. currently have disk usage not showing up in like 1/6 graphs, even though the configuration is pretty much the same for all of the nodes in question - in general, it's just not as popular as other solutions and might not have as much tutorials around setting things up in it I'd probably compare Zabbix against something like Nagios or LibreNMS, rather than netdata or Prometheus/Grafana, but perhaps that's just me. Edit: That said, the docs of Zabbix are pretty decent. Here's how configuring something in it typically looks like in the UI: https://www.zabbix.com/documentation/current/en/manual/web_monitoring https://www.zabbix.com/documentation/current/en/manual/web_m... Here's an example of some of the dashboard functionality: https://www.zabbix.com/documentation/6.2/en/manual/web_interface/frontend_sections/monitoring/dashboard https://www.zabbix.com/documentation/6.2/en/manual/web_inter... And here's maps that you can embed in the dashboard: https://www.zabbix.com/documentation/6.2/en/manual/web_interface/frontend_sections/monitoring/maps https://www.zabbix.com/documentation/6.2/en/manual/web_inter...

mr-karan 4y ago

Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.

bornfreddy 4y ago
How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network. This is my main gripe with Prometheus (didn't use it yet).
- whateveracct 4y ago
  Feels like some combination of VPCs and bastion edges would work
- luismedel 4y ago
  Exactly. I feel that monitoring solutions have to be the opposite of backup ones. For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups. But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
  - noja 4y ago
    For backups you can push with no security problems (unless you are doing a straight copy). If your backup client has rights to overwrite previous backups, that's a design problem.
staticassertion 4y ago
At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure). I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache. This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.

bullen 4y ago

I made my own monitor in 500 lines of code. It uses a HTTP client to poll all instances every minute which return /proc/stat so that I can see how much CPU they use: http://monitor.rupy.se http://monitor.rupy.se

PanosJee 4y ago

I always default to Netdata as it is batteries-included and can rarely outgrow it.

sccxy 4y ago

I have tried multiple monitoring tools, but I have found Uptime Kuma most useful for me. Alerts work well and it is not too complicated for simple things. https://github.com/louislam/uptime-kuma https://github.com/louislam/uptime-kuma

ssddanbrown 4y ago
My monitoring needs are very simple and I've had good success with Uptime Kuma. For checking server metrics i built a simple go service [1] to expose metrics with simple rules applied so uptime kurma will notify me when issues arise (Since the go service will change http status upon alert scenarios). [1]: https://github.com/ssddanbrown/wirestat https://github.com/ssddanbrown/wirestat

viraptor 4y ago

Since this post was interested in resource usage, I'm using telegraf for pushing system information (similar idea as collectd) and it's at 24kB of res memory. It reads from / writes to pretty much anything you can think of https://docs.influxdata.com/telegraf/v1.23/plugins/ https://docs.influxdata.com/telegraf/v1.23/plugins/

linsomniac 4y ago
Also a telegraf user here, been using it for ~5 years across 100+ machines, and it's been rock solid.

lupimiguel 4y ago

I find it very difficult to understand how "modern server monitoring" requires 6 or 7 components, when you can just deploy netdata on all your nodes with a 1-line installer and get way higher fidelity monitoring.

infocollector 4y ago
Is anyone using netdata in production? Has anyone looked at their data sharing/selling policies? How does it compare with other tools (https://github.com/louislam/uptime-kuma https://github.com/louislam/uptime-kuma) ? Looks very impressive.
- JustFinishedBSG 4y ago
  They aren't even in the same category though ( except the "monitoring" super-category ). Uptime-kuma monitors uptime only while netdata doesn't even (afaik) monitor uptime, it's mostly machine related metrics.
  - cakrit 4y ago
    It monitors uptime too. Netdata has over 200 integrations, including all Prometheus endpoints. So yeah, it's not in the same category.

countryclub 4y ago

[flagged]

AishwaryaVenkat 4y ago

For server monitoring, Have a look at Prometheus, Grafana, Zabbix