11 ms·

Internet-monitoring – A Docker stack that monitors your home network

231 points by simonpure 6y ago

sdfhbdf 6y ago
I use telegraf with Ping and speedtest cli that both output to influxdb which then I browse with Grafana as well. Kind of similar but not quite. I also orchestrate with docker-compose on my RPi and I find this quite pleasent but in the end you should choose what you’re most familiar with. I didn’t want to spend my innovation tokens on promotheus hence I’ve gonna in what I viewed them as simpler, with telegraf. I ping Google, CloudFlare, Quad9 and Amazon EC2 and plot it to see the any unusual spikes.
corndoge 6y ago
See also smokeping, vaping https://github.com/20c/vaping https://github.com/20c/vaping https://oss.oetiker.ch/smokeping/ https://oss.oetiker.ch/smokeping/
- avh02 6y ago
  I recently set up smokeping to gather some data, I can't say it was a breeze - documentation appears complete but misses some key pieces of information you have to dig for or trial/error it out. E.g: i struggled figuring out where to put a non-standard probe Failure modes were also quite weird... adding a slave to the mix broke my mind a little. Was never sure it worked but then didn't need it so retired it before i figured it out Changing what you think is something trivial (e.g: frequency of data collection) invalidates your previously collected data files. Other than that - fairly solid, did what i needed, runs without a fuss.
  - linsomniac 6y ago
    I've set up smokeping a number of times over decades. Always a minor pain until the most recent one... I used a docker container, and it was a breeze! `sudo docker run -it -p 8000:80 -e WIPE=y -e TARGET="ISP;NextHop;$IP" -d dperson/smokeping` https://hub.docker.com/r/dperson/smokeping https://hub.docker.com/r/dperson/smokeping
- mjlee 6y ago
  https://github.com/SuperQ/smokeping_prober https://github.com/SuperQ/smokeping_prober This project is quite good for a smokeping-a-like for prometheus.
windexh8er 6y ago
I did something similar for continuous speedtest and tracking of home Internet [0]. [0] https://gitlab.com/splatops/cntn-speedtest https://gitlab.com/splatops/cntn-speedtest
aequitas 6y ago
Grafana and prometheus are a very nice combo for all sorts of monitoring. For my current setup I have added victoriametrics which is a drop-in replacement for prometheus. I like it more for home setup as it is easier on disk-space usage and the collection client (vmagent) does disk buffering. So I run the collection on a battery backupped raspberry pi which uses remote_write to send it to victoriametrics on my home server (which has a less higher uptime than the raspberry pi) without the risk of losing precious metrics.
jcims 6y ago
I've been tangentially looking at this as a 'beta' customer of Starlink, as the service is presently extremely variable with many small outages throughout the day. This kind of a collection of internet monitoring is super useful for the individual, but what I would really love to see is a way to federate the data in a meaningful way so that you can start to see operating patterns in the provider(s) being used.
- fy20 6y ago
  RIPE Atlas does exactly this: https://atlas.ripe.net/ https://atlas.ripe.net/ If you are on a ASN that they don't have many existing probes for, they'll send you one for free.
  - imup 6y ago
    We do something similar at https://imup.io https://imup.io but we run ping & speed tests from your home PC rather than shipping you a device. We approach the situation wanting to save folks money on their internet services for degradation in addition to providing analytics. Thanks for sharing that link for Ripe though, super cool project.
  - jcims 6y ago
    This is awesome thank you!
guillaumerose 6y ago
Remember the time the same job could be done with a simple cron, rrd and cacti for the storage and apache. Now you need "complicated" things: prometheus, grafana, docker, etc. I am a bit puzzled. Is it because sysadmin tutorials from 2000's are not showing up anymore in Google ? Why these tools failed to stay popular ?
- nine_zeros 6y ago
  Reinventing the wheel is the new mantra.
  - _joel 6y ago
    It's really not, Prometheus and Grafana can do a hell of a lot more than Cacti/RRDs
- colemannugent 6y ago
  They called to stay popular because they kinda suck. The setup you described works on one host and produces static graphs. With Prometheus and Grafana you can monitor an entire data center and add visualizations for metrics as you need them with no change to the monitored hosts. TLDR: the old software stack could be made to do this, modern monitoring stacks do it out of the box.
  - izacus 6y ago
    So people regularly run datacenters at home that need monitoring? That's what the original article is about after all.
    - orhmeh09 6y ago
      I think they meant to give an example to indicate the flexibility of the platform. You know nobody is claiming people regularly run data centers at home — why are you asking?
    - deadbunny 6y ago
programd 6y ago
Use node exporter on your machines and Prometheus to scrape the metrics and you will be drowning in pretty graphs. Super easy to set up and gives you system details across all your machines, all in one place. https://github.com/prometheus/node_exporter https://github.com/prometheus/node_exporter
- _joel 6y ago
  There are also script exporters (a few implementations) for simpler Nagios style checks and more specialised ones for getting things like Cloudwatch data (shout out to YACE[1] - with dashboards from promcat). It's really quite vibrant the ecosystem. [1] https://github.com/ivx/yet-another-cloudwatch-exporter https://github.com/ivx/yet-another-cloudwatch-exporter
  - nitrogen 6y ago
    Does anyone still use SNMP to announce metrics/monitors?
    - _joel 6y ago
      Of course but it's still in the traditional domain of networking kit (generally). As there are many components virtualised or software defined then I think in those areas, not so much as there are alternative ways to get the metrics. SNMP is a bit long in the tooth and can be a bit unweildy sometimes, ymmv etc.
syoc 6y ago
Cool. I have read a lot of approaches to home network monitoring and everybody seems to do things a little bit different. Telegraf has built-in modules for ping and dns latency. I recommend it as an alternative to prometheus.
- linsomniac 6y ago
  Telegraf is just going to do the "probe" component, right? You're still going to need Prometheus or Influxdb for storing the timeseries, and Grafana for visualizing I expect.
Noah-Huppert 6y ago
Cool! It's so funny my internet has been bad just recently so I took an old bash script of mine which did something similar and have been upgrading it into a proper application which exports metrics to Prometheus and Grafana: https://github.com/Noah-Huppert/net-test https://github.com/Noah-Huppert/net-test I saw a comment below where some was rolling their eyes that you "complicated" stuff with Prometheus, Grafana, and Docker and how you could just use Bash scripts and crons. As I just upgraded my codebase from this more bare metal approach to this "more complex setup" I'd like to mention: there's no way you could do time series statistical analysis easily with "just a cron job and a bash script". Prometheus and Grafana are for more than just buzz words. Prometheus offers an advanced time series database which allows you to, at minimum, do more robust analysis using data techniques like Histograms. As for Grafana, it makes exploring data dead easy. Providing users with a Docker Compose setup is also something I did with my tool and the benefits are huge. It lets me distribute a setup which relies on multiple moving parts working smoothly together. Sure I could write a whole wiki on how you should setup Prometheus Grafana and my tool, or I could distribute the setup with a configuration as code tool. Ensuring that even if someone doesn't want to use Docker Compose they can at least read my configuration as code and see exactly what I did to setup my tool.
- nitrogen 6y ago
  The closest analogous tool people would use back in the classic cron and bash days was MRTG.
- midrus 6y ago
  Recently I was looking to add some metrics to a small script and I didn't want to deal with all the complexity of setting up grafana, prometheus, the push-proxy-thing (sorry, don't remember the name). Looking for alternatives, I found InfluxDB, which was just one apt-get install away. It comes with a web interface to create dashboards, supports push based metrics, and overall I'm loving it. The web interface doesn't seem as powerful as grafana, but it covers most basic needs. And I think you can use it with grafana too.
  - armoredkitten 6y ago
jtchang 6y ago
I've been toying around with the idea of building a small device or widget to display things like this. There is hardware like this for temperature and humidity and such but nothing for internet health. If anyone is interested feel free to contact me. Would love to learn about what you would want in such a device.
- Namidairo 6y ago
  Sounds similar to the SamKnows Whitebox, which is a MT7621+MT7603+MT7612 router from which they run their speedtests and monitor various metrics. They don't actually run clients off those radios, they just scan and sniff. I don't think regular consumer is their target audience however, they seem more geared to ISP's and government agencies who need performance data over a large amount of connections. I suppose the problem with wanting to build a small device that does this, is finding the balance between cost-effective and actually being able to pull off 1gbit throughput on it. Of course, this all assumes you were going the repurposed hardware route as opposed to design from scratch. Doing it from scratch would have it's own costs and pitfalls. (Board design, manufacture, licensing costs, etc)
Avamander 6y ago
In addition to monitoring your own network, you might want to monitor your ISP's and share the results for research - a software RIPE Atlas probe is a great method for that: https://labs.ripe.net/Members/alun_davies/ripe-atlas-software-probes https://labs.ripe.net/Members/alun_davies/ripe-atlas-softwar...
DavideNL 6y ago
Another similar alternative: https://github.com/henrywhitaker3/Speedtest-Tracker https://github.com/henrywhitaker3/Speedtest-Tracker
Havoc 6y ago
Reminder that Speedtest can chew through a fair bit of data for those on metered connections
neilv 6y ago
For simpler needs, like being able to see when my network dropped out, without needing another device... On a OpenWrt home router, I usually install `collectd-mod-ping` and `luci-app-statistics`, and have it periodically ping a couple of different hosts that will answer pings reliably. (This can double for monitoring your third-party hosting.) Caveat: since you're probably running OpenWrt on a consumer SoHo router, I recommend sending this logging data to a cheap USB flash drive, not wearing out the router's onboard flash. An even simpler alternative would be to keep a browser window open with the OpenWrt admin UI's "Status -> Realtime Graphs" displaying. (If you're new to OpenWrt, a fairly recent router hardware model that's easy to re-flash with solid OpenWrt, and fairly inexpensive: https://openwrt.org/toh/netgear/r7800 https://openwrt.org/toh/netgear/r7800 )
dikaio 6y ago
Simon releases so many community based projects how can you not love this guy.
2dvisio 6y ago
Interesting lots of people had the same idea. I have done something similar for my home network using an Ethernet connected RaspberryPI dedicated to this task using a cronjob, the speedtest-cli JSON format, MongoDB and just some JS graphs. - very rudimentary. One thing I found very important was to plot the scatter of PING vs. Download speed (my Upload is super stable). Then I created the probability density function of that (using KDE) and used it to get a better deal from my network provider - I could prove they were selling me a slower network most of the time - they now allocate 70Mb/s on a 50Mb/s deal so just that they hit the 50Mb/s most of the time. It also allowed me to clearly identify some outlier situations where my network suddenly became 'bad' and I could show evidence of it being bad over time. Certainly a project that should see more traction!

231 points by simonpure 6y ago

I use telegraf with Ping and speedtest cli that both output to influxdb which then I browse with Grafana as well. Kind of similar but not quite. I also orchestrate with docker-compose on my RPi and I find this quite pleasent but in the end you should choose what you’re most familiar with. I didn’t want to spend my innovation tokens on promotheus hence I’ve gonna in what I viewed them as simpler, with telegraf. I ping Google, CloudFlare, Quad9 and Amazon EC2 and plot it to see the any unusual spikes.

corndoge 6y ago

See also smokeping, vaping https://github.com/20c/vaping https://github.com/20c/vaping https://oss.oetiker.ch/smokeping/ https://oss.oetiker.ch/smokeping/

avh02 6y ago
I recently set up smokeping to gather some data, I can't say it was a breeze - documentation appears complete but misses some key pieces of information you have to dig for or trial/error it out. E.g: i struggled figuring out where to put a non-standard probe Failure modes were also quite weird... adding a slave to the mix broke my mind a little. Was never sure it worked but then didn't need it so retired it before i figured it out Changing what you think is something trivial (e.g: frequency of data collection) invalidates your previously collected data files. Other than that - fairly solid, did what i needed, runs without a fuss.
- linsomniac 6y ago
  I've set up smokeping a number of times over decades. Always a minor pain until the most recent one... I used a docker container, and it was a breeze! `sudo docker run -it -p 8000:80 -e WIPE=y -e TARGET="ISP;NextHop;$IP" -d dperson/smokeping` https://hub.docker.com/r/dperson/smokeping https://hub.docker.com/r/dperson/smokeping
mjlee 6y ago
https://github.com/SuperQ/smokeping_prober https://github.com/SuperQ/smokeping_prober This project is quite good for a smokeping-a-like for prometheus.

windexh8er 6y ago

I did something similar for continuous speedtest and tracking of home Internet [0]. [0] https://gitlab.com/splatops/cntn-speedtest https://gitlab.com/splatops/cntn-speedtest

aequitas 6y ago

Grafana and prometheus are a very nice combo for all sorts of monitoring. For my current setup I have added victoriametrics which is a drop-in replacement for prometheus. I like it more for home setup as it is easier on disk-space usage and the collection client (vmagent) does disk buffering. So I run the collection on a battery backupped raspberry pi which uses remote_write to send it to victoriametrics on my home server (which has a less higher uptime than the raspberry pi) without the risk of losing precious metrics.

jcims 6y ago

I've been tangentially looking at this as a 'beta' customer of Starlink, as the service is presently extremely variable with many small outages throughout the day. This kind of a collection of internet monitoring is super useful for the individual, but what I would really love to see is a way to federate the data in a meaningful way so that you can start to see operating patterns in the provider(s) being used.

fy20 6y ago
RIPE Atlas does exactly this: https://atlas.ripe.net/ https://atlas.ripe.net/ If you are on a ASN that they don't have many existing probes for, they'll send you one for free.
- imup 6y ago
  We do something similar at https://imup.io https://imup.io but we run ping & speed tests from your home PC rather than shipping you a device. We approach the situation wanting to save folks money on their internet services for degradation in addition to providing analytics. Thanks for sharing that link for Ripe though, super cool project.
- jcims 6y ago
  This is awesome thank you!

guillaumerose 6y ago

Remember the time the same job could be done with a simple cron, rrd and cacti for the storage and apache. Now you need "complicated" things: prometheus, grafana, docker, etc. I am a bit puzzled. Is it because sysadmin tutorials from 2000's are not showing up anymore in Google ? Why these tools failed to stay popular ?

nine_zeros 6y ago
Reinventing the wheel is the new mantra.
- _joel 6y ago
  It's really not, Prometheus and Grafana can do a hell of a lot more than Cacti/RRDs
colemannugent 6y ago
They called to stay popular because they kinda suck. The setup you described works on one host and produces static graphs. With Prometheus and Grafana you can monitor an entire data center and add visualizations for metrics as you need them with no change to the monitored hosts. TLDR: the old software stack could be made to do this, modern monitoring stacks do it out of the box.
- izacus 6y ago
  So people regularly run datacenters at home that need monitoring? That's what the original article is about after all.
  - orhmeh09 6y ago
    I think they meant to give an example to indicate the flexibility of the platform. You know nobody is claiming people regularly run data centers at home — why are you asking?
  - deadbunny 6y ago

programd 6y ago

Use node exporter on your machines and Prometheus to scrape the metrics and you will be drowning in pretty graphs. Super easy to set up and gives you system details across all your machines, all in one place. https://github.com/prometheus/node_exporter https://github.com/prometheus/node_exporter

_joel 6y ago
There are also script exporters (a few implementations) for simpler Nagios style checks and more specialised ones for getting things like Cloudwatch data (shout out to YACE[1] - with dashboards from promcat). It's really quite vibrant the ecosystem. [1] https://github.com/ivx/yet-another-cloudwatch-exporter https://github.com/ivx/yet-another-cloudwatch-exporter
- nitrogen 6y ago
  Does anyone still use SNMP to announce metrics/monitors?
  - _joel 6y ago
    Of course but it's still in the traditional domain of networking kit (generally). As there are many components virtualised or software defined then I think in those areas, not so much as there are alternative ways to get the metrics. SNMP is a bit long in the tooth and can be a bit unweildy sometimes, ymmv etc.

syoc 6y ago

Cool. I have read a lot of approaches to home network monitoring and everybody seems to do things a little bit different. Telegraf has built-in modules for ping and dns latency. I recommend it as an alternative to prometheus.

linsomniac 6y ago
Telegraf is just going to do the "probe" component, right? You're still going to need Prometheus or Influxdb for storing the timeseries, and Grafana for visualizing I expect.

Noah-Huppert 6y ago

Cool! It's so funny my internet has been bad just recently so I took an old bash script of mine which did something similar and have been upgrading it into a proper application which exports metrics to Prometheus and Grafana: https://github.com/Noah-Huppert/net-test https://github.com/Noah-Huppert/net-test I saw a comment below where some was rolling their eyes that you "complicated" stuff with Prometheus, Grafana, and Docker and how you could just use Bash scripts and crons. As I just upgraded my codebase from this more bare metal approach to this "more complex setup" I'd like to mention: there's no way you could do time series statistical analysis easily with "just a cron job and a bash script". Prometheus and Grafana are for more than just buzz words. Prometheus offers an advanced time series database which allows you to, at minimum, do more robust analysis using data techniques like Histograms. As for Grafana, it makes exploring data dead easy. Providing users with a Docker Compose setup is also something I did with my tool and the benefits are huge. It lets me distribute a setup which relies on multiple moving parts working smoothly together. Sure I could write a whole wiki on how you should setup Prometheus Grafana and my tool, or I could distribute the setup with a configuration as code tool. Ensuring that even if someone doesn't want to use Docker Compose they can at least read my configuration as code and see exactly what I did to setup my tool.

nitrogen 6y ago
The closest analogous tool people would use back in the classic cron and bash days was MRTG.
midrus 6y ago
Recently I was looking to add some metrics to a small script and I didn't want to deal with all the complexity of setting up grafana, prometheus, the push-proxy-thing (sorry, don't remember the name). Looking for alternatives, I found InfluxDB, which was just one apt-get install away. It comes with a web interface to create dashboards, supports push based metrics, and overall I'm loving it. The web interface doesn't seem as powerful as grafana, but it covers most basic needs. And I think you can use it with grafana too.
- armoredkitten 6y ago

jtchang 6y ago

I've been toying around with the idea of building a small device or widget to display things like this. There is hardware like this for temperature and humidity and such but nothing for internet health. If anyone is interested feel free to contact me. Would love to learn about what you would want in such a device.

Namidairo 6y ago
Sounds similar to the SamKnows Whitebox, which is a MT7621+MT7603+MT7612 router from which they run their speedtests and monitor various metrics. They don't actually run clients off those radios, they just scan and sniff. I don't think regular consumer is their target audience however, they seem more geared to ISP's and government agencies who need performance data over a large amount of connections. I suppose the problem with wanting to build a small device that does this, is finding the balance between cost-effective and actually being able to pull off 1gbit throughput on it. Of course, this all assumes you were going the repurposed hardware route as opposed to design from scratch. Doing it from scratch would have it's own costs and pitfalls. (Board design, manufacture, licensing costs, etc)

Avamander 6y ago

In addition to monitoring your own network, you might want to monitor your ISP's and share the results for research - a software RIPE Atlas probe is a great method for that: https://labs.ripe.net/Members/alun_davies/ripe-atlas-software-probes https://labs.ripe.net/Members/alun_davies/ripe-atlas-softwar...

DavideNL 6y ago

Another similar alternative: https://github.com/henrywhitaker3/Speedtest-Tracker https://github.com/henrywhitaker3/Speedtest-Tracker

Havoc 6y ago

Reminder that Speedtest can chew through a fair bit of data for those on metered connections

neilv 6y ago

For simpler needs, like being able to see when my network dropped out, without needing another device... On a OpenWrt home router, I usually install `collectd-mod-ping` and `luci-app-statistics`, and have it periodically ping a couple of different hosts that will answer pings reliably. (This can double for monitoring your third-party hosting.) Caveat: since you're probably running OpenWrt on a consumer SoHo router, I recommend sending this logging data to a cheap USB flash drive, not wearing out the router's onboard flash. An even simpler alternative would be to keep a browser window open with the OpenWrt admin UI's "Status -> Realtime Graphs" displaying. (If you're new to OpenWrt, a fairly recent router hardware model that's easy to re-flash with solid OpenWrt, and fairly inexpensive: https://openwrt.org/toh/netgear/r7800 https://openwrt.org/toh/netgear/r7800 )

dikaio 6y ago

Simon releases so many community based projects how can you not love this guy.

2dvisio 6y ago

Interesting lots of people had the same idea. I have done something similar for my home network using an Ethernet connected RaspberryPI dedicated to this task using a cronjob, the speedtest-cli JSON format, MongoDB and just some JS graphs. - very rudimentary. One thing I found very important was to plot the scatter of PING vs. Download speed (my Upload is super stable). Then I created the probability density function of that (using KDE) and used it to get a better deal from my network provider - I could prove they were selling me a slower network most of the time - they now allocate 70Mb/s on a 50Mb/s deal so just that they hit the 50Mb/s most of the time. It also allowed me to clearly identify some outlier situations where my network suddenly became 'bad' and I could show evidence of it being bad over time. Certainly a project that should see more traction!