3 ms·

Some people have stated that GitHub Actions (or even GitLab CI) might now have an uptime as good as you'd expect for something like this, while others have brou

Some people have stated that GitHub Actions (or even GitLab CI) might now have an uptime as good as you'd expect for something like this, while others have brought ToS into the question and whether this is okay to do. On that note, I feel like this is probably a fun and clever hack, though might also cause problems if it would happen at too large a scale, much like people tried mining crypto in CI a while ago, which was really problematic. But then that leaves another question: what is everyone using for uptime monitoring? A cloud service of some sort? Self-hosted software in a different region/VPS provider with some alerting integrations? My current setup is pretty boring, but I figured I'd share nonetheless: - for uptime monitoring itself, I use Uptime Kuma: https://github.com/louislam/uptime-kuma - it's a reasonably simple and lightweight piece of software, with nice alerting capabilities - I have it hooked up to a self-hosted Mattermost instance for alerting, with the app on my phone - should the instance or the server itself fail, I have a separate solution to monitor the infrastructure itself - for infrastructure monitoring, I use Zabbix: https://www.zabbix.com/ - Zabbix will send me alerts to my e-mail (also self-hosted) should any issues become apparent - I use Let's Encrypt, Caddy/Nginx/Apache (depends) and also containers where applicable: Swarm/Nomad/K3s (depends) - storage uses bind mounts and DNS records use CNAME, so I can easily move stuff around Some of the aspects might be old fashioned, I know that people really enjoy Prometheus and Grafana, but personally it all seems to work decently - even if Zabbix for example is a bit old fashioned, it has a pretty sane monitoring setup out of the box (using Zabbix agent in a passive configuration).

majkinetor 4y ago
> I know that people really enjoy Prometheus and Grafana, but personally it all seems to work decently Those are for hard core metrics - not for status metrics which are trivial and you have only couple of them. When you have thousands of metrics, uptime kuma and bunch of friends wont help you. Detailed App/Infra metrics can also run on your own infrastructure unlike status pages that should use something independent. In your case, if your local mattermost fails, you will get 0 notifications. It doesn't even have to fail, its enough that your company Internet stops working (like it happened to me today, 20 + services were not available, including local statping [1] instance) > even if Zabbix for example is a bit old fashioned, Its not a problem that it is old fashioned but that it is harder then it needs to be - compared for example to telegraf/influxdb/grafana its setup on both sides is far from trivial. With influxdb, I can send metrics even from PowerShell scripts with 0 effort [2]. [1]: https://github.com/statping/statping https://github.com/statping/statping [2]: https://github.com/majkinetor/psinflux https://github.com/majkinetor/psinflux
- KronisLV 4y ago
  > When you have thousands of metrics, uptime kuma and bunch of friends wont help you. This is fair! Actually Uptime Kuma still doesn't support a multi-user mode (e.g. one admin user, multiple users that can edit values/setup, maybe some for viewing data). That said, I'm also at the scale where it makes perfect sense to use something this simplistic and there are few things that give me joy than running/building a container image and getting working software in less than an hour, which at my scale is also good for most if not all "day 2" concerns. > Detailed App/Infra metrics can also run on your own infrastructure unlike status pages that should use something independent In your case, if your local mattermost fails, you will get 0 notifications. Another fair point! That said, there's very little preventing you from choosing the most boring and stable multi-cloud setup that you can find. A Docker container for the software, with a reverse proxy and connected to the aforementioned infrastructure monitoring. Has the Docker service failed? I'll get a notification. Docker bridge network down? I'll get a notification. Containers fail health checks? Might still need to work on this, but totally doable as well with minimal work. Of course, there's also a lot of variability to how you can lay everything out - for example, I run some of my personal infrastructure from nodes that are in another room at my place, most other parts off of rented VMs in a semi-local company. My homepage, for example, has both Uptime Kuma as well as external monitoring service connected to it, just to compare how believable those values are. At work, though? For development/test environments, Uptime Kuma on a separate server is enough (say, if you have one that controls the container cluster or aggregates other metrics, might as well spin up a simple container there), or any other software package that's necessary, like Apache Skywalking etc. For production? Frankly, depending on what you're running, you might as well get a team of people together and come up with something that has proper redundancies in place, as well as a multi-cloud strategy.
  - majkinetor 4y ago
    > Has the Docker service failed? I'll get a notification. Docker bridge network down? I'll get a notification. If you rely on cloud services yes. If you run your own infra, then no, you will have to metric/alert that in a custom manner as with everything else. So that thing you mention is NOT a borring technology (which should be promoted) but outsourceing (which should NOT get promoted in general). > For development/test environments, Uptime Kuma on a separate server is enough It doesn't matter as your network will fail. There is nothing worse then status page having false positives.
GordonS 4y ago
I've been using Site24x7 for at least 6 years now, probably closer to 10. They have a "free forever" plan that's good enough for small projects: https://www.site24x7.com https://www.site24x7.com It's been rock solid, haven't had a single issue. I'm at the point where I now want a bit more, such as a status page, more frequent polling, monitoring from different continents and a record of historical outages. So I'm either going to pony up for a paid Site24x7 plan (£7/m), or self-host something myself - Kuma looks awesome BTW, I hadn't come across it before!
dmd 4y ago
OK, I'll bite: Why is Zabbix old-fashioned? What would you use instead? (I have ~60 bare metal servers of various vintages and OSes to monitor.)
- KronisLV 4y ago
  It's a sentiment that I've heard a lot - Zabbix focuses a lot on a core set of functionality, which is mostly just monitoring the state of some number of servers/VMs, with some optional integrations. While the metrics that come out of the box are nice, everything does feel a bit cumbersome, personally - such as creating network maps (which aren't automatically generated), dashboards for getting input on CPU/RAM/storage at a glance, as well as viewing web monitoring statistics. Actually, previously I used it for web monitoring, though for some reason by default it does not allow you to have triggers for those that would send an e-mail once a site goes down, even if the performance information and configuration was passable. But overall it's just views after views, nested inside of tables with weird UI/UX choices, such as having Update/Add buttons, both of which are necessary to persist changes (for example, in the web monitoring section when adding new monitoring steps and also wanting to save those changes). Like another commenter suggested, there are systems out there that make ingesting and visualizing data more easy and sometimes also more pleasant... Though personally Zabbix is fine for what I need.
jpeeler 4y ago
What made you go with Mattermost for relaying messages to your phone?
- KronisLV 4y ago
  Mostly the fact that I already had an instance up and running for chatting with some people and because it was pretty trivial to integrate Uptime Kuma with it. Sending e-mails would have also been a decent idea, but honestly configuring a Mattermost app/account/token is easier than messing around with a new mail account and all of the SMTP settings. As for Mattermost itself, previously I also enjoyed running Rocket.Chat, though somehow Mattermost felt a bit more like Slack (what I'm used to), runs with PostgreSQL as a backing store and also has the Boards and Playbooks integrations, alongside lots of other goodies. I do miss the ability to integrate Jitsi or something else in the actual interface like Rocket.Chat did, but it's not really a large problem overall.

majkinetor 4y ago

> I know that people really enjoy Prometheus and Grafana, but personally it all seems to work decently Those are for hard core metrics - not for status metrics which are trivial and you have only couple of them. When you have thousands of metrics, uptime kuma and bunch of friends wont help you. Detailed App/Infra metrics can also run on your own infrastructure unlike status pages that should use something independent. In your case, if your local mattermost fails, you will get 0 notifications. It doesn't even have to fail, its enough that your company Internet stops working (like it happened to me today, 20 + services were not available, including local statping [1] instance) > even if Zabbix for example is a bit old fashioned, Its not a problem that it is old fashioned but that it is harder then it needs to be - compared for example to telegraf/influxdb/grafana its setup on both sides is far from trivial. With influxdb, I can send metrics even from PowerShell scripts with 0 effort [2]. [1]: https://github.com/statping/statping https://github.com/statping/statping [2]: https://github.com/majkinetor/psinflux https://github.com/majkinetor/psinflux

KronisLV 4y ago
> When you have thousands of metrics, uptime kuma and bunch of friends wont help you. This is fair! Actually Uptime Kuma still doesn't support a multi-user mode (e.g. one admin user, multiple users that can edit values/setup, maybe some for viewing data). That said, I'm also at the scale where it makes perfect sense to use something this simplistic and there are few things that give me joy than running/building a container image and getting working software in less than an hour, which at my scale is also good for most if not all "day 2" concerns. > Detailed App/Infra metrics can also run on your own infrastructure unlike status pages that should use something independent In your case, if your local mattermost fails, you will get 0 notifications. Another fair point! That said, there's very little preventing you from choosing the most boring and stable multi-cloud setup that you can find. A Docker container for the software, with a reverse proxy and connected to the aforementioned infrastructure monitoring. Has the Docker service failed? I'll get a notification. Docker bridge network down? I'll get a notification. Containers fail health checks? Might still need to work on this, but totally doable as well with minimal work. Of course, there's also a lot of variability to how you can lay everything out - for example, I run some of my personal infrastructure from nodes that are in another room at my place, most other parts off of rented VMs in a semi-local company. My homepage, for example, has both Uptime Kuma as well as external monitoring service connected to it, just to compare how believable those values are. At work, though? For development/test environments, Uptime Kuma on a separate server is enough (say, if you have one that controls the container cluster or aggregates other metrics, might as well spin up a simple container there), or any other software package that's necessary, like Apache Skywalking etc. For production? Frankly, depending on what you're running, you might as well get a team of people together and come up with something that has proper redundancies in place, as well as a multi-cloud strategy.
- majkinetor 4y ago
  > Has the Docker service failed? I'll get a notification. Docker bridge network down? I'll get a notification. If you rely on cloud services yes. If you run your own infra, then no, you will have to metric/alert that in a custom manner as with everything else. So that thing you mention is NOT a borring technology (which should be promoted) but outsourceing (which should NOT get promoted in general). > For development/test environments, Uptime Kuma on a separate server is enough It doesn't matter as your network will fail. There is nothing worse then status page having false positives.

GordonS 4y ago

I've been using Site24x7 for at least 6 years now, probably closer to 10. They have a "free forever" plan that's good enough for small projects: https://www.site24x7.com https://www.site24x7.com It's been rock solid, haven't had a single issue. I'm at the point where I now want a bit more, such as a status page, more frequent polling, monitoring from different continents and a record of historical outages. So I'm either going to pony up for a paid Site24x7 plan (£7/m), or self-host something myself - Kuma looks awesome BTW, I hadn't come across it before!

dmd 4y ago

OK, I'll bite: Why is Zabbix old-fashioned? What would you use instead? (I have ~60 bare metal servers of various vintages and OSes to monitor.)

KronisLV 4y ago
It's a sentiment that I've heard a lot - Zabbix focuses a lot on a core set of functionality, which is mostly just monitoring the state of some number of servers/VMs, with some optional integrations. While the metrics that come out of the box are nice, everything does feel a bit cumbersome, personally - such as creating network maps (which aren't automatically generated), dashboards for getting input on CPU/RAM/storage at a glance, as well as viewing web monitoring statistics. Actually, previously I used it for web monitoring, though for some reason by default it does not allow you to have triggers for those that would send an e-mail once a site goes down, even if the performance information and configuration was passable. But overall it's just views after views, nested inside of tables with weird UI/UX choices, such as having Update/Add buttons, both of which are necessary to persist changes (for example, in the web monitoring section when adding new monitoring steps and also wanting to save those changes). Like another commenter suggested, there are systems out there that make ingesting and visualizing data more easy and sometimes also more pleasant... Though personally Zabbix is fine for what I need.

jpeeler 4y ago

What made you go with Mattermost for relaying messages to your phone?

KronisLV 4y ago
Mostly the fact that I already had an instance up and running for chatting with some people and because it was pretty trivial to integrate Uptime Kuma with it. Sending e-mails would have also been a decent idea, but honestly configuring a Mattermost app/account/token is easier than messing around with a new mail account and all of the SMTP settings. As for Mattermost itself, previously I also enjoyed running Rocket.Chat, though somehow Mattermost felt a bit more like Slack (what I'm used to), runs with PostgreSQL as a backing store and also has the Boards and Playbooks integrations, alongside lots of other goodies. I do miss the ability to integrate Jitsi or something else in the actual interface like Rocket.Chat did, but it's not really a large problem overall.