8 ms·

My takeaway: 1. Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when you

My takeaway: 1. Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working. 2. Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account. 3. Be able to jump ship easily, have backups of your zone, already know where you will transfer to.

djsumdog 7y ago
> UptimeRobot and Hetrixtools are good starters with generous free tier Are there any open source status pages/monitor programs that have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary port checks, etc? I'd rather just setup a status page/alert app on a $5 minimal DO/Vultr node and self-host/support/contribute to a FOSS program than use a commercial provider.
- iudqnolq 7y ago
  If you want email or text message alerts I would assume that's a complicated enough system you would want uptime alerts on it, and so on recursively ad infinitum.
  - unilynx 7y ago
    If you can set up nagios (which one would probably consider an interesting evening challenge if you were already willing to go for your own monitoring droplet) setting up pushover or amazon sns (for sms) should be easy enough.
    - iudqnolq 7y ago
      I would try to set up a completely open source monitoring setup just for fun, but once I'm paying for SNS I personally would rather just pay epsilon more and buy/rent the whole system. I get that may just be personal taste. I absolutely don't trust myself to run my own highly-reliable mailserver to send status alerts.
      - petre 7y ago
        Just send them locally and pull them with IMAP onto your phone.
        iudqnolq 7y ago
        I didn't even know that was possible. Thanks for teaching me something new. It's always nice to learn I need less SAAS magic than I thought.
    - falcolas 7y ago
      FWIW, a lot of cellular providers have an email gateway for delivering SMS messages. There's also paid SMS gateways, and options for providing arbitrary push notifications to smartphones.
      - iudqnolq 7y ago
        I'm pretty sure the free email gateways have no posted SLA. Plus, that requires a reliable email server, which would also need its own monitoring.
        falcolas 7y ago
        This was a few (3) years past, but they accepted root@localhost sendmail messages just fine in most cases, and delivered alerts within a minute or two of sending. We didn't rely on this long term, but it was a "good enough" first pass. I'd probably recommend using one of the gateways (or a more fully-featured service like Pagerduty) for more serious businesses, but for personal use (or where an outage detected the next day isn't crippling), it's remarkably useful.
- hedsht 7y ago
  check out https://github.com/hunterlong/statping https://github.com/hunterlong/statping - thats what i'm using.
- vorpalhex 7y ago
  You need to host across several nodes in different geographic locations and data centers to resist network splits. Then you need some way to slowly roll out upgrades to your monitoring platform over time.
  - djsumdog 7y ago
    I'm just talking about my personal infrastructure. If I host my crap in Vultr or Linode, I should be able to buy one cheap node on another provider just to run a simple status app: something with celary or sidekiq jobs to check my other stuff and intervals and generate a page with some red/yellow/green dots.
    - vorpalhex 7y ago
      How do you know if the monitoring node goes down at the same time as the other servers? Remember that Linode/Vultr/etc don't run their own datacenters, they share datacenters and sometimes downtime events can exist outside of datacenters.
- falcolas 7y ago
  <opinion class="unpopular"> Nagios. Or its descendant with a better configuration language, Icinga2. They're fairly easy to do a minimal install and configure in a container or on a VM. </opinion>
  - deleted 7y ago
    [deleted]
- isclever 7y ago
  Here is a good list: https://github.com/n1trux/awesome-sysadmin#monitoring https://github.com/n1trux/awesome-sysadmin#monitoring Maybe one fits what you are looking for.
- stevekemp 7y ago
  I wrote a scalable system for this: https://github.com/skx/overseer/ https://github.com/skx/overseer/ Handles SSL-checks, DNS-checks, SMTP-checks, & etc. Runs a thousand-checks every two minutes for me, give or take. Pluggable output via a redis-queue.
iudqnolq 7y ago
> Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working. Lesson learned :) > Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account. I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money. > Be able to jump ship easily, have backups of your zone, Luckily I have that > already know where you will transfer to. Any suggestions? Ironically I recently moved from Google Domains to Cloudflare because I was worried about issues with opaque support. I've learned my lesson picking based on cost alone, but I'm a college student who can't afford something too heavy-duty.
- woofcat 7y ago
  >I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money. Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
  - iudqnolq 7y ago
    > Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare. I believe it was both. If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
    - petre 7y ago
      > But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well? Because they have just proved being uncapable of doing it? Because redundancy? Because you shouldn't keep all your eggs in the same basket. I've been self hosting for at least 15 years and did not have any huge problems like the domain becoming non resolvable. I would never host my DNS on my registrar's infrastructure. It's being sloppy and lazy and it gets you embarassed.

djsumdog 7y ago

> UptimeRobot and Hetrixtools are good starters with generous free tier Are there any open source status pages/monitor programs that have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary port checks, etc? I'd rather just setup a status page/alert app on a $5 minimal DO/Vultr node and self-host/support/contribute to a FOSS program than use a commercial provider.

iudqnolq 7y ago
If you want email or text message alerts I would assume that's a complicated enough system you would want uptime alerts on it, and so on recursively ad infinitum.
- unilynx 7y ago
  If you can set up nagios (which one would probably consider an interesting evening challenge if you were already willing to go for your own monitoring droplet) setting up pushover or amazon sns (for sms) should be easy enough.
  - iudqnolq 7y ago
    I would try to set up a completely open source monitoring setup just for fun, but once I'm paying for SNS I personally would rather just pay epsilon more and buy/rent the whole system. I get that may just be personal taste. I absolutely don't trust myself to run my own highly-reliable mailserver to send status alerts.
    - petre 7y ago
      Just send them locally and pull them with IMAP onto your phone.
      - iudqnolq 7y ago
        I didn't even know that was possible. Thanks for teaching me something new. It's always nice to learn I need less SAAS magic than I thought.
  - falcolas 7y ago
    FWIW, a lot of cellular providers have an email gateway for delivering SMS messages. There's also paid SMS gateways, and options for providing arbitrary push notifications to smartphones.
    - iudqnolq 7y ago
      I'm pretty sure the free email gateways have no posted SLA. Plus, that requires a reliable email server, which would also need its own monitoring.
      - falcolas 7y ago
        This was a few (3) years past, but they accepted root@localhost sendmail messages just fine in most cases, and delivered alerts within a minute or two of sending. We didn't rely on this long term, but it was a "good enough" first pass. I'd probably recommend using one of the gateways (or a more fully-featured service like Pagerduty) for more serious businesses, but for personal use (or where an outage detected the next day isn't crippling), it's remarkably useful.
hedsht 7y ago
check out https://github.com/hunterlong/statping https://github.com/hunterlong/statping - thats what i'm using.
vorpalhex 7y ago
You need to host across several nodes in different geographic locations and data centers to resist network splits. Then you need some way to slowly roll out upgrades to your monitoring platform over time.
- djsumdog 7y ago
  I'm just talking about my personal infrastructure. If I host my crap in Vultr or Linode, I should be able to buy one cheap node on another provider just to run a simple status app: something with celary or sidekiq jobs to check my other stuff and intervals and generate a page with some red/yellow/green dots.
  - vorpalhex 7y ago
    How do you know if the monitoring node goes down at the same time as the other servers? Remember that Linode/Vultr/etc don't run their own datacenters, they share datacenters and sometimes downtime events can exist outside of datacenters.
falcolas 7y ago
<opinion class="unpopular"> Nagios. Or its descendant with a better configuration language, Icinga2. They're fairly easy to do a minimal install and configure in a container or on a VM. </opinion>
- deleted 7y ago
  [deleted]
isclever 7y ago
Here is a good list: https://github.com/n1trux/awesome-sysadmin#monitoring https://github.com/n1trux/awesome-sysadmin#monitoring Maybe one fits what you are looking for.
stevekemp 7y ago
I wrote a scalable system for this: https://github.com/skx/overseer/ https://github.com/skx/overseer/ Handles SSL-checks, DNS-checks, SMTP-checks, & etc. Runs a thousand-checks every two minutes for me, give or take. Pluggable output via a redis-queue.

iudqnolq 7y ago

> Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working. Lesson learned :) > Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account. I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money. > Be able to jump ship easily, have backups of your zone, Luckily I have that > already know where you will transfer to. Any suggestions? Ironically I recently moved from Google Domains to Cloudflare because I was worried about issues with opaque support. I've learned my lesson picking based on cost alone, but I'm a college student who can't afford something too heavy-duty.

woofcat 7y ago
>I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money. Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
- iudqnolq 7y ago
  > Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare. I believe it was both. If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
  - petre 7y ago
    > But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well? Because they have just proved being uncapable of doing it? Because redundancy? Because you shouldn't keep all your eggs in the same basket. I've been self hosting for at least 15 years and did not have any huge problems like the domain becoming non resolvable. I would never host my DNS on my registrar's infrastructure. It's being sloppy and lazy and it gets you embarassed.