4 ms·

I agree that centralization is a real risk, but I don't buy for a minute that your self-hosted Zulip instance is apt to have lower unplanned downtime than Slack

I agree that centralization is a real risk, but I don't buy for a minute that your self-hosted Zulip instance is apt to have lower unplanned downtime than Slack.

dijit 5y ago
Why? My central IRC node at a previous job had a better uptime than slack, and my previous company Gitlab server has lower unplanned downtime than Github... The key is that when I do my maintenances (on gitlab, for example) I can take the thing offline to do my work, which makes it much easier than trying to canary a change. It means I can plan a rollback or keep the backup in place. It's not that I'd have a higher uptime than slack really (though, I have experienced this) it's that the downtime works for the company and if it ever did fail we have the knowledge to fix it on our own schedule. I know that's scary to some, but it's a comfort to others.
iso1631 5y ago
4 months ago https://news.ycombinator.com/item?id=25632346 https://news.ycombinator.com/item?id=25632346 7 months ago https://news.ycombinator.com/item?id=24800558 https://news.ycombinator.com/item?id=24800558 8 months ago https://news.ycombinator.com/item?id=24269767 https://news.ycombinator.com/item?id=24269767 12 months ago https://news.ycombinator.com/item?id=23161623 https://news.ycombinator.com/item?id=23161623 I don't specifically know about zulip, but I've got many internal servers and services with uptimes far in excess of that Here's one, not exactly a lightly loaded server 19:10:57 up 1511 days, 7:18, 1 user, load average: 45.70, 41.32, 38.48
- jrockway 5y ago
  What practices do you use to ensure higher availability than Slack? Hope is not a strategy. (Answering my own question, I'd guess that the strategy is "never change anything". To some extent, I don't need 80% of the features that Slack offers, and code that you don't have is code that can't break. Not restarting stuff to release new code is also good for uptime. Would be interesting to see a SaaS that has a super-stable-never-change-anything release track, hosted on infra with similar goals, and see how that uptime compares to people's homegrown version of the same strategy.)
  - iso1631 5y ago
    My public facing servers (well https with client certificates that access them) are constantly upgraded, there's two of them, occasionally one goes down for a reboot overnight to get a new kernel, but clients will point to the other one. Radius does vrrp, not instant, but acceptable For true high availability (when downtime of milliseconds being unacceptable), we have A and B systems (sometimes more). E.g. syslog - router sends the messages out of two interfaces to two different servers which don't go down at the same time. For local staff wanting access to internal webpages, we can cope with 20 second downtimes at an appropiate time. I don't run a local irc server, but I'd be very surprised if it was any less stable than any other standard linux box -- do an apt or yum upgrade occasionally, any service outages you plan during a team meeting (thus we don't care if the chat system is down as we're talking via another method). Stability is far easier when 1) Your server isn't open to the whole world 2) Your server isn't scaling to millions of users Hell I've got a tcpdump session on one box that's been running since before the last slack outage 3 months ago.
- Hamuko 5y ago
  >up 1511 days What's the security update status on that?
  - iso1631 5y ago
    Awful I'm sure, but doesn't need to be, all it does it run a lot of snmp queries on an isolated network, store the data, and serve static webpages to via a proxy
    - Hamuko 5y ago
      >Awful I'm sure I don't really want to describe our company messaging platform to have "awful" security.
      - dijit 5y ago
        The truth is that you don’t know. And slack has been breached before. So this is a weird argument to make. The point being made is that you can determine your own trade-offs. With slack: you simply can’t.
- caturopath 5y ago
  Because it's hard to keep services up and most shops aren't especially competent. Having a process running for 1511 days does not mean a service has 1511 days of uptime -- there are a ton of places stuff can go wrong.
  - iso1631 5y ago
    Running a simple service on a single server for a year without issue is not hard. Running a service like slack is hard, but modern HN attitude is if you can't scale to a billion users your architecture is rubbish.
    - caturopath 5y ago
      Running a service isn't hard. Going a year without any sort of outage is. For example, if you're just serving from one place, any network outage between the users and that place will be an outage -- seamless failover is pointlessly hard and very few shops do this for their internally-hosted services.
      - iso1631 5y ago
        Depends how you define a network outage. One of my transatlantic circuits has been a bit ropey - had a 2 second outage on it a couple of nights ago, does that count as a network outage? If a user has a local network problem, that's not a loss in the service. If multiple users do that's an issue. My home broadband went out for 3 minutes at 23:33:43 GMT on May 13th, but then I wouldn't run a service needing 24/7 on my home broadband. My frickin nasdrive has a better uptime than slack.

I agree that centralization is a real risk, but I don't buy for a minute that your self-hosted Zulip instance is apt to have lower unplanned downtime than Slack.

dijit 5y ago

Why? My central IRC node at a previous job had a better uptime than slack, and my previous company Gitlab server has lower unplanned downtime than Github... The key is that when I do my maintenances (on gitlab, for example) I can take the thing offline to do my work, which makes it much easier than trying to canary a change. It means I can plan a rollback or keep the backup in place. It's not that I'd have a higher uptime than slack really (though, I have experienced this) it's that the downtime works for the company and if it ever did fail we have the knowledge to fix it on our own schedule. I know that's scary to some, but it's a comfort to others.

iso1631 5y ago

4 months ago https://news.ycombinator.com/item?id=25632346 https://news.ycombinator.com/item?id=25632346 7 months ago https://news.ycombinator.com/item?id=24800558 https://news.ycombinator.com/item?id=24800558 8 months ago https://news.ycombinator.com/item?id=24269767 https://news.ycombinator.com/item?id=24269767 12 months ago https://news.ycombinator.com/item?id=23161623 https://news.ycombinator.com/item?id=23161623 I don't specifically know about zulip, but I've got many internal servers and services with uptimes far in excess of that Here's one, not exactly a lightly loaded server 19:10:57 up 1511 days, 7:18, 1 user, load average: 45.70, 41.32, 38.48

jrockway 5y ago
What practices do you use to ensure higher availability than Slack? Hope is not a strategy. (Answering my own question, I'd guess that the strategy is "never change anything". To some extent, I don't need 80% of the features that Slack offers, and code that you don't have is code that can't break. Not restarting stuff to release new code is also good for uptime. Would be interesting to see a SaaS that has a super-stable-never-change-anything release track, hosted on infra with similar goals, and see how that uptime compares to people's homegrown version of the same strategy.)
- iso1631 5y ago
  My public facing servers (well https with client certificates that access them) are constantly upgraded, there's two of them, occasionally one goes down for a reboot overnight to get a new kernel, but clients will point to the other one. Radius does vrrp, not instant, but acceptable For true high availability (when downtime of milliseconds being unacceptable), we have A and B systems (sometimes more). E.g. syslog - router sends the messages out of two interfaces to two different servers which don't go down at the same time. For local staff wanting access to internal webpages, we can cope with 20 second downtimes at an appropiate time. I don't run a local irc server, but I'd be very surprised if it was any less stable than any other standard linux box -- do an apt or yum upgrade occasionally, any service outages you plan during a team meeting (thus we don't care if the chat system is down as we're talking via another method). Stability is far easier when 1) Your server isn't open to the whole world 2) Your server isn't scaling to millions of users Hell I've got a tcpdump session on one box that's been running since before the last slack outage 3 months ago.
Hamuko 5y ago
>up 1511 days What's the security update status on that?
- iso1631 5y ago
  Awful I'm sure, but doesn't need to be, all it does it run a lot of snmp queries on an isolated network, store the data, and serve static webpages to via a proxy
  - Hamuko 5y ago
    >Awful I'm sure I don't really want to describe our company messaging platform to have "awful" security.
    - dijit 5y ago
      The truth is that you don’t know. And slack has been breached before. So this is a weird argument to make. The point being made is that you can determine your own trade-offs. With slack: you simply can’t.
caturopath 5y ago
Because it's hard to keep services up and most shops aren't especially competent. Having a process running for 1511 days does not mean a service has 1511 days of uptime -- there are a ton of places stuff can go wrong.
- iso1631 5y ago
  Running a simple service on a single server for a year without issue is not hard. Running a service like slack is hard, but modern HN attitude is if you can't scale to a billion users your architecture is rubbish.
  - caturopath 5y ago
    Running a service isn't hard. Going a year without any sort of outage is. For example, if you're just serving from one place, any network outage between the users and that place will be an outage -- seamless failover is pointlessly hard and very few shops do this for their internally-hosted services.
    - iso1631 5y ago
      Depends how you define a network outage. One of my transatlantic circuits has been a bit ropey - had a 2 second outage on it a couple of nights ago, does that count as a network outage? If a user has a local network problem, that's not a loss in the service. If multiple users do that's an issue. My home broadband went out for 3 minutes at 23:33:43 GMT on May 13th, but then I wouldn't run a service needing 24/7 on my home broadband. My frickin nasdrive has a better uptime than slack.