4 ms·

> And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. It's really tempting to believe that with the right hardware, we can put

> And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box. A couple of problems with that: 1. What happens when you have to reboot to apply a kernel update? 2. The geographic location of that single box is itself a gap in redundancy. This is one thing I like about AWS and the other hyperscalers, with their regions that each have multiple data centers connected by a private network, with load balancers and other things spanning the region.

FpUser 5y ago
>"It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box" This is what I have. 2 boxes only. One on my own premises in Canada and another on Hetzner in Europe. Each one can go down, no sweat. Also I do not strive for "live uninterrupted updates". All my applications and processes allow for short disruptions not to interfere with the main course. Also I write native servers in C++ for backend and they process thousands of requests per sec without breaking much sweat. Way more than my business would ever need. Same situations with my clients as well with the difference that for legal reason they rent computers or whatever goes as such from Amazon or Azure.
aduitsis 5y ago
Correct me, but I'd dare say admins have been rebooting their machines and running services without geographic redundancy for decades. Uptime was still very high.
bob1029 5y ago
This is a question that I have spent a ridiculous amount of time pondering. My conclusions so far are this: Single node application systems are by far the most reliable and manageable from a business logic standpoint. At no point does spreading a problem across more than 1 computer make that problem easier to solve. If you are concerned about latency, you need to get really abstract with your problem and ask what is even possible in information theoretic terms. If you are truly constrained to 1 serialized, synchronous context (i.e. a competitive counterstrike match or a stock exchange), there is little you can do to alleviate the root problem as your users get further from the server. You can certainly look at using some consensus protocol like multi-paxos, but then your transaction latency goes from microseconds (if you were clever) to milliseconds, representing orders of magnitude slowdown in the typical case. The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system. For the customers I work with, it is wayyy easier to build & sell a system that operates on a single box with sync replication + manual failover. Our customers are tolerant to production having a brief outage for a few minutes during the business day. Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.
- gravypod 5y ago
  > Single node application systems are by far the most reliable and manageable from a business logic standpoint. Manageable? Maybe. Reliable? No. Most companies don't need it but if a pipe bursts above the server room and now all of your medical records for your hospital are unavailable you're going to have a bad time. > At no point does spreading a problem across more than 1 computer make that problem easier to solve. I don't think anyone claimed it was easier (yet). The main thing people strive for is: 1. Dynamically reshaping your compute needs to match any needs. 2. Surviving failures of nodes or crashes in your application. 3. Managing all of the BS that goes with obtaining this (logs, etc). > Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable. I hope this isn't the case but this sounds like quite the death flag.
  - brian_cunnie 5y ago
    I’ve only seen one burst pipe bring down a computer (Pitney Bowes, 1995), but I’ve witnessed several S3 outages.
    - zthrowaway 5y ago
      You should spend a week working on a infra / ops team then.
      - Demiurge 5y ago
        I worked with multiple ops/infra teams, and have seen standard server rooms and advanced server rooms. They’re not supposed to have pipes leaking on the racks, that’s ridiculous. I know many stable redhat servers that didn’t need to reboot for years. I also maintain websites on hetzner servers that haven’t needed to reboot in years, under pretty high load. On the other hand, I got a hostnoc dedicated server and that thing went offline three times a month. So yeah, you have to know what you’re doing regardless of whether you’re building a server room or a k8s cluster.
rtsil 5y ago
1. I reboot it and my users will need to wait. 99.99% uptime is achieved with 53 minutes of downtime every year, largely enough for the occasional kernel update here and there. If needed, I can choose to apply the update in the middle of the night in the timezone of most of my visitors. If it really is a sensitive app and I can't afford any uptime, I just add another inexpensive box, put a load balancer in front (a Cloudflare load balancer would work fine). And since I now have 2 servers, I need a way to manage them without having to manually log in to each of them each time. Enter Ansible. And that's it. 2. Now that I have two cheap boxes, nothing prevents me from having them in two separate data centers and two separate providers.
pjmlp 5y ago
Point 2. is relevant, point 1. not really, unless one isn't willing to pay for the specific OS version that allow live kernel updates.
fxtentacle 5y ago
1. Planned maintenance in the middle of the night. Even Amazon and Heroku need to do it sometimes. At least we had a forced Postgres version update recently which was 30 minutes of downtime. 2. The latency within AWS is easily as high as US-EU backbone latency. Also, most web apps these days need 2000ms to load all the tracking and advertising crap, so 100ms location latency are negligible in comparison. Plus for most real companies, you'll have one website per country anyway. One server for .com and one server for .eu

FpUser 5y ago

>"It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box" This is what I have. 2 boxes only. One on my own premises in Canada and another on Hetzner in Europe. Each one can go down, no sweat. Also I do not strive for "live uninterrupted updates". All my applications and processes allow for short disruptions not to interfere with the main course. Also I write native servers in C++ for backend and they process thousands of requests per sec without breaking much sweat. Way more than my business would ever need. Same situations with my clients as well with the difference that for legal reason they rent computers or whatever goes as such from Amazon or Azure.

aduitsis 5y ago

Correct me, but I'd dare say admins have been rebooting their machines and running services without geographic redundancy for decades. Uptime was still very high.

bob1029 5y ago

This is a question that I have spent a ridiculous amount of time pondering. My conclusions so far are this: Single node application systems are by far the most reliable and manageable from a business logic standpoint. At no point does spreading a problem across more than 1 computer make that problem easier to solve. If you are concerned about latency, you need to get really abstract with your problem and ask what is even possible in information theoretic terms. If you are truly constrained to 1 serialized, synchronous context (i.e. a competitive counterstrike match or a stock exchange), there is little you can do to alleviate the root problem as your users get further from the server. You can certainly look at using some consensus protocol like multi-paxos, but then your transaction latency goes from microseconds (if you were clever) to milliseconds, representing orders of magnitude slowdown in the typical case. The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system. For the customers I work with, it is wayyy easier to build & sell a system that operates on a single box with sync replication + manual failover. Our customers are tolerant to production having a brief outage for a few minutes during the business day. Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.

gravypod 5y ago
> Single node application systems are by far the most reliable and manageable from a business logic standpoint. Manageable? Maybe. Reliable? No. Most companies don't need it but if a pipe bursts above the server room and now all of your medical records for your hospital are unavailable you're going to have a bad time. > At no point does spreading a problem across more than 1 computer make that problem easier to solve. I don't think anyone claimed it was easier (yet). The main thing people strive for is: 1. Dynamically reshaping your compute needs to match any needs. 2. Surviving failures of nodes or crashes in your application. 3. Managing all of the BS that goes with obtaining this (logs, etc). > Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable. I hope this isn't the case but this sounds like quite the death flag.
- brian_cunnie 5y ago
  I’ve only seen one burst pipe bring down a computer (Pitney Bowes, 1995), but I’ve witnessed several S3 outages.
  - zthrowaway 5y ago
    You should spend a week working on a infra / ops team then.
    - Demiurge 5y ago
      I worked with multiple ops/infra teams, and have seen standard server rooms and advanced server rooms. They’re not supposed to have pipes leaking on the racks, that’s ridiculous. I know many stable redhat servers that didn’t need to reboot for years. I also maintain websites on hetzner servers that haven’t needed to reboot in years, under pretty high load. On the other hand, I got a hostnoc dedicated server and that thing went offline three times a month. So yeah, you have to know what you’re doing regardless of whether you’re building a server room or a k8s cluster.

rtsil 5y ago

1. I reboot it and my users will need to wait. 99.99% uptime is achieved with 53 minutes of downtime every year, largely enough for the occasional kernel update here and there. If needed, I can choose to apply the update in the middle of the night in the timezone of most of my visitors. If it really is a sensitive app and I can't afford any uptime, I just add another inexpensive box, put a load balancer in front (a Cloudflare load balancer would work fine). And since I now have 2 servers, I need a way to manage them without having to manually log in to each of them each time. Enter Ansible. And that's it. 2. Now that I have two cheap boxes, nothing prevents me from having them in two separate data centers and two separate providers.

pjmlp 5y ago

Point 2. is relevant, point 1. not really, unless one isn't willing to pay for the specific OS version that allow live kernel updates.

fxtentacle 5y ago

1. Planned maintenance in the middle of the night. Even Amazon and Heroku need to do it sometimes. At least we had a forced Postgres version update recently which was 30 minutes of downtime. 2. The latency within AWS is easily as high as US-EU backbone latency. Also, most web apps these days need 2000ms to load all the tracking and advertising crap, so 100ms location latency are negligible in comparison. Plus for most real companies, you'll have one website per country anyway. One server for .com and one server for .eu