3 ms·

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expec

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect resilience to host failure?

DoubleFree 3y ago
Fly postgres is not managed postgres, it's cli sugar over a normal fly app, which the [docs](https://fly.io/docs/postgres/ https://fly.io/docs/postgres/) make quite clear. Their docs also make clear that if you run postgres in a single-instance configuration, if the hardware it's running on has problems, you database will go down. I believe the underlying reason that precludes failing over to a different host machine, is that fly volumes are slices of host-attached nvme drives. If the host goes down, these can't be migrated. I _think_ instances without attached volumes will fail-over to a different host. Of course, that's not ideal, and maybe their CLI should also warn about this loudly when creating the cluster.
smallerfish 3y ago
If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.
- marcinzm 3y ago
  Downtime but limited downtime since the data is stored with redundantly across multiple machines in the same AZ. So unless the AZ goes down (which is a different failure than what happened here) you can restart the DB on a different instance pretty quickly and I'm guessing AWS will do it automatically for you. edit: Remove triple as not certain about level of redundancy
  - truetraveller 3y ago
    I don't believe their RDS / EBS has 3x redundancy. With SSD, that would be super costly for them. But if that's correct, that would be incredible.
    - marcinzm 3y ago
      May not be 3x but it is replicated so even a total instance failure would not make you lose data: >Amazon EBS volumes are designed to be highly available, reliable, and durable. At no additional charge to you, Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. For more details, see the Amazon EBS Service Level Agreement. https://aws.amazon.com/ebs/features/#Amazon_EBS_availability_and_durability https://aws.amazon.com/ebs/features/#Amazon_EBS_availability...
- williamdclt 3y ago
  > Maybe not with Aurora If a read replica fails, I'd expect no downtime (possibly a few errors as connections get cut off abruptly). Although there's always the risk that the remaining instances aren't able to handle the additional load. If the master fails, you'll get a ~2min downtime
- api 3y ago
  Yeah but you won't lose your data. They have backup infrastructure and EBS is rock solid. Down time is one thing. Data loss is something else.
capableweb 3y ago
> Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Depends on where in your development cycle you are. If you just got started and haven't even figured out what you're actually building (prototyping), you shouldn't really use a hosting provider that randomly lose instances. If you're on the other hand have done everything to improve your applications performance, had to resolve through-output issues with a distributed architecture and now running 10+ instances, then losing one host shouldn't impact you too much. But you really shouldn't start this way, it's doing web services the hard way and introduces a lot of complexity you shouldn't want to deal with when you're still trying to find product market fit.
- riwsky 3y ago
  GP is referring to fly.io architecting for single instance failures, not its customers.

DoubleFree 3y ago

Fly postgres is not managed postgres, it's cli sugar over a normal fly app, which the [docs](https://fly.io/docs/postgres/ https://fly.io/docs/postgres/) make quite clear. Their docs also make clear that if you run postgres in a single-instance configuration, if the hardware it's running on has problems, you database will go down. I believe the underlying reason that precludes failing over to a different host machine, is that fly volumes are slices of host-attached nvme drives. If the host goes down, these can't be migrated. I _think_ instances without attached volumes will fail-over to a different host. Of course, that's not ideal, and maybe their CLI should also warn about this loudly when creating the cluster.

smallerfish 3y ago

If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.

marcinzm 3y ago
Downtime but limited downtime since the data is stored with redundantly across multiple machines in the same AZ. So unless the AZ goes down (which is a different failure than what happened here) you can restart the DB on a different instance pretty quickly and I'm guessing AWS will do it automatically for you. edit: Remove triple as not certain about level of redundancy
- truetraveller 3y ago
  I don't believe their RDS / EBS has 3x redundancy. With SSD, that would be super costly for them. But if that's correct, that would be incredible.
  - marcinzm 3y ago
    May not be 3x but it is replicated so even a total instance failure would not make you lose data: >Amazon EBS volumes are designed to be highly available, reliable, and durable. At no additional charge to you, Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. For more details, see the Amazon EBS Service Level Agreement. https://aws.amazon.com/ebs/features/#Amazon_EBS_availability_and_durability https://aws.amazon.com/ebs/features/#Amazon_EBS_availability...
williamdclt 3y ago
> Maybe not with Aurora If a read replica fails, I'd expect no downtime (possibly a few errors as connections get cut off abruptly). Although there's always the risk that the remaining instances aren't able to handle the additional load. If the master fails, you'll get a ~2min downtime
api 3y ago
Yeah but you won't lose your data. They have backup infrastructure and EBS is rock solid. Down time is one thing. Data loss is something else.

capableweb 3y ago

> Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Depends on where in your development cycle you are. If you just got started and haven't even figured out what you're actually building (prototyping), you shouldn't really use a hosting provider that randomly lose instances. If you're on the other hand have done everything to improve your applications performance, had to resolve through-output issues with a distributed architecture and now running 10+ instances, then losing one host shouldn't impact you too much. But you really shouldn't start this way, it's doing web services the hard way and introduces a lot of complexity you shouldn't want to deal with when you're still trying to find product market fit.

riwsky 3y ago
GP is referring to fly.io architecting for single instance failures, not its customers.