3 ms·

Could somebody please explain, how come such a sophisticated PaaS provider as Heroku was not able to properly set up multiple AWS availability zones for their o

Could somebody please explain, how come such a sophisticated PaaS provider as Heroku was not able to properly set up multiple AWS availability zones for their own services? I would think that if somebody had expertise to do it flawlessly, it should have been them. I don't understand much in cloud devops, I just judge from recurring HN comments saying left and right "your startup should have configured up multiple AZ" when some AWS region is down.

samwillis 5y ago
I don’t know know how Heroku use AZs but I do know their platform in AWS EU-West-1 (Dublin) if funtioning fine. Dashboard + metrics all ok, haven’t deployed though. So that’s to say they aren’t dependent on US-East-1 for all internal operations if you aren’t in it. Edit: maybe the dashboard is just back up and I don’t see it down.
stickfigure 5y ago
Most heroku users are using relational databases like postgres, and relational databases don't failover transparently. If client apps can't failover to other AZs, why does it matter for heroku's service itself to do so? Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.
- more_corn 5y ago
  RDS has this mostly solved. They silently mirror the master into another AZ. If the primary fails the internal routing redirects to the clone.
  - stickfigure 5y ago
    Tell me, are transactions guaranteed to have replicated before they commit? If not, you have a major consistency issue during failover. Maybe your workload can accept dropping a window of transactions, maybe not.
    - electroly 5y ago
      They are. RDS multi-AZ replication is synchronous. Aurora is asynchronous, and multi-region replication is asynchronous, but a multi-AZ deployment of the "traditional" database engines in RDS is synchronous. It's rather the opposite of what you said before: if you stick to the old fashioned databases, this is easy to set up, failover is automatic, and no app-level changes are needed. I think you may have been thinking of multi-region, since you mentioned geographic failover. Multi-AZ isn't really geographic failover.
more_corn 5y ago
It is usually some unexpected secondary service that wasn’t known to be in the critical path. Or a hidden dependency on an aws service that didn’t fail over gracefully. I look forward to reading the post mortem.

samwillis 5y ago

I don’t know know how Heroku use AZs but I do know their platform in AWS EU-West-1 (Dublin) if funtioning fine. Dashboard + metrics all ok, haven’t deployed though. So that’s to say they aren’t dependent on US-East-1 for all internal operations if you aren’t in it. Edit: maybe the dashboard is just back up and I don’t see it down.

stickfigure 5y ago

Most heroku users are using relational databases like postgres, and relational databases don't failover transparently. If client apps can't failover to other AZs, why does it matter for heroku's service itself to do so? Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.

more_corn 5y ago
RDS has this mostly solved. They silently mirror the master into another AZ. If the primary fails the internal routing redirects to the clone.
- stickfigure 5y ago
  Tell me, are transactions guaranteed to have replicated before they commit? If not, you have a major consistency issue during failover. Maybe your workload can accept dropping a window of transactions, maybe not.
  - electroly 5y ago
    They are. RDS multi-AZ replication is synchronous. Aurora is asynchronous, and multi-region replication is asynchronous, but a multi-AZ deployment of the "traditional" database engines in RDS is synchronous. It's rather the opposite of what you said before: if you stick to the old fashioned databases, this is easy to set up, failover is automatic, and no app-level changes are needed. I think you may have been thinking of multi-region, since you mentioned geographic failover. Multi-AZ isn't really geographic failover.

more_corn 5y ago

It is usually some unexpected secondary service that wasn’t known to be in the critical path. Or a hidden dependency on an aws service that didn’t fail over gracefully. I look forward to reading the post mortem.