11 ms·

Heroku's AWS outage post-mortem

202 points by mileszs 15y ago

watchandwait 15y ago
The AWS outage is definitely not over. Apparently RDS is built on EBS and they have not all been restored, I can tell you that first hand.
- watchandwait 15y ago
  UPDATE: we were fully restored after midnight last night. It is a very happy feeling!
adriand 15y ago
I'm very impressed by how they take responsibility for this, in their words: "HEROKU TAKES 100% OF THE RESPONSIBILITY FOR THE DOWNTIME AFFECTING OUR CUSTOMERS LAST WEEK." It would be both easy, tempting and heck, even reasonable to assign at least a portion of the blame to Amazon. Their approach is interesting because their customers already know that, but are likely to appreciate their forthright acceptance of responsibility. It's a good lesson. If I'm being totally honest I'd have to admit that, as a developer, I sometimes blame external services or events for things that I have at least partial control over. Perhaps I should adopt Heroku's approach instead.
- jarin 15y ago
  It's sort of counter-intuitive, but taking responsibility for something (even if it's not directly your fault) often has the effect of deflecting some of the anger from your customers/clients/boss/etc. Personally, I prefer to just get the blame part out of the way by taking responsibility and concentrate on the important things: fixing the problem and making sure it doesn't happen again. I think that deep down, people aren't that concerned with who's fault it was. They just want to know that someone is going to fix it.
  - ghshephard 15y ago
    The reason why you don't want to take responsibility, is that the liability comes along with that. If Heroku took the position that the AWS outage was a force de majeure, then their liability for recompense to their customers would have been minimized. By suggesting they take responsibility, they also are in a position where they have to make good for all of the downtime their customers experienced. Short term - that will be an expensive decision. Long term, I think it's the right thing to do. It certainly builds up my confidence level in them.
    - kowsik 15y ago
      As a PaaS vendor, they are supposed to abstract away from IaaS failures. And they were not to use a single region to host all their apps. I love Heroku and will continue to use them as long as I have the option to add affinity to my dynos and workers to spread across multiple regions of my choice. Coupled with anycast DNS support, this will be a very compelling offering, if they can pull it off. During the outage, all of our scale engines (http://blitz.io http://blitz.io) and our CouchDB cluster across the other AWS regions held up, but since the web-tier was down, the whole app went down.
- theoj 15y ago
  Do they have another option? If no site had survived the outage, then they may have been on to something. But with some sites surviving the outage, they just have no excuse. Sure you can blame AWS because they said multiple availability zones in the same region would work. But at the same time there is an expectation that a site like Heroku is knowledgeable enough and sophisticated enough to intelligently process what AWS says and determine what's appropriate for them.
waxman 15y ago
Thank you for taking full responsibility. Everyone makes mistakes, so what matters is how you deal with them. This was the right way to respond. Thanks.
bdb 15y ago
Where is Amazon's?
- deleted 15y ago
  [deleted]
dpcan 15y ago
What the hell? Why is everyone taking responsibilty and giving amazon a free ride? I'm a firm believer that only victims make excuses, and it's admirable to take responsibility, and maybe they should have more redundancy in place, but the way aws has been advertised, most of us felt this kind of thing should never happen even without a 100% uptime guarantee. So, take 100% of the responsibility, but I wouldn't think any less of heroku if they only took 50%.
- spicyj 15y ago
  Because Heroku isn't positioned as a way to manage your EC2 instances, it's a whole hosting solution. How much more confusing would it be for customers if Heroku said that they take responsibility for downtime that's not Amazon's fault? What counts as Amazon's fault? What about customers that know only Rails and don't care about how Heroku works on the backend?
  - dpcan 15y ago
    I get what they are doing, and I would probably handle it the same way, but I think it is perfectly ok for them to place some blame here as their downtime WAS because aws was down technically. I think they took it too far by taking 100% of the blame. I'm on heroku's side, I don't think it was fair to themselves to take 100% pf the blame.
    - elliottcarlson 15y ago
      As a developer who uses AWS, it's complicated, but I agree that Heroku is at 100% fault. If I were providing a paid service and my dedicated servers went offline due to some reason (let's say the fiber line gets cut by road maintenance crews) - my customers wouldn't care what happened - perhaps some will offer sympathy - but in the end it's my fault, and my responsibility to offer any kind of contingency plan - which generally includes load balancing over multiple datacenters etc. The same view should have been taken here and too much reliance was set on one region (even though Amazon promised it would be safe - this is their fault ;)). In the end, they should take full blame and now learn from their mistakes.
ekidd 15y ago
Kudos to Heroku for taking full responsibility, and for planning to engineer around these kinds of Amazon problems in the future. In particular, I'm delighted to hear that they plan to perform continuous backups on their shared databases: 3) CONTINUOUS DATABASE BACKUPS FOR ALL. One reason why we were able to fix the dedicated databases quicker has to do with the way that we do backups on them. In the new Heroku PostgreSQL service, we have a continuous backup mechanism that allows for automated recovery of databases... We are in the process of rolling out this updated backup system to all of our shared database servers; it’s already running on some of them and we are aiming to have it deployed to the remainder of our fleet in the next two weeks. Combined with multi-region support, this should make Heroku far more resilient in the future.
- tmeasday 15y ago
  Is this reasonable? I'm sure a lot of amazon hosted companies are thinking similar thoughts. But being on multiple Availability Zones was supposed to be bulletproof (according to Amazon). Now that we know that wasn't the case, is being hosted on multiple regions going to provide the necessary level of protection? Is it an over-reaction to say that relying completely on Amazon could now be seen as irresponsible to your users, given the magnitude of this event?
  - PakG1 15y ago
    The one problem I have with such concerns is this: what other viable options are there? Google AppSpot? Windows Azure? Perhaps. But AWS is flexible, very few stack limitations. The only other alternative I think is to go back to the pre-cloud era, when hosting was much more expensive, and outages were still possible, especially when you couldn't keep up with big traffic spikes. Honestly, I would prefer this kind of mass outage than the alternative. It's cheaper, easier, and I bet you there's still better uptime overall.
    - tmeasday 15y ago
      Agreed. On both counts. However obviously it's good PR, and we all appreciate the Mea Cupla from Heroku, the fact is, they are proposing to migrate to a situation where they are still completely reliant on AWS for their hosting. I'm just not sure you can really say "We don’t want to ever put our customers through something like this again and we’re working as hard as we can on making sure that we won’t ever have to.", when at the end of the day, you are again relying on a company that has failed you in the past. Not trying to attack Amazon or Heroku, I'm honestly intrigued by this issue; not to mention the fact that we are facing the exact same decision at work.
oomkiller 15y ago
I'd really love to know some details on the continuous backup stuff. Sounds cool.
- deleted 15y ago
  [deleted]
- fdr 15y ago
  The mechanism is PostgreSQL continuous archiving. http://github.com/heroku/wal-e http://github.com/heroku/wal-e This tool is still quite nascent. It received quite a trial by fire, having not (before this point) been revealed as a value-added feature to the service just yet in a wide scale.
- mechanical_fish 15y ago
  If I'm translating it correctly, this phrase is referring to database replication. I'm not deeply familiar with the Postgres versions of that (to my regret), but for the MySQL version you can read something like this: http://dev.mysql.com/doc/refman/5.0/en/replication.html http://dev.mysql.com/doc/refman/5.0/en/replication.html Better yet, find yourself a copy of High Performance MySQL.
  - fdr 15y ago
    WAL archive replay is also used for the PostgreSQL hot standby feature, aka replication. Combined with streaming you can get sub-second latency, but there's no reason you could just not use streaming and use WAL-E to syndicate WAL to thousands of replicants with hot standby enabled (albeit with lag). Use your imagination if you want to write a event-driven, high performance WAL streaming server. I haven't found the use case yet. MySQL has long relied on statement based replication, which can lead to server drift in the case of any nondeterministic query. This is a total killer for extensibility of the database, as well as correctness in general. It also has a row-based-replication variant that showed up around 2008 that represents a significant improvement, but the search results for "mysql rbr" might give you pause... My guess is this is why Amazon made the sensible choice to back their RDS (MySQL) product via DRBD and synchronous, block-device level replication, because there is no good application-level option for MySQL that is to be trusted. This technique can also be used with PostgreSQL. However, use of DRBD tends to have punishing performance impact, is complicated, and is not very suitable for a hot standby unless you write very complex shared-storage database software like Oracle RAC, hence why so much effort went into WAL-streaming hot standby in the PostgreSQL community. The DRBD option is venerable, dating back to the LiveJournal early days as their MySQL HA option, and probably before that (Credit to LiveJournal for well-documenting their HA setup, including their use of DRBD).
chubs 15y ago
This is why i love hosting on heroku: they'll work their butt off to get it fixed when its down, and i don't have to lift a finger. However, EBS has been long known to be a turd, its a pity they relied on it. Plus, if they had a way to bring it back up in a different region (eg the euro AWS infrastructure) at the flick of a switch, that'd make me less nervous...
chrisbaglieri 15y ago
I wish more companies (hell people) were as forthright, pragmatic, and sensible as the Heroku gang. Their breakdown and response to the outage is exactly what me as a paying customer wants to hear. Kudos!
chrisbaglieri 15y ago
"Block storage is not a cloud-friendly technology". Based on every post-mortem I've read thus far, it's clear how AWS and it's customers approach EBS will change.
greattypo 15y ago
It's impressive that they're taking full responsibility, but very surprised there's no mention of refunds..
- JonWood 15y ago
  Given that Heroku charges based on the time your application is up I wouldn't be surprised if everyone just gets a bill which doesn't include the time their sites were offline.
chrishenn 15y ago
Our monitoring systems picked up the problems right away. The on-call engineer quickly determined the magnitude of the problem and woke up the on-call Incident Commander. The IC contacted AWS, and began waking Heroku engineers to work on the problem. Once it became clear that this was going to be a lengthy outage, the Ops team instituted an emergency incident commander rotation of 8 hours per shift, keeping a fresh mind in charge of the situation at all time. Our support, data, and other engineering teams also worked around the clock. The system they are using (IC, ops, engineer teams, operational periods) is extremely similar to the Incident Command System. The ICS was developed about 40 years ago for fighting wildfires, but now most government agencies use it to manage any type of incident. I've experienced it first hand and can say it works very well, but I have never seen it used in this context. The great thing about it is it's expandability---it will work for teams of nearly any size. I'd be interested in seeing if any other technology companies/backend teams are using it. http://en.wikipedia.org/wiki/Incident_command_system http://en.wikipedia.org/wiki/Incident_command_system
markbao 15y ago
I wish Amazon was as good at communication and accountability as Heroku is.
mtw 15y ago
what about also spreading to multiple providers (i.e. also use rackspace cloud)? they'd be less dependant from amazon issues
deleted 15y ago
[deleted]
awicklander 15y ago
They gloss over their biggest failure; they weren't communicating or interacting with their customers at all. * http://twitter.com/#!/heroku http://twitter.com/#!/heroku * http://twitter.com/#!/herokustatus http://twitter.com/#!/herokustatus
- daniel02216 15y ago
  They show a history of updates to their status blog and to the herokustatus Twitter account since April 21, what do you mean by 'they weren't communicating'?
  - runningdogx 15y ago
    From 9:07 to 20:43, the status updates were generic and not very helpful in answering two questions customers want to know: 1. What exactly is going on? 2. When will it be fixed? In the middle of a crisis, saying "we're aware of the problem, and we're working hard to fix it," for hours does not really count as communication. It increases customer aggravation rather than decreasing it. Customers want to know answers to the above two questions. They don't care that you know about the problem and that you're working on it, unless you're not doing those two things in which case they will be (and should be) furious; those two things are expected. Barring the ability to tell your customers "we will be back up at X:00", I think the best approach is to share as much information as you can without getting into proprietary information. That's why I think GP considered their communication a failure. That's why I consider their communication a failure, although I've seen this pattern enough from different companies that I don't hold it against Heroku as long as they learn from it.
    - ceejayoz 15y ago
      Both questions were unanswerable. Even Amazon's estimates were wildly off, and they can actually look at the infrastructure. Heroku's only way to answer your questions would have been to lie.
trezor 15y ago
And now reddit is down again (posting/submitting is impossible). Probably yet another Amazon issue, yet again. In all fairness I've read that the reddit devs have made lots of boneheaded mistakes in their general infrastructure-design, but it still seems Amazon is not a very reliable platform to build your stuff on. Platforms built on top of Amazon's even less so.
- showerst 15y ago
  I think boneheaded is a strong word. They're solving a problem that very few sites have to solve (huge traffic with low cache-ability) with vastly less resources than the others who do solve them have. (FB, Twitter, etc). AMZN in general is a pretty solid 'platform' (especially if you're not using EBS), but because this whole 'cloud' thing is still partially uncharted territory, there are still holes, and you can't treat it like a normal web host.
- akl 15y ago
  Are you seriously not aware that reddit falls over pretty much all the time on its own?
AffableSpatula 15y ago
I don't think this is particularly 'honorable' or anything like that.. it's the only sensible stance for them to take. Let's be realistic about this; for most people using heroku the alternative would have been bare ec2, and could easily have suffered the same fate as on heroku. Everyone should feel positive that they got to spend ~60 hours just sitting around moaning about being let down, instead of having to sweat their nuts off attempting to rehabilitate crazy, suicidal infrastructure. Even taking this downtime into account, heroku is still cost effective for me in a lot of cases.
metageek 15y ago
>It's a big project, and it will inescapably require pushing more configuration options out to users (for example, pointing your DNS at a router chosen by geographic homing Heroku should save the customers this pain, by setting up anycast: https://secure.wikimedia.org/wikipedia/en/wiki/Anycast#Domain_Name_System https://secure.wikimedia.org/wikipedia/en/wiki/Anycast#Domai...

202 points by mileszs 15y ago

The AWS outage is definitely not over. Apparently RDS is built on EBS and they have not all been restored, I can tell you that first hand.

watchandwait 15y ago
UPDATE: we were fully restored after midnight last night. It is a very happy feeling!

adriand 15y ago

I'm very impressed by how they take responsibility for this, in their words: "HEROKU TAKES 100% OF THE RESPONSIBILITY FOR THE DOWNTIME AFFECTING OUR CUSTOMERS LAST WEEK." It would be both easy, tempting and heck, even reasonable to assign at least a portion of the blame to Amazon. Their approach is interesting because their customers already know that, but are likely to appreciate their forthright acceptance of responsibility. It's a good lesson. If I'm being totally honest I'd have to admit that, as a developer, I sometimes blame external services or events for things that I have at least partial control over. Perhaps I should adopt Heroku's approach instead.

jarin 15y ago
It's sort of counter-intuitive, but taking responsibility for something (even if it's not directly your fault) often has the effect of deflecting some of the anger from your customers/clients/boss/etc. Personally, I prefer to just get the blame part out of the way by taking responsibility and concentrate on the important things: fixing the problem and making sure it doesn't happen again. I think that deep down, people aren't that concerned with who's fault it was. They just want to know that someone is going to fix it.
- ghshephard 15y ago
  The reason why you don't want to take responsibility, is that the liability comes along with that. If Heroku took the position that the AWS outage was a force de majeure, then their liability for recompense to their customers would have been minimized. By suggesting they take responsibility, they also are in a position where they have to make good for all of the downtime their customers experienced. Short term - that will be an expensive decision. Long term, I think it's the right thing to do. It certainly builds up my confidence level in them.
  - kowsik 15y ago
    As a PaaS vendor, they are supposed to abstract away from IaaS failures. And they were not to use a single region to host all their apps. I love Heroku and will continue to use them as long as I have the option to add affinity to my dynos and workers to spread across multiple regions of my choice. Coupled with anycast DNS support, this will be a very compelling offering, if they can pull it off. During the outage, all of our scale engines (http://blitz.io http://blitz.io) and our CouchDB cluster across the other AWS regions held up, but since the web-tier was down, the whole app went down.
theoj 15y ago
Do they have another option? If no site had survived the outage, then they may have been on to something. But with some sites surviving the outage, they just have no excuse. Sure you can blame AWS because they said multiple availability zones in the same region would work. But at the same time there is an expectation that a site like Heroku is knowledgeable enough and sophisticated enough to intelligently process what AWS says and determine what's appropriate for them.

waxman 15y ago

Thank you for taking full responsibility. Everyone makes mistakes, so what matters is how you deal with them. This was the right way to respond. Thanks.

bdb 15y ago

Where is Amazon's?

deleted 15y ago
[deleted]

dpcan 15y ago

What the hell? Why is everyone taking responsibilty and giving amazon a free ride? I'm a firm believer that only victims make excuses, and it's admirable to take responsibility, and maybe they should have more redundancy in place, but the way aws has been advertised, most of us felt this kind of thing should never happen even without a 100% uptime guarantee. So, take 100% of the responsibility, but I wouldn't think any less of heroku if they only took 50%.

spicyj 15y ago
Because Heroku isn't positioned as a way to manage your EC2 instances, it's a whole hosting solution. How much more confusing would it be for customers if Heroku said that they take responsibility for downtime that's not Amazon's fault? What counts as Amazon's fault? What about customers that know only Rails and don't care about how Heroku works on the backend?
- dpcan 15y ago
  I get what they are doing, and I would probably handle it the same way, but I think it is perfectly ok for them to place some blame here as their downtime WAS because aws was down technically. I think they took it too far by taking 100% of the blame. I'm on heroku's side, I don't think it was fair to themselves to take 100% pf the blame.
  - elliottcarlson 15y ago
    As a developer who uses AWS, it's complicated, but I agree that Heroku is at 100% fault. If I were providing a paid service and my dedicated servers went offline due to some reason (let's say the fiber line gets cut by road maintenance crews) - my customers wouldn't care what happened - perhaps some will offer sympathy - but in the end it's my fault, and my responsibility to offer any kind of contingency plan - which generally includes load balancing over multiple datacenters etc. The same view should have been taken here and too much reliance was set on one region (even though Amazon promised it would be safe - this is their fault ;)). In the end, they should take full blame and now learn from their mistakes.

ekidd 15y ago

Kudos to Heroku for taking full responsibility, and for planning to engineer around these kinds of Amazon problems in the future. In particular, I'm delighted to hear that they plan to perform continuous backups on their shared databases: 3) CONTINUOUS DATABASE BACKUPS FOR ALL. One reason why we were able to fix the dedicated databases quicker has to do with the way that we do backups on them. In the new Heroku PostgreSQL service, we have a continuous backup mechanism that allows for automated recovery of databases... We are in the process of rolling out this updated backup system to all of our shared database servers; it’s already running on some of them and we are aiming to have it deployed to the remainder of our fleet in the next two weeks. Combined with multi-region support, this should make Heroku far more resilient in the future.

tmeasday 15y ago
Is this reasonable? I'm sure a lot of amazon hosted companies are thinking similar thoughts. But being on multiple Availability Zones was supposed to be bulletproof (according to Amazon). Now that we know that wasn't the case, is being hosted on multiple regions going to provide the necessary level of protection? Is it an over-reaction to say that relying completely on Amazon could now be seen as irresponsible to your users, given the magnitude of this event?
- PakG1 15y ago
  The one problem I have with such concerns is this: what other viable options are there? Google AppSpot? Windows Azure? Perhaps. But AWS is flexible, very few stack limitations. The only other alternative I think is to go back to the pre-cloud era, when hosting was much more expensive, and outages were still possible, especially when you couldn't keep up with big traffic spikes. Honestly, I would prefer this kind of mass outage than the alternative. It's cheaper, easier, and I bet you there's still better uptime overall.
  - tmeasday 15y ago
    Agreed. On both counts. However obviously it's good PR, and we all appreciate the Mea Cupla from Heroku, the fact is, they are proposing to migrate to a situation where they are still completely reliant on AWS for their hosting. I'm just not sure you can really say "We don’t want to ever put our customers through something like this again and we’re working as hard as we can on making sure that we won’t ever have to.", when at the end of the day, you are again relying on a company that has failed you in the past. Not trying to attack Amazon or Heroku, I'm honestly intrigued by this issue; not to mention the fact that we are facing the exact same decision at work.

oomkiller 15y ago

I'd really love to know some details on the continuous backup stuff. Sounds cool.

deleted 15y ago
[deleted]
fdr 15y ago
The mechanism is PostgreSQL continuous archiving. http://github.com/heroku/wal-e http://github.com/heroku/wal-e This tool is still quite nascent. It received quite a trial by fire, having not (before this point) been revealed as a value-added feature to the service just yet in a wide scale.
mechanical_fish 15y ago
If I'm translating it correctly, this phrase is referring to database replication. I'm not deeply familiar with the Postgres versions of that (to my regret), but for the MySQL version you can read something like this: http://dev.mysql.com/doc/refman/5.0/en/replication.html http://dev.mysql.com/doc/refman/5.0/en/replication.html Better yet, find yourself a copy of High Performance MySQL.
- fdr 15y ago
  WAL archive replay is also used for the PostgreSQL hot standby feature, aka replication. Combined with streaming you can get sub-second latency, but there's no reason you could just not use streaming and use WAL-E to syndicate WAL to thousands of replicants with hot standby enabled (albeit with lag). Use your imagination if you want to write a event-driven, high performance WAL streaming server. I haven't found the use case yet. MySQL has long relied on statement based replication, which can lead to server drift in the case of any nondeterministic query. This is a total killer for extensibility of the database, as well as correctness in general. It also has a row-based-replication variant that showed up around 2008 that represents a significant improvement, but the search results for "mysql rbr" might give you pause... My guess is this is why Amazon made the sensible choice to back their RDS (MySQL) product via DRBD and synchronous, block-device level replication, because there is no good application-level option for MySQL that is to be trusted. This technique can also be used with PostgreSQL. However, use of DRBD tends to have punishing performance impact, is complicated, and is not very suitable for a hot standby unless you write very complex shared-storage database software like Oracle RAC, hence why so much effort went into WAL-streaming hot standby in the PostgreSQL community. The DRBD option is venerable, dating back to the LiveJournal early days as their MySQL HA option, and probably before that (Credit to LiveJournal for well-documenting their HA setup, including their use of DRBD).

chubs 15y ago

This is why i love hosting on heroku: they'll work their butt off to get it fixed when its down, and i don't have to lift a finger. However, EBS has been long known to be a turd, its a pity they relied on it. Plus, if they had a way to bring it back up in a different region (eg the euro AWS infrastructure) at the flick of a switch, that'd make me less nervous...

chrisbaglieri 15y ago

I wish more companies (hell people) were as forthright, pragmatic, and sensible as the Heroku gang. Their breakdown and response to the outage is exactly what me as a paying customer wants to hear. Kudos!

chrisbaglieri 15y ago

"Block storage is not a cloud-friendly technology". Based on every post-mortem I've read thus far, it's clear how AWS and it's customers approach EBS will change.

greattypo 15y ago

It's impressive that they're taking full responsibility, but very surprised there's no mention of refunds..

JonWood 15y ago
Given that Heroku charges based on the time your application is up I wouldn't be surprised if everyone just gets a bill which doesn't include the time their sites were offline.

chrishenn 15y ago

Our monitoring systems picked up the problems right away. The on-call engineer quickly determined the magnitude of the problem and woke up the on-call Incident Commander. The IC contacted AWS, and began waking Heroku engineers to work on the problem. Once it became clear that this was going to be a lengthy outage, the Ops team instituted an emergency incident commander rotation of 8 hours per shift, keeping a fresh mind in charge of the situation at all time. Our support, data, and other engineering teams also worked around the clock. The system they are using (IC, ops, engineer teams, operational periods) is extremely similar to the Incident Command System. The ICS was developed about 40 years ago for fighting wildfires, but now most government agencies use it to manage any type of incident. I've experienced it first hand and can say it works very well, but I have never seen it used in this context. The great thing about it is it's expandability---it will work for teams of nearly any size. I'd be interested in seeing if any other technology companies/backend teams are using it. http://en.wikipedia.org/wiki/Incident_command_system http://en.wikipedia.org/wiki/Incident_command_system

markbao 15y ago

I wish Amazon was as good at communication and accountability as Heroku is.

mtw 15y ago

what about also spreading to multiple providers (i.e. also use rackspace cloud)? they'd be less dependant from amazon issues

deleted 15y ago

[deleted]

awicklander 15y ago

They gloss over their biggest failure; they weren't communicating or interacting with their customers at all. * http://twitter.com/#!/heroku http://twitter.com/#!/heroku * http://twitter.com/#!/herokustatus http://twitter.com/#!/herokustatus

daniel02216 15y ago
They show a history of updates to their status blog and to the herokustatus Twitter account since April 21, what do you mean by 'they weren't communicating'?
- runningdogx 15y ago
  From 9:07 to 20:43, the status updates were generic and not very helpful in answering two questions customers want to know: 1. What exactly is going on? 2. When will it be fixed? In the middle of a crisis, saying "we're aware of the problem, and we're working hard to fix it," for hours does not really count as communication. It increases customer aggravation rather than decreasing it. Customers want to know answers to the above two questions. They don't care that you know about the problem and that you're working on it, unless you're not doing those two things in which case they will be (and should be) furious; those two things are expected. Barring the ability to tell your customers "we will be back up at X:00", I think the best approach is to share as much information as you can without getting into proprietary information. That's why I think GP considered their communication a failure. That's why I consider their communication a failure, although I've seen this pattern enough from different companies that I don't hold it against Heroku as long as they learn from it.
  - ceejayoz 15y ago
    Both questions were unanswerable. Even Amazon's estimates were wildly off, and they can actually look at the infrastructure. Heroku's only way to answer your questions would have been to lie.

trezor 15y ago

And now reddit is down again (posting/submitting is impossible). Probably yet another Amazon issue, yet again. In all fairness I've read that the reddit devs have made lots of boneheaded mistakes in their general infrastructure-design, but it still seems Amazon is not a very reliable platform to build your stuff on. Platforms built on top of Amazon's even less so.

showerst 15y ago
I think boneheaded is a strong word. They're solving a problem that very few sites have to solve (huge traffic with low cache-ability) with vastly less resources than the others who do solve them have. (FB, Twitter, etc). AMZN in general is a pretty solid 'platform' (especially if you're not using EBS), but because this whole 'cloud' thing is still partially uncharted territory, there are still holes, and you can't treat it like a normal web host.
akl 15y ago
Are you seriously not aware that reddit falls over pretty much all the time on its own?

AffableSpatula 15y ago

I don't think this is particularly 'honorable' or anything like that.. it's the only sensible stance for them to take. Let's be realistic about this; for most people using heroku the alternative would have been bare ec2, and could easily have suffered the same fate as on heroku. Everyone should feel positive that they got to spend ~60 hours just sitting around moaning about being let down, instead of having to sweat their nuts off attempting to rehabilitate crazy, suicidal infrastructure. Even taking this downtime into account, heroku is still cost effective for me in a lot of cases.

metageek 15y ago

>It's a big project, and it will inescapably require pushing more configuration options out to users (for example, pointing your DNS at a router chosen by geographic homing Heroku should save the customers this pain, by setting up anycast: https://secure.wikimedia.org/wikipedia/en/wiki/Anycast#Domain_Name_System https://secure.wikimedia.org/wikipedia/en/wiki/Anycast#Domai...