More info: Sounds like the backup generators operated fine, but only have about an hour of fuel in their individual storage tanks. There's an automated fuel del

More info: Sounds like the backup generators operated fine, but only have about an hour of fuel in their individual storage tanks. There's an automated fuel delivery system to keep them topped up which is the piece that failed, meaning the individual generators didn't get their tanks refilled and eventually ran out of fuel. This was compounded by the operations staff being busy with trying to restore the main power feed and somehow (not explained) didn't get a timely notification that the fuel delivery system itself had failed. AWS says they plan to install a backup fuel delivery system at this particular datacenter (it wasn't detailed whether this SPOF is common at other datacenters or if this was an outlier) and that they have already updated their notifications to be more aggressive.