3 ms·
I don't understand how this is even possible in a company operating on that scale. Granted, I'm a lowly scientific programmer with no clue about running a cloud
by keldaris 10y ago
I don't understand how this is even possible in a company operating on that scale. Granted, I'm a lowly scientific programmer with no clue about running a cloud infrastructure, but I would have imagined that there would be at least a pretense of oversight for destructive commands run in such an environment. A scheme as simple as "any destructive command run on S3 subsystems is automatically run in a dry run form, and requires independent confirmation by 2-3 other engineers to actually come into effect" would have prevented this altogether. Given the overall prominence of S3, this incident seems to demonstrate a rather callous attitude on the part of the organization.
- umanwizard 10y agoI thought the same thing before I went into the industry but now that I've been in it for a few years (including two at Amazon), it doesn't surprise me. I suspect locking everyone down in the way you suggest would cost more in lost productivity (and costs for the infrastructure that would be required for greater auditing, etc.) than is lost in outages like this.