6 ms·

"by nbashaw 28 minutes ago" I thought for sure I'd have missed it and this would be one of those reports where the service was back up before the story gained

"by nbashaw 28 minutes ago" I thought for sure I'd have missed it and this would be one of those reports where the service was back up before the story gained traction, but as of 12:07 PM Pacific/US time I cannot navigate to Amazon's home page.. The amazing thing about this for me is that it reminds me that it was only a few years ago that even the biggest sites would have fairly frequent multi-hour outages, but these days it is pretty rare for this sort of thing to happen, particularly on a retail or otherwise direct-money generating site.

dguaraglia 14y ago
12:18 PST, still down. Someone must be getting his ass chewed about this.
- garretruh 14y ago
  Fired is more like it. Though I doubt Jeff Bezos is the kind of guy to do something like that.
  - cl8ton 14y ago
    12:22 PST still down... Looking for Mushroom cloud to the NW
    - farnsworth 14y ago
      12:26 PST and no problems accessing from Seattle.
      - bink-lynch 14y ago
        12:34 PST and OK from Vegas.
  - potatolicious 14y ago
    I used to work for Amazon. Unless this was willful I doubt anyone is going to get fired for it. After all, you'd be firing the guy who's least likely to make such a mistake ever again.
    - mattmaroon 14y ago
      It's so silly to fire people for mistakes that no organization that exerts any mental capacity toward human resources would do such a thing. Even Apple didn't fire the guy who left a prototype at the bar. In a factory in 1965, maybe, but no good employer is going to fire someone for making a mistake, no matter how costly.
      - stcredzero 14y ago
        > In a factory in 1965, maybe Depends on the factory, and I doubt it's much more or less likely in the 21st century than it was in 1965. In all times and places, you have enlightened and unenlightened people. In all times and places you have good and bad leaders. "In every time, in every place, the deeds of men remain the same."
        scaphandre 14y ago
        > "In every time, in every place, the deeds of men remain the same." Citation needed. I'm pretty sure there are fewer war deaths and fewer kittens burned for entertainment per head now than 200 years ago.
        stcredzero 14y ago
        It's a quote. I leave that as homework. Yes, there is less bad stuff, but the quote says nothing of frequency.
    - kami8845 14y ago
      The story of the plane servicer who never mis-serviced a plane again after almost killing the pilot he worked for is cute, but screwing up big-time doesn't turn a (presumably) sloppy engineer into one that never messes up again.
dos1 14y ago
On the plus side, if they're willing to share, I bet this will be a very interesting postmortem. Presumably Amazon.com is one of the more bulletproof web properties in the world. Whatever could have occurred to take it down for nearly an hour (at this point) can only be interesting!
- nirvana 14y ago
  I can't compare to other web properties, but when I worked at Amazon, the store going down was a regular event. Something broke almost daily, though it was rare for the whole store to go down. (EG: You might not be able to search, or checkout might be down, etc.) The store went thru periods of relative stability, and relative lack of stability, and in the periods where it was not doing so well, it (or a major piece of functionality) would go down in some key area at least once a week, sometimes multiple times a week during the holidays. While it's been several years and I'm sure they've improved reliability, the sheer mass of the store made it very slow to evolve. And as an ex-amazonian sometimes I go and check for bugs that were issues back in the day- several of them have come back over the years, which is not surprising given that the entire group that was working on the parts I was working on disbanded because so many people were driven off by bad management. (A one-two punch in that case, a bad manager backed by another bad manager, neither of which had any technical knowledge.) At the time I worked there, large swaths of code in the store had no team who was responsible because the team had been disbanded in one of the regular shuffles of employees. Amazon had a tendency to get a team together to do a feature, launch it, get the PR and the stock bump, then disband the team and put them on other projects. Of course some of these things stuck around if they were successful, but there was a lot of cruft from past efforts like: Local restaurant menus, the movie times system, various "social shopping features" (a perennial favorite to try again and again.) Hell, they used to have catalogs for mail order merchants- scanned paper catalogs! At the time, they were claiming that "AWS is what we built the amazon store on!" (which was totally false, S3 was engineered completely separately from the store, and to its credit, as obidos and gurupa were crap. The only thing the store shared with AWS for at least the first several years was being hosted in some of the same datacenters.) At least at the time I worked there, I'd call it a mess held together by the code equivalents of duct tape and bailing wire. One of the things Amazon excels at is customer service, so when these problems would impact the customer, their bacon was often saved by customer support fixing the problem manually (eg: messed up orders, etc.) Granted, operating at Amazon's scale is not trivial matter. But Amazon is a retailer and stock marketing company (Eg: one of their primary products is Amazon stock), more than an engineering company. I'm kinda amazed that people perceive them as a "tech giant" along with Google, Facebook and Amazon. Shows the power of a good (actually, GREAT) side business like AWS. They get the credit for building something good and scalable with AWS, but of course it was a separate team lead by a senior executive with enough political clout to shelter that team.
chaz 14y ago
eBay's outage in 1999 was for 22 hours [1]. That painful experience completely changed their entire internal engineering process. [1] http://www.internetnews.com/ec-news/article.php/137251/Cost+Of+eBays+22Hour+Outage+Put+At+2+Million.htm http://www.internetnews.com/ec-news/article.php/137251/Cost+...

dguaraglia 14y ago

12:18 PST, still down. Someone must be getting his ass chewed about this.

garretruh 14y ago
Fired is more like it. Though I doubt Jeff Bezos is the kind of guy to do something like that.
- cl8ton 14y ago
  12:22 PST still down... Looking for Mushroom cloud to the NW
  - farnsworth 14y ago
    12:26 PST and no problems accessing from Seattle.
    - bink-lynch 14y ago
      12:34 PST and OK from Vegas.
- potatolicious 14y ago
  I used to work for Amazon. Unless this was willful I doubt anyone is going to get fired for it. After all, you'd be firing the guy who's least likely to make such a mistake ever again.
  - mattmaroon 14y ago
    It's so silly to fire people for mistakes that no organization that exerts any mental capacity toward human resources would do such a thing. Even Apple didn't fire the guy who left a prototype at the bar. In a factory in 1965, maybe, but no good employer is going to fire someone for making a mistake, no matter how costly.
    - stcredzero 14y ago
      > In a factory in 1965, maybe Depends on the factory, and I doubt it's much more or less likely in the 21st century than it was in 1965. In all times and places, you have enlightened and unenlightened people. In all times and places you have good and bad leaders. "In every time, in every place, the deeds of men remain the same."
      - scaphandre 14y ago
        > "In every time, in every place, the deeds of men remain the same." Citation needed. I'm pretty sure there are fewer war deaths and fewer kittens burned for entertainment per head now than 200 years ago.
        stcredzero 14y ago
        It's a quote. I leave that as homework. Yes, there is less bad stuff, but the quote says nothing of frequency.
  - kami8845 14y ago
    The story of the plane servicer who never mis-serviced a plane again after almost killing the pilot he worked for is cute, but screwing up big-time doesn't turn a (presumably) sloppy engineer into one that never messes up again.

dos1 14y ago

On the plus side, if they're willing to share, I bet this will be a very interesting postmortem. Presumably Amazon.com is one of the more bulletproof web properties in the world. Whatever could have occurred to take it down for nearly an hour (at this point) can only be interesting!

nirvana 14y ago
I can't compare to other web properties, but when I worked at Amazon, the store going down was a regular event. Something broke almost daily, though it was rare for the whole store to go down. (EG: You might not be able to search, or checkout might be down, etc.) The store went thru periods of relative stability, and relative lack of stability, and in the periods where it was not doing so well, it (or a major piece of functionality) would go down in some key area at least once a week, sometimes multiple times a week during the holidays. While it's been several years and I'm sure they've improved reliability, the sheer mass of the store made it very slow to evolve. And as an ex-amazonian sometimes I go and check for bugs that were issues back in the day- several of them have come back over the years, which is not surprising given that the entire group that was working on the parts I was working on disbanded because so many people were driven off by bad management. (A one-two punch in that case, a bad manager backed by another bad manager, neither of which had any technical knowledge.) At the time I worked there, large swaths of code in the store had no team who was responsible because the team had been disbanded in one of the regular shuffles of employees. Amazon had a tendency to get a team together to do a feature, launch it, get the PR and the stock bump, then disband the team and put them on other projects. Of course some of these things stuck around if they were successful, but there was a lot of cruft from past efforts like: Local restaurant menus, the movie times system, various "social shopping features" (a perennial favorite to try again and again.) Hell, they used to have catalogs for mail order merchants- scanned paper catalogs! At the time, they were claiming that "AWS is what we built the amazon store on!" (which was totally false, S3 was engineered completely separately from the store, and to its credit, as obidos and gurupa were crap. The only thing the store shared with AWS for at least the first several years was being hosted in some of the same datacenters.) At least at the time I worked there, I'd call it a mess held together by the code equivalents of duct tape and bailing wire. One of the things Amazon excels at is customer service, so when these problems would impact the customer, their bacon was often saved by customer support fixing the problem manually (eg: messed up orders, etc.) Granted, operating at Amazon's scale is not trivial matter. But Amazon is a retailer and stock marketing company (Eg: one of their primary products is Amazon stock), more than an engineering company. I'm kinda amazed that people perceive them as a "tech giant" along with Google, Facebook and Amazon. Shows the power of a good (actually, GREAT) side business like AWS. They get the credit for building something good and scalable with AWS, but of course it was a separate team lead by a senior executive with enough political clout to shelter that team.

chaz 14y ago

eBay's outage in 1999 was for 22 hours [1]. That painful experience completely changed their entire internal engineering process. [1] http://www.internetnews.com/ec-news/article.php/137251/Cost+Of+eBays+22Hour+Outage+Put+At+2+Million.htm http://www.internetnews.com/ec-news/article.php/137251/Cost+...