9 ms·

I understand a certain level of displeasure at their lack of specificity while they mitigated the issue. But... in this case, the time to remediate doesn't rea

by davidu 13y ago

I understand a certain level of displeasure at their lack of specificity while they mitigated the issue. But... in this case, the time to remediate doesn't really change your response to the threat. No matter what, you need to change all your keys, generate new private keys, etc. They got it fixed within 48 hours, globally, which, if you ask me, is incredible at their scale. I would hardly describe anything AWS does as amateur. But maybe that's just me.

deleted 13y ago
[deleted]
- deleted 13y ago
  [deleted]
thathonkey 13y ago
Calling the team at AWS "amateurs" is a great way to discredit everything the author wrote. AWS is a gigantic infrastructure and they got everything fixed IN LESS THAN 48 HOURS. That is not an amateur response time. Sorry their updates weren't to your liking but they were responding and posting bulletins the whole time and again: they solved the issue very quickly given the number of clients they support.
- chris_wot 13y ago
  The way they communicated it was very, very amateurish. They needed to know when they could re-key their certs - seems like it was impossible to tell. That's something that needed to be done ASAP. Without knowing precisely when the environment that affected them was updated, this client couldn't get that done as quickly as they might have.
  - exelius 13y ago
    It's not amateurish, it was just poor communication in a situation that they've (fortunately) not had to deal with. Which happens from time to time; and good companies recognize their failures and fix them. One thing I know about Amazon from friends who work there is that they don't tolerate failure. They have a culture of owning your mistakes and fixing them; anyone who doesn't buy into that attitude will get fired pretty quickly (and Amazon fires a lot of people.) It's pretty fucking professional to update the infrastructure that runs half the internet in under 48 hours with no issues. But again, communication can be a problem when you have as many customers as they do. OP raised some legitimate concerns, but his credibility was undercut by attacking Amazon and calling them names. Ironically, his post was a much more amateur move as his concerns would likely be taken more seriously if he had stuck to the issues and not resorted to name-calling. The essence of professionalism is sticking to the issues at hand and not being sidetracked by extraneous factors.
    - alttab 13y ago
      The nail in the coffin was some comment at the end about moving off of AWS to something else over blog posts lacking timestamps.
- kordless 13y ago
  > sorry their updates weren't to your liking but... Folks, this is an account created 19 hours ago making half an apology and then rationalizing not listening to customers because they solved a "big" problem quickly. That's an ad hominum argument and as a result, a big tell on intent. From my perspective, pushing out a new SSL build to a bunch of load balancers in a highly automated network like AWS is probably, by this point, a trivial task. Actually listening to the customer and responding decently is MUCH harder. Clearly it could be done better, which is the point of the post. Rise above getting offended/scared about being called "amateurs" and start talking more about what goes on in that creepy black box that is AWS. You owe the world that much, at the very least.
  - thathonkey 13y ago
    Hey there super sleuth... not sure what you're trying to imply but I'm not affiliated with Amazon nor do I use their AWS service. I do not speak for them. I am not making an apology on their behalf. I suggest you stop levying false accusations and avoid using words that you simply don't understand ie. ad hominum (not only did you spell it wrong, but you've misapplied it). To clarify: I'm a long time HN reader that finally got around to making an account (and certainly not for the express purpose of defending Amazon). However, I did want to call the author out on writing a terribly unfair knee-jerk, heat-of-the-moment indictment of AWS (this type of thing is unfortunately all-too-common in the tech community: actual amateurs writing as if they are a central authority about subjects that they have something approaching 0 understanding of. For example: the multitude of complex engineering and PR challenges a service provider like AWS faces during something like the Great OpenSSL Exploit of 2014). What I'm trying to say is: cut them some slack. Their response seemed perfectly reasonable to me. Hope this helps.
    - narrowrail 13y ago
      As a long time HN reader, you should know that the condescending "super sleuth" mention was unecessary. It should also be apparent that your highly defensive comment with a very new account would raise eyebrows. The "hope this helps" ending also comes off as passive-aggressive. We can do better.
sebgeelen 13y ago
The thing AWS does which can be described as amateur, was the communication. AWS has always been really quiet when there's an issue on their services. It's a nightmare to know when your problem is due to your infrastructure or if their's a bigger scale issue at AWS cause they never talk about it...
- mkr-hn 13y ago
  Amazon has a status page for all AWS services and provides RSS feeds for each: http://status.aws.amazon.com/ http://status.aws.amazon.com/
  - sebgeelen 13y ago
    I know. Sadly this page is not updated often enough. I mean, when there's a known issue on one of the aws service this page display a little "i" icon, which is barely visible. And when, you encounter some problem with your AWS stuff that clearly come from their side, if the problem it's not wide, they just say nothing. At that point you can search yourself for hour to be sure it's not your responsibility, and after-woods, you just wait, blind.
  - deleted 13y ago
    [deleted]
  - madaxe_again 13y ago
    Trust me, this doesn't cover all their issues. Don't get me wrong, we love our AWS stack, but we've had all sorts of weird stuff happen over the years, from zombie ELBs hitting the wrong hosts to invisible SGs... sometimes we see from twitter that others are seeing the same things, but AWS don't ever confirm or deny anything, they just stall then tell you when it's fixed. I guess its support in terms of something to give to your PHB, but when something goes wrong within AWS, it's a very black box - which isn't surprising.
- madaxe_again 13y ago
  They're not amateur. We pay for their premium support and their communication is professional, polite, and usually totally useless.
coherentpony 13y ago
You're absolutely right. The author of the post is just whining for the sake of whining. For every post hating on amazon there's one hating on heroku<http://www.holovaty.com/writing/aws-notes> http://www.holovaty.com/writing/aws-notes>. Best course of action is to just deal with it.

by davidu 13y ago

deleted 13y ago

[deleted]

deleted 13y ago
[deleted]

thathonkey 13y ago

Calling the team at AWS "amateurs" is a great way to discredit everything the author wrote. AWS is a gigantic infrastructure and they got everything fixed IN LESS THAN 48 HOURS. That is not an amateur response time. Sorry their updates weren't to your liking but they were responding and posting bulletins the whole time and again: they solved the issue very quickly given the number of clients they support.

chris_wot 13y ago
The way they communicated it was very, very amateurish. They needed to know when they could re-key their certs - seems like it was impossible to tell. That's something that needed to be done ASAP. Without knowing precisely when the environment that affected them was updated, this client couldn't get that done as quickly as they might have.
- exelius 13y ago
  It's not amateurish, it was just poor communication in a situation that they've (fortunately) not had to deal with. Which happens from time to time; and good companies recognize their failures and fix them. One thing I know about Amazon from friends who work there is that they don't tolerate failure. They have a culture of owning your mistakes and fixing them; anyone who doesn't buy into that attitude will get fired pretty quickly (and Amazon fires a lot of people.) It's pretty fucking professional to update the infrastructure that runs half the internet in under 48 hours with no issues. But again, communication can be a problem when you have as many customers as they do. OP raised some legitimate concerns, but his credibility was undercut by attacking Amazon and calling them names. Ironically, his post was a much more amateur move as his concerns would likely be taken more seriously if he had stuck to the issues and not resorted to name-calling. The essence of professionalism is sticking to the issues at hand and not being sidetracked by extraneous factors.
  - alttab 13y ago
    The nail in the coffin was some comment at the end about moving off of AWS to something else over blog posts lacking timestamps.
kordless 13y ago
> sorry their updates weren't to your liking but... Folks, this is an account created 19 hours ago making half an apology and then rationalizing not listening to customers because they solved a "big" problem quickly. That's an ad hominum argument and as a result, a big tell on intent. From my perspective, pushing out a new SSL build to a bunch of load balancers in a highly automated network like AWS is probably, by this point, a trivial task. Actually listening to the customer and responding decently is MUCH harder. Clearly it could be done better, which is the point of the post. Rise above getting offended/scared about being called "amateurs" and start talking more about what goes on in that creepy black box that is AWS. You owe the world that much, at the very least.
- thathonkey 13y ago
  Hey there super sleuth... not sure what you're trying to imply but I'm not affiliated with Amazon nor do I use their AWS service. I do not speak for them. I am not making an apology on their behalf. I suggest you stop levying false accusations and avoid using words that you simply don't understand ie. ad hominum (not only did you spell it wrong, but you've misapplied it). To clarify: I'm a long time HN reader that finally got around to making an account (and certainly not for the express purpose of defending Amazon). However, I did want to call the author out on writing a terribly unfair knee-jerk, heat-of-the-moment indictment of AWS (this type of thing is unfortunately all-too-common in the tech community: actual amateurs writing as if they are a central authority about subjects that they have something approaching 0 understanding of. For example: the multitude of complex engineering and PR challenges a service provider like AWS faces during something like the Great OpenSSL Exploit of 2014). What I'm trying to say is: cut them some slack. Their response seemed perfectly reasonable to me. Hope this helps.
  - narrowrail 13y ago
    As a long time HN reader, you should know that the condescending "super sleuth" mention was unecessary. It should also be apparent that your highly defensive comment with a very new account would raise eyebrows. The "hope this helps" ending also comes off as passive-aggressive. We can do better.

sebgeelen 13y ago

The thing AWS does which can be described as amateur, was the communication. AWS has always been really quiet when there's an issue on their services. It's a nightmare to know when your problem is due to your infrastructure or if their's a bigger scale issue at AWS cause they never talk about it...

mkr-hn 13y ago
Amazon has a status page for all AWS services and provides RSS feeds for each: http://status.aws.amazon.com/ http://status.aws.amazon.com/
- sebgeelen 13y ago
  I know. Sadly this page is not updated often enough. I mean, when there's a known issue on one of the aws service this page display a little "i" icon, which is barely visible. And when, you encounter some problem with your AWS stuff that clearly come from their side, if the problem it's not wide, they just say nothing. At that point you can search yourself for hour to be sure it's not your responsibility, and after-woods, you just wait, blind.
- deleted 13y ago
  [deleted]
- madaxe_again 13y ago
  Trust me, this doesn't cover all their issues. Don't get me wrong, we love our AWS stack, but we've had all sorts of weird stuff happen over the years, from zombie ELBs hitting the wrong hosts to invisible SGs... sometimes we see from twitter that others are seeing the same things, but AWS don't ever confirm or deny anything, they just stall then tell you when it's fixed. I guess its support in terms of something to give to your PHB, but when something goes wrong within AWS, it's a very black box - which isn't surprising.
madaxe_again 13y ago
They're not amateur. We pay for their premium support and their communication is professional, polite, and usually totally useless.

coherentpony 13y ago

You're absolutely right. The author of the post is just whining for the sake of whining. For every post hating on amazon there's one hating on heroku<http://www.holovaty.com/writing/aws-notes> http://www.holovaty.com/writing/aws-notes>. Best course of action is to just deal with it.