4 ms·

All I can say is this: If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

by bagosm 12y ago

maxlybbert 12y ago
Not to be too flippant, but the company's closing shop. So, yeah, the DevOps are fired, along with everybody else. As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.
- judk 12y ago
  Two-factor authentication is a second basket. Sending a copy to Glacier is a second basket. Does Amazon not have 30-day undelete for bulk storage? Seems crazy.
  - MaulingMonkey 12y ago
    > Two-factor authentication is a second basket. Not in my books. Recommended to be sure, but this has far too many single points of failure. To name a few: - Software corrupts data - Hardware corrupts data - Social engineering bypasses 2FA
  - JohnTHaller 12y ago
    > Two-factor authentication is a second basket. Two-factor authentication is a second basket is a better carrying strap but it's still on a single basket.
dijit 12y ago
DevOps means different things to different people. most people use the term to mean 'a coder who can configure apache' this whole incident reeks of a lack of OpSec and general Ops knowledge. Lack of offsite backups is one thing, but if you're using cloud infrastructure you never place all eggs in one basket. your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.
- robryk 12y ago
  > your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild. Or rather shutdown, preserve, start replacements. IMHO your automation should be able to deal with the old preserved instances being left alone.
deleted 12y ago
[deleted]
toomuchtodo 12y ago
DevOps here. There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access). Show of hands how many people here are doing it this way.
- deleted 12y ago
  [deleted]
- mentat 12y ago
  Seriously, if your root account and all full admin accounts aren't using MFA you're just asking for it. Also if you're not using purpose specific access keys, you're just asking for it. If the first thing you do isn't calling AWS support, wow...
  - guiambros 12y ago
    Couldn't agree more. Everything under a single platform, no MFA, no (real) offsite backup, and on top of that they spent 12 hours corresponding with the attacker, instead of immediately calling Amazon to ask their help to shut down everything, while they still had time? I'm sorry, but this is a succession of things not to do in terms of system operations. Probably the team never managed mission critical platforms before, and hopefully they now learned the lesson.
    - jacquesm 12y ago
      How many companies have not yet learned that lesson? There are probably a lot of codespaces on AWS. My reasoning is that if you make it so that a developer can set up a virtual datacenter but does not have the background of actually running such an installation then you're going to have to assume that it is probably quite fragile. Software people tend to make all kinds of assumptions about hardware that do not work out in practice.
      - 12y ago

All I can say is this: If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

by bagosm 12y ago

All I can say is this: If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

maxlybbert 12y ago

Not to be too flippant, but the company's closing shop. So, yeah, the DevOps are fired, along with everybody else. As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.

judk 12y ago
Two-factor authentication is a second basket. Sending a copy to Glacier is a second basket. Does Amazon not have 30-day undelete for bulk storage? Seems crazy.
- MaulingMonkey 12y ago
  > Two-factor authentication is a second basket. Not in my books. Recommended to be sure, but this has far too many single points of failure. To name a few: - Software corrupts data - Hardware corrupts data - Social engineering bypasses 2FA
- JohnTHaller 12y ago
  > Two-factor authentication is a second basket. Two-factor authentication is a second basket is a better carrying strap but it's still on a single basket.

dijit 12y ago

DevOps means different things to different people. most people use the term to mean 'a coder who can configure apache' this whole incident reeks of a lack of OpSec and general Ops knowledge. Lack of offsite backups is one thing, but if you're using cloud infrastructure you never place all eggs in one basket. your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.

robryk 12y ago
> your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild. Or rather shutdown, preserve, start replacements. IMHO your automation should be able to deal with the old preserved instances being left alone.

deleted 12y ago

[deleted]

toomuchtodo 12y ago

DevOps here. There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access). Show of hands how many people here are doing it this way.

deleted 12y ago
[deleted]
mentat 12y ago
Seriously, if your root account and all full admin accounts aren't using MFA you're just asking for it. Also if you're not using purpose specific access keys, you're just asking for it. If the first thing you do isn't calling AWS support, wow...
- guiambros 12y ago
  Couldn't agree more. Everything under a single platform, no MFA, no (real) offsite backup, and on top of that they spent 12 hours corresponding with the attacker, instead of immediately calling Amazon to ask their help to shut down everything, while they still had time? I'm sorry, but this is a succession of things not to do in terms of system operations. Probably the team never managed mission critical platforms before, and hopefully they now learned the lesson.
  - jacquesm 12y ago
    How many companies have not yet learned that lesson? There are probably a lot of codespaces on AWS. My reasoning is that if you make it so that a developer can set up a virtual datacenter but does not have the background of actually running such an installation then you're going to have to assume that it is probably quite fragile. Software people tend to make all kinds of assumptions about hardware that do not work out in practice.
    - 12y ago