4 ms·

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody who knows it well, or who learns it well in the process. I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more. That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.

bjclark 6y ago
The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them. The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.
- orf 6y ago
  Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.
  - morvita 6y ago
    > and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.
    - aldarisbm 6y ago
      But terraform is using the aws sdk behind the covers. Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)
      - deleted 6y ago
        [deleted]
    - orf 6y ago
      You can use Cloudformation. Like, what do you want here? Some super console that nobody asked for because anyone who knows what they are doing knows why you want infrastructure as code?
- aldarisbm 6y ago
  Might be a war-crime, but people at the enterprise level like twilio, don't do anything at the UI level, and if someone was to do something at the UI level, AWS blocks public access _by default_ AWS console was only complicated as a complete noob. Once you understand how services work you actually realize is decent.
txcwpalpha 6y ago
>That said, I don't blame Twilio for not catching this I do. Twilio is a multi-billion dollar company and there is no excuse for them not having proper security processes to catch stuff like this early. Even if we take the "S3 is hard" arguments at face value, this wasn't a 0-day or some complicated unpredictable exploit. This was an extremely basic misconfiguration on a mission-critical part of their architecture that would have been caught by even the simplest out-of-the-box penetration test or audit. For a company like Twilio to not be doing basic, fundamental stuff like that is a big deal and they certainly should be blamed for it.
aaronbrethorst 6y ago
your S3 bucket permissions are clearly worthy of periodic review. better tooling makes this easier to do. And you should have this done by somebody who knows it well, or who learns it well in the process. No one knows S3 better than the AWS S3 team. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK Or a small feature team could spend a month or two improving the user experience so that literally every S3 customer doesn't have to have the technical acumen, knowledge of risks, and time to get this right. 0.5 days * a million users (to be conservative) is 500,000 days.

bjclark 6y ago

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them. The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.

orf 6y ago
Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.
- morvita 6y ago
  > and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.
  - aldarisbm 6y ago
    But terraform is using the aws sdk behind the covers. Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)
    - deleted 6y ago
      [deleted]
  - orf 6y ago
    You can use Cloudformation. Like, what do you want here? Some super console that nobody asked for because anyone who knows what they are doing knows why you want infrastructure as code?
aldarisbm 6y ago
Might be a war-crime, but people at the enterprise level like twilio, don't do anything at the UI level, and if someone was to do something at the UI level, AWS blocks public access _by default_ AWS console was only complicated as a complete noob. Once you understand how services work you actually realize is decent.

txcwpalpha 6y ago

>That said, I don't blame Twilio for not catching this I do. Twilio is a multi-billion dollar company and there is no excuse for them not having proper security processes to catch stuff like this early. Even if we take the "S3 is hard" arguments at face value, this wasn't a 0-day or some complicated unpredictable exploit. This was an extremely basic misconfiguration on a mission-critical part of their architecture that would have been caught by even the simplest out-of-the-box penetration test or audit. For a company like Twilio to not be doing basic, fundamental stuff like that is a big deal and they certainly should be blamed for it.

aaronbrethorst 6y ago

your S3 bucket permissions are clearly worthy of periodic review. better tooling makes this easier to do. And you should have this done by somebody who knows it well, or who learns it well in the process. No one knows S3 better than the AWS S3 team. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK Or a small feature team could spend a month or two improving the user experience so that literally every S3 customer doesn't have to have the technical acumen, knowledge of risks, and time to get this right. 0.5 days * a million users (to be conservative) is 500,000 days.