7 ms·

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very litt

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably to provide the piping, and leave the rest of the end user, but at this point it's not enough. In my opinion, they need to heavily invest into the customer-facing layer of their applications that shows all the status, gaps and holes, or these things keep happening. Then again, they can continue as is, and the incidents will keep as is, and as long as nothing implodes and no big customer leaves, financially it makes sense to do nothing. Just thinking out loud.

MrBuddyCasino 6y ago
I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.
- marcocampos 6y ago
  How will Pulumi fix the issues you mentioned?
  - MrBuddyCasino 6y ago
    By doing what XML-with-schema did 15 years ago: give compile-time type checking, making sure at least the syntax and types are valid. Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything? Not my idea of fun. All of this is solved by using a proper programming language.
- boarnoah 6y ago
  Could you elaborate on what you think it does better than say Terraform. I looked at their comparison page https://www.pulumi.com/docs/intro/vs/terraform/ https://www.pulumi.com/docs/intro/vs/terraform/ and while there is an argument to be made re: being able to write in a more familiar language, its kind of a moot point when config is mostly key value pairs. The biggest paint point with something like Terraform is the fact that it can't actually validate cloud provider specific problems (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way (granted its much easier to write your own checks like that in language as opposed to writing a go module for Terraform providers).
  - jen20 6y ago
    > (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way Yes, the AWS Pulumi SDK package includes a discriminated union of valid instance types, and the type of the property `aws.ec2.Instance::instanceType` is that union. (Disclaimer: I am a former maintainer of Terraform and a contributor to Pulumi).
Osiris 6y ago
I used Azure for a year on a project and they did seem to have a pretty big focus on UI/UX. Without having previously done any cloud management work directly, I found it fairly straightforward.
ericbarrett 6y ago
I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody who knows it well, or who learns it well in the process. I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more. That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.
- bjclark 6y ago
  The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them. The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.
  - orf 6y ago
somethingwitty1 6y ago
I agree AWS can do a lot better with the tooling UX, but adding write permissions like this is a conscious decision. In this case, someone purposefully added it (outside the creation UX flow). I'm not sure we can put the blame squarely on AWS for this one. And they definitely haven't been doing nothing. They've drastically redesigned the permissions UX around S3 over the years. They introduced GuardDuty to allow automatic threat detection (such as S3 bucket having open permissions) which can be applied across your organization (not just an account). They definitely have to keep doing more, but implying they are doing nothing just isn't true.
rumanator 6y ago
> They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket." Afterwards you still get a bucket that does not have write access by default. To allow write access you need to specifically set bucket access policies that explicitly allow others to write to your bucket.
- ficklepickle 6y ago
  This. It is actually quite difficult to make a bucket writable by anyone. I don't understand how it can be done accidentally.
  - Corrado 6y ago
    In the past, S3 buckets were created with much less sane defaults and you didn't have the ability to unilaterally forbid any public access that you do today. Also, configuring Bucket Profiles is a confusing and hard to test situation. And if that weren't enough, S3 buckets also have ACLs (though they are deprecated) that represent yet another way to configure access to your buckets. So, from the confusing UX to the multiple permissions settings, I can easily see how a company could get in this situation. In fact, the problem is so bad that AWS has revised the whole S3 thing several times, with each new iteration more secure than the last. Unfortunately, that doesn't help the company that configured their buckets 5 years ago and haven't touched them since.
jedberg 6y ago
I disagree. They started by just providing the plumbing, and Twillio is using that plumbing. But today if you make an S3 bucket, they take you through a wizard that makes it secure by default. They also run audits for free and send you the results. They also have Lightsail that just does everything for you securely. They have lots of user friendly options.
politelemon 6y ago
I'm not seeing any UX problems here. They configured an S3 access policy incorrectly while troubleshooting, 5 years ago. FTFA: > During our incident review, we identified that this path was not initially configured with public write access when it was added in 2015. We implemented a change 5 months later while troubleshooting a problem with one of our build systems and the permissions on that path were not properly reset once the issue had been fixed. > { "Sid": "AllowPublicRead", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::media.twiliocdn.com/taskrouter/" }
- soco 6y ago
  This is exactly one of the things Terraform would catch and correct in a snap. I rather think Twilio was so confident before they had everything understood and smooth running that they didn't need other tools like Terraform. Infrastructure as code emerged as a thing EXACTLY because cloud's hundreds of consoles and checkboxes and whatnot, although each one simple and logical, spinned out of our mind's capacity to hold.
nathas 6y ago
There's a new tool around that helps these cases be prevented at an organization level called CfnGuard: https://aws.amazon.com/about-aws/whats-new/2020/06/introducing-aws-cloudformation-guard-preview/ https://aws.amazon.com/about-aws/whats-new/2020/06/introduci... Obviously you have to be using CloudFormation, so if they're using i.e. Terraform they're out of luck.
- aldarisbm 6y ago
  Hashicorp sentinel
zimbatm 6y ago
Things are much better than what they used to be. AWS Config is easy to setup and can be used to warn for that type of access-control issues. It's a good layer of additional defense. The new S3 UI is also much better and won't let you do insecure things by default. AWS also sends emails when they find an open bucket. I don't know exactly when those are triggered but I got them in the past.
cwhiz 6y ago
I agree to an extent. AWS is beginning to feel like a duct taped spaghetti mess. It is becoming increasingly difficult to do things “right” without getting deep into their confusing documentation. However, I’m not necessarily sure that it is the wrong approach. You can use AWS at almost any level you like and if you go deep into the woods with any service you’re eventually going to find yourself buried in documentation.
earthboundkid 6y ago
Amazon probably can't change S3 for backwards compatibility reasons (there was an outcry about just changing some legacy URLs last year), but they definitely can and should make an "S4" which would be an S3 compatible storage layer with sane defaults. Essentially, people only ever want one of two things: - A private FTP bucket - A public static asset website (with built-in CDN and HTTPS!) Thing people never want: - A public upload bucket Seriously, no one wants that in any circumstance, ever. It's a huge liability. I literally cannot imagine anyone wanting anything even similar to it. But many, many companies have accidentally made one with S3. While they're at it, S4 should also actually use IAM and all that other "modern" stuff that S3 doesn't have because it's so ancient.
- hackmiester 6y ago
  What's the fourth S stand for?
  - ls612 6y ago
    Super Simple Storage Service ;)
    - earthboundkid 6y ago
      Bingo!
  - rumanator 6y ago
    > What's the fourth S stand for? Second Simple Storage Service.
- rumanator 6y ago
  >Thing people never want: > >- A public upload bucket Thus is outright wrong. Public upload s3 buckets are AWS's go-to basic technique to handle file uploads. AWS's API Gateway has a max file limit, and AWS advises developers to leverage S3 and lambda triggers to implement that usecases.
  - 6y ago

MrBuddyCasino 6y ago

I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.

marcocampos 6y ago
How will Pulumi fix the issues you mentioned?
- MrBuddyCasino 6y ago
  By doing what XML-with-schema did 15 years ago: give compile-time type checking, making sure at least the syntax and types are valid. Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything? Not my idea of fun. All of this is solved by using a proper programming language.
boarnoah 6y ago
Could you elaborate on what you think it does better than say Terraform. I looked at their comparison page https://www.pulumi.com/docs/intro/vs/terraform/ https://www.pulumi.com/docs/intro/vs/terraform/ and while there is an argument to be made re: being able to write in a more familiar language, its kind of a moot point when config is mostly key value pairs. The biggest paint point with something like Terraform is the fact that it can't actually validate cloud provider specific problems (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way (granted its much easier to write your own checks like that in language as opposed to writing a go module for Terraform providers).
- jen20 6y ago
  > (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way Yes, the AWS Pulumi SDK package includes a discriminated union of valid instance types, and the type of the property `aws.ec2.Instance::instanceType` is that union. (Disclaimer: I am a former maintainer of Terraform and a contributor to Pulumi).

Osiris 6y ago

I used Azure for a year on a project and they did seem to have a pretty big focus on UI/UX. Without having previously done any cloud management work directly, I found it fairly straightforward.

ericbarrett 6y ago

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody who knows it well, or who learns it well in the process. I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more. That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.

bjclark 6y ago
The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them. The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.
- orf 6y ago

somethingwitty1 6y ago

I agree AWS can do a lot better with the tooling UX, but adding write permissions like this is a conscious decision. In this case, someone purposefully added it (outside the creation UX flow). I'm not sure we can put the blame squarely on AWS for this one. And they definitely haven't been doing nothing. They've drastically redesigned the permissions UX around S3 over the years. They introduced GuardDuty to allow automatic threat detection (such as S3 bucket having open permissions) which can be applied across your organization (not just an account). They definitely have to keep doing more, but implying they are doing nothing just isn't true.

rumanator 6y ago

> They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket." Afterwards you still get a bucket that does not have write access by default. To allow write access you need to specifically set bucket access policies that explicitly allow others to write to your bucket.

ficklepickle 6y ago
This. It is actually quite difficult to make a bucket writable by anyone. I don't understand how it can be done accidentally.
- Corrado 6y ago
  In the past, S3 buckets were created with much less sane defaults and you didn't have the ability to unilaterally forbid any public access that you do today. Also, configuring Bucket Profiles is a confusing and hard to test situation. And if that weren't enough, S3 buckets also have ACLs (though they are deprecated) that represent yet another way to configure access to your buckets. So, from the confusing UX to the multiple permissions settings, I can easily see how a company could get in this situation. In fact, the problem is so bad that AWS has revised the whole S3 thing several times, with each new iteration more secure than the last. Unfortunately, that doesn't help the company that configured their buckets 5 years ago and haven't touched them since.

jedberg 6y ago

I disagree. They started by just providing the plumbing, and Twillio is using that plumbing. But today if you make an S3 bucket, they take you through a wizard that makes it secure by default. They also run audits for free and send you the results. They also have Lightsail that just does everything for you securely. They have lots of user friendly options.

politelemon 6y ago

I'm not seeing any UX problems here. They configured an S3 access policy incorrectly while troubleshooting, 5 years ago. FTFA: > During our incident review, we identified that this path was not initially configured with public write access when it was added in 2015. We implemented a change 5 months later while troubleshooting a problem with one of our build systems and the permissions on that path were not properly reset once the issue had been fixed. > { "Sid": "AllowPublicRead", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::media.twiliocdn.com/taskrouter/" }

soco 6y ago
This is exactly one of the things Terraform would catch and correct in a snap. I rather think Twilio was so confident before they had everything understood and smooth running that they didn't need other tools like Terraform. Infrastructure as code emerged as a thing EXACTLY because cloud's hundreds of consoles and checkboxes and whatnot, although each one simple and logical, spinned out of our mind's capacity to hold.

nathas 6y ago

There's a new tool around that helps these cases be prevented at an organization level called CfnGuard: https://aws.amazon.com/about-aws/whats-new/2020/06/introducing-aws-cloudformation-guard-preview/ https://aws.amazon.com/about-aws/whats-new/2020/06/introduci... Obviously you have to be using CloudFormation, so if they're using i.e. Terraform they're out of luck.

aldarisbm 6y ago
Hashicorp sentinel

zimbatm 6y ago

Things are much better than what they used to be. AWS Config is easy to setup and can be used to warn for that type of access-control issues. It's a good layer of additional defense. The new S3 UI is also much better and won't let you do insecure things by default. AWS also sends emails when they find an open bucket. I don't know exactly when those are triggered but I got them in the past.

cwhiz 6y ago

I agree to an extent. AWS is beginning to feel like a duct taped spaghetti mess. It is becoming increasingly difficult to do things “right” without getting deep into their confusing documentation. However, I’m not necessarily sure that it is the wrong approach. You can use AWS at almost any level you like and if you go deep into the woods with any service you’re eventually going to find yourself buried in documentation.

earthboundkid 6y ago

Amazon probably can't change S3 for backwards compatibility reasons (there was an outcry about just changing some legacy URLs last year), but they definitely can and should make an "S4" which would be an S3 compatible storage layer with sane defaults. Essentially, people only ever want one of two things: - A private FTP bucket - A public static asset website (with built-in CDN and HTTPS!) Thing people never want: - A public upload bucket Seriously, no one wants that in any circumstance, ever. It's a huge liability. I literally cannot imagine anyone wanting anything even similar to it. But many, many companies have accidentally made one with S3. While they're at it, S4 should also actually use IAM and all that other "modern" stuff that S3 doesn't have because it's so ancient.

hackmiester 6y ago
What's the fourth S stand for?
- ls612 6y ago
  Super Simple Storage Service ;)
  - earthboundkid 6y ago
    Bingo!
- rumanator 6y ago
  > What's the fourth S stand for? Second Simple Storage Service.
rumanator 6y ago
>Thing people never want: > >- A public upload bucket Thus is outright wrong. Public upload s3 buckets are AWS's go-to basic technique to handle file uploads. AWS's API Gateway has a max file limit, and AWS advises developers to leverage S3 and lambda triggers to implement that usecases.
- 6y ago