5 ms·

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scar

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time. Before using AWS I was in the 'what morons!' camp whenever some company got breached via leaving things wide open on AWS. Now I'm in the 'ok I understand why its so common' camp.

txcwpalpha 6y ago
The problem with S3 is the exact same problem with everything else in AWS. In their quest to be compatible and attractive to every single possible niche use case of the world's largest companies, they completely forgot the lone developer who just wants to upload a couple files. You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible, but if you ask them a simple question about how to run a single bucket that doesn't also involve using all of their ML tools and global accelerator and cloudfront, they'll be blindsided. It's almost as if they consider the common use cases to be "too simple/obvious" and therefor they never bothered to create any documentation or put any thought into it.
- pantulis 6y ago
  Your point is good, but one could suppose Twilio is not a "lone developer".
  - txcwpalpha 6y ago
    Certainly not, and in their layers of defense against this stuff, they should have had processes to train people, audit bucket policies, conduct penetration tests etc, and there's no excuse for them not having caught it before. But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.
    - pantulis 6y ago
      I kind of miss the times when working with the cloud was easier or more straightforward. Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience. Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.
- sgt-serverless 6y ago
  We have been solving this issue with the app I have built Commandeer. Here is the main S3 dashboard https://imgur.com/gallery/JutGUI3 https://imgur.com/gallery/JutGUI3 And here is the view of an individual bucket https://imgur.com/gallery/0OlKY7Q https://imgur.com/gallery/0OlKY7Q . It let's you completely manage S3 and 25 other AWS and Cloud services from a desktop app. You can easily see permissions and also preview every type of file. We have also done this for DynamoDB, Lambda, CloudWatch Alarms/Logs/Rules. The tool was built out of our frustration that many have.
mjayhn 6y ago
You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even create buckets then do so. A lot of you should just focus on development work and stay far away from operations/sec tasks, and that's fine. I'm going to assume most of these buckets aren't even made via code and are manual. That's bad enough. I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.
- somehnguy 6y ago
  I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right? The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it. Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly. Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it. Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.
  - SteveNuts 6y ago
    To me this is like buying a high end sports car, and neglecting the maintenance because you never factored in the expensive maintenance, and therefore the true cost of what you're buying.
fisherjeff 6y ago
It's gotten better, if you can believe it. Regardless, I've managed to come up with one foolproof method for securing my data: Never put anything important on S3.

txcwpalpha 6y ago

The problem with S3 is the exact same problem with everything else in AWS. In their quest to be compatible and attractive to every single possible niche use case of the world's largest companies, they completely forgot the lone developer who just wants to upload a couple files. You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible, but if you ask them a simple question about how to run a single bucket that doesn't also involve using all of their ML tools and global accelerator and cloudfront, they'll be blindsided. It's almost as if they consider the common use cases to be "too simple/obvious" and therefor they never bothered to create any documentation or put any thought into it.

pantulis 6y ago
Your point is good, but one could suppose Twilio is not a "lone developer".
- txcwpalpha 6y ago
  Certainly not, and in their layers of defense against this stuff, they should have had processes to train people, audit bucket policies, conduct penetration tests etc, and there's no excuse for them not having caught it before. But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.
  - pantulis 6y ago
    I kind of miss the times when working with the cloud was easier or more straightforward. Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience. Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.
sgt-serverless 6y ago
We have been solving this issue with the app I have built Commandeer. Here is the main S3 dashboard https://imgur.com/gallery/JutGUI3 https://imgur.com/gallery/JutGUI3 And here is the view of an individual bucket https://imgur.com/gallery/0OlKY7Q https://imgur.com/gallery/0OlKY7Q . It let's you completely manage S3 and 25 other AWS and Cloud services from a desktop app. You can easily see permissions and also preview every type of file. We have also done this for DynamoDB, Lambda, CloudWatch Alarms/Logs/Rules. The tool was built out of our frustration that many have.

mjayhn 6y ago

You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even create buckets then do so. A lot of you should just focus on development work and stay far away from operations/sec tasks, and that's fine. I'm going to assume most of these buckets aren't even made via code and are manual. That's bad enough. I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.

somehnguy 6y ago
I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right? The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it. Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly. Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it. Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.
- SteveNuts 6y ago
  To me this is like buying a high end sports car, and neglecting the maintenance because you never factored in the expensive maintenance, and therefore the true cost of what you're buying.

fisherjeff 6y ago

It's gotten better, if you can believe it. Regardless, I've managed to come up with one foolproof method for securing my data: Never put anything important on S3.