6 ms·
S3: "Block Public Access is now enabled by default on new buckets." On the one hand, this is obviously the right decision. The number of giant data breeches ca
by simonw 1y ago
S3: "Block Public Access is now enabled by default on new buckets."
On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous.
But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!
- SOLAR_FIELDS 1y agoI honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug
- crinkly 1y agoI just stick CloudFront in front of those buckets. You don't need to expose the bucket at all then and can point it at a canonical hostname in your DNS.
- hnlmorg 1y agoThat’s definitely the “correct” way of doing things if you’re writing infra professionally. But I do also get that more casual users might prefer not to incur the additional costs nor complexity of having CloudFront in front. Though at that point, one could reasonably ask if S3 is the right choice for causal users.
- crinkly 1y agoIt's actually incredibly cheap. I think our software distribution costs, in the account I run, are around $2.00 a month. That's pushing out several thousand MSI packages a day.
- oblio 1y agoWith CloudFront?
- hnlmorg 1y agoS3 is actually quite expensive compared to the competition for both storage costs and egress costs. At a previous start-up, we had terrabytes of data on S3 and it was our second largest cost (after GPUs) and by some margin. For small scale stuff, S3s storage and egress charges are unlikely to be impactful. But it doesn’t mean they’re cheap relative to the competition. There are also ways you can reduce S3 costs, but then you're trading the costs received from AWS with the costs of hiring competent DevOps. Either way, you pay.
- gchamonlive 1y agoS3 + cloudfront is also incredibly popular so you can just find recipes for automating that in any technology you want, Terraform, ansible, plain bash scripts, Cloudformation (god forbid)
- gigatexal 1y agoYeah holy crap why is cloud formation so terrible?
- SteveNuts 1y agoLast time I tried to use CF, the third party IAC tools were faster to release new features than the functionality of CF itself. (Like Terraform would support some S3 bucket feature when creating a bucket, but CF did not). I'm not sure if that's changed recently, I've stopped using it.
- herpderperator 1y agoFor the sake of understanding, can you explain why putting CloudFront in front of the buckets helps?
- bhattisatish 1y agoCloudfront allows you to map your S3 with both - signed url's in case you want a session base files download - default public files, for e.g. a static site. You can also map a domain (sub-domain) to Cloudfront with a CNAME record and serve the files via your own domain. Cloudfront distributions are also CDN based. This way you serve files local to the users location, thus increasing the speed of your site. For lower to mid range traffic, cloudfront with s3 is cheaper as the network cost of cloudfront is cheaper. But for large network traffic, cloudfront cost can balloon very fast. But in those scenarios S3 costs are prohibitive too!
- dcminter 1y agoNot always that simple - for example if you want to automatically load /foo/index.html when the browser requests /foo/ you'll need to either use the web serving feature of S3 (bucket can't be private) or set up some lambda at edge or similar fiddly shenanigans.
- andrewmcwatters 1y agoThis sort of thing drives me nuts in interviews, when people are like, are you familiar with such-and-such technology? Yeah, what month?
- tester756 1y agoIf you're aware of changes, then explain that there were changes over time, that's it
- deleted 1y ago[deleted]
- andrewmcwatters 1y agoYou seem to be lacking the experience of what actually happens in interviews.
- reactordev 1y agoYou say this, someone challenges you, now you're on the defensive during an interview and everyone has a bad taste in their mouth. Yeah, that's how it goes.
- pas 1y agoThat's just the taste of iron from the blood after the duel. But this is completely normal after a formal challenge! Companies want real cyberwarriors, and the old (lame) rockstar ninjas that they hired 10 years ago are very prone to issuing these.
- reactordev 1y agoI don’t want to go to war, I just want a quiet house in the mountains and a career that allows me to think about things.
- andrewmcwatters 1y ago
- sylens 1y agoThe thing to keep in mind with the "Block Public Access" setting is that is a redundancy built in to save people from making really big mistakes. Even if you have a terrible and permissive bucket policy or ACLs (legacy but still around) configured for the S3 bucket, if you have Block Public Access turned on - it won't matter. It still won't allow public access to the objects within. If you turn it off but you have a well scoped and ironclad bucket policy - you're still good! The bucket policy will dictate who, if anyone, has access. Of course, you have to make sure nobody inadvertantly modifies that bucket policy over time, or adds an IAM role with access, or modifies the trust policy for an existing IAM role that has access, and so on.
- simonw 1y agoI think this is the key of why I find it confusing: I need a very clear diagram showing which rules override which other rules.
- saghm 1y agoMy understanding is that there isn't actually any "overriding" in the sense of two rules conflicting and one of them having to "win" and take effect. I think it's more that an enabled rule always is in effect, but it might overlap with another rule, in which case removing one of them still won't remove the restrictions on the area of overlap. It's possible I'm reading too much into your choice of words, but it does sound like there's a chance that the confusion is stemming from an incorrect assumption of how various permissions interact. That being said, there's certain a lot more that could into making a system like that easier for developers. One thing that springs to mind is tooling that can describe what rules are currently in effect that limit (or grant, depending on the model) permissions for something. That would make it more clear when there are overlapping rules that affect the permissions of something, which in turn would make it much more clear why something is still not accessible from a given context despite one of the rules being removed.
- jagged-chisel 1y agoIf one rule explicitly restricts access and another explicitly grants access, which one is in effect? Do restrictions override grants? Does a grant to GroupOne override a restriction to GroupAlpha when the authenticated use in is both groups? Do rules set by GodAdmin override rules set by AngelAdmin?
- awongh 1y agoThis is exactly what I use LLMs for. To just read the docs for me and pull out the base level demo code that's buried in all the AWS documentation. Once I have that I can also ask it for the custom tweaks I need.
- dcminter 1y agoThis could not possibly go wrong... You're braver than me if you're willing to trust the LLM here - fine if you're ready to properly review all the relevant docs once you have code in hand, but there are some very expensive risks otherwise.
- simianwords 1y agoThere’s nothing brave in this. It generally works the way it should and even if it doesn’t - you just go back to see what went wrong. I take code from stack overflow all the time and there’s like a 90% chance it can work. What’s the difference here?
- dcminter 1y agoWell, the "accidentally making the S3 bucket public" scenario would be a good one. If you review carefully with full understanding of what e.g. all your policies are doing then great, no problem. If you don't do that will you necessarily notice that you accidentally leaked customer data to the world? The problem isn't the LLM it's assuming its output is correct just the same as assuming Stack Overflow answers are correct without verifying/understanding them.
- simianwords 1y agoI agree but its about the extent. I'm willing to accept the risk of ocassionally making S3 public but getting things done much faster, much like I don't meticulously read documentation when I can get the answer from stackoverflow. If you are comparing with stackoverflow then I guess we are on the same page - most people are fine with taking stuff from stackoverflow and it doesn't count as "brave".
- reactordev 1y agoThey'll teach you how for $250 and a certification test...
- cedws 1y agoI’m getting deja vu, didn’t they already do this like 10 years ago because people kept leaving their buckets wide open?