18 ms·
Amazon S3 will no longer support path-style API requests
- gcb0 7y agoDoes that mean people still have tons of public-by-mistake s3 buckets because of their clumsy UI, and they just gave up and are swiping what's left under the rug?
- ceejayoz 7y agoThis wouldn't have any impact on that issue. Those accidentally-public buckets are already accessible in both the path-based and the virtualhost-based method.
- scarface74 7y agoYou really have to try to make a bucket public. Even when you do, you get warnings within the UI, and there is a column showing you it’s public. Is there even a UI option to make a bucket public anymore? I always edit the bucket policy and add the JSON to make it public read only.
- ceejayoz 7y agoA lot of that is a fairly recent development, though. Lots of buckets laying around from the years it largely didn’t warn you.
- marcinzm 7y agoThey're not public by "mistake". They're public because someone was lazy, wanted to share something and didn't want to bother with authentication. They knew they were making it public but figured no one would find it except the desired recipient so why put in the effort to make a better solution.
- cddotdotslash 7y agoThis is going to break so many legacy codebases in ways I can't even imagine. Edit: Could they have found a better place to announce this than a forum post?
- Sytten 7y agoThere is probably a PR document in the process of being released, it's in more than a year after all.
- Rexxar 7y agoCouldn't they do a redirection (301) to not break code ?
- notmyname 7y agoNo, because path-style bucket names weren't originally required to conform to dns naming limitations. I don't know how they're going to migrate those older non-conforming buckets to the host-style form.
- pluc 7y agoYou're on Hacker News..
- mark242 7y agoThis will hopefully prevent malicious sites hosted on v1-style buckets from stealing cookies/localstorage/credentials/etc.
- judge2020 7y agoCare to elaborate? Why would there be any secrets stored via s3.amazonaws.com?
- ARandomerDude 7y agoTL;DR Migrate from: s3.amazonaws.com/<bucketname>/key to: <bucketname>.s3.amazonaws.com/key no later than: September 30th, 2020
- TheLoneTechNerd 7y agoDoes anyone have insight on why they're making this change? All they say in this post is "In our effort to continuously improve customer experience". From my point of view as a customer, I don't really see an experiential difference between a subdomain style and a path style - one's a ".", the other's a "/" - but I imagine there's a good reason for the change.
- ynniv 7y agoI have no visibility into Amazon, but using subdomains let you shard across multiple IP addresses.
- zmmmmm 7y agoThe only obvious thing that occurs to me is that bringing the bucket into the domain name puts it under the same-origin policy in the browser security model. Perhaps there are a significant number of people hosting their buckets and compromising security this way? Not something I have heard of but it seems possible. Makes me wonder if they are specifically not mentioning it because this is the reason and they know there are vulnerable applications in the wild and they don't want to draw attention to it?
- TheLoneTechNerd 7y agoRemoving my comments because I can't seem to delete them...
- tedunangst 7y agoDoes it bother you the domain is amazon.com and not com.amazon?
- yjftsjthsd-h 7y agoI can't read what you're replying to, but it absolutely bothers me. The current scheme has this completely random double reversal in the middle of the URL; it would have been so trivial to just make it actually big-endian, but instead we have this big-little-big endian nonsense. Far too late to change it now, but it is ugly and annoying.
- jasonpeacock 7y agoIMO, this is an improvement - it makes it clear that the bucket is global and public, whereas with the path you could believe that it was only visible when logged into your account. It also helps people understand why the bucket name is restricted in it's naming.
- ceejayoz 7y ago> it makes it clear that the bucket is global and public How does it do that? You can host a private bucket at foo.s3.amazonaws.com just fine.
- geofft 7y agoI think the claim is that the namespace is global and public, i.e., you and I can't both have buckets named "foo". There is only one S3 bucket named "foo" in the world. If it's https://s3.amazonaws.com/foo/ https://s3.amazonaws.com/foo/ you could believe that it's based on your cookies or something, but if it's https://foo.s3.amazonaws.com/ https://foo.s3.amazonaws.com/ it's more obvious that it's a global namespace in the same way DNS domain names are (and that it's possible to tell if a name is already in use by someone else, too).
- manigandham 7y agoThis is most likely to help mitigate the domain being abused for browser security due to the same-origin policy. This is very common when dealing with malware, phishing, and errant JS files.
- astrocat 7y agoAmazon explicitly recommends naming buckets like "example.com" and "www.example.com" : https://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html https://docs.aws.amazon.com/AmazonS3/latest/dev/website-host... Now, it seems, this is a big problem. V2 resource requests will look like this: https://example.com.s3.amazonaws.com/.. https://example.com.s3.amazonaws.com/... or https://www.example.com.s3.amazonaws.com/.. https://www.example.com.s3.amazonaws.com/... And, of course, this ruins https. Amazon has you covered for * .s3.amazonaws.com, but not for * .* .s3.amazonaws.com or even * .* .* .s3.amazonaws... and so on. So... I guess I have to rename/move all my buckets now? Ugh.
- dfsegoat 7y agoFWIW - I found it fairly trivial to set up CloudFront in front of my buckets [1], so that I can use HTTPS with AWS Cert Mgr (ACM) to serve our s3 sites on https://mydomain.com https://mydomain.com [2]. I set this up some time ago using our domain name and ACM, and I don't think I will need to change anything in light of this announcement. 1 - https://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-cloudfront-walkthrough.html https://docs.aws.amazon.com/AmazonS3/latest/dev/website-host... 2 - https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html https://docs.aws.amazon.com/acm/latest/userguide/acm-overvie...
- thayne 7y agoThat isn't a solution for every use case. For example, it means you can't use the s3 VPC gateway for those buckets.
- jimmychangas 7y agoCare to elaborate? Do you mean S3 VPC Endpoints? Because this could screw many in-VPC Lambdas that need S3.
- thayne 7y agoyes, that is what I mean. If your bucket name contains a dot, you will no longer be able to access it with https with an S3 VPC Endpoint. (using http or going to cloudfront instead of the S3 VPC Endpoint would still work)
- deleted 7y ago[deleted]
- samat 7y agoOne important implication is that collateral freedom techniques [1] using Amazon S3 will no longer work. To put it simply, right now I could put some stuff not liked by Russian or Chinese government (maybe entire website) and give a direct s3 link to https:// https:// s3 .amazonaws.com/mywebsite/index.html. Because it's https — there is no way man in the middle knows what people read on s3.amazonaws.com. With this change — dictators see my domain name and block requests to it right away. I don't know if they did it on purpose or just forgot about those who are less fortunate in regards to access to information, but this is a sad development. This censorship circumvention technique is actively used in the wild and loosing Amazon is no good. 1 https://en.wikipedia.org/wiki/Collateral_freedom https://en.wikipedia.org/wiki/Collateral_freedom
- samat 7y agoIf there is anyone from Amazon caring about freedom of speech and censorship — please contact me at s@samat.me, I'd love to give you more perspective on this.
- cle 7y agoHow about asking someone from Amazon to give you more perspective on this, too? You only seem interested in immediately accusing Amazon of not caring about freedom of speech and schooling them about it, without considering if there are legitimate technical reasons that might justify the tradeoff. I can think of many such reasons off the top of my head.
- sieabahlpark 7y agoWhatever gets the clicks amirite?
- ironmagma 7y agoWith Amazon being the ones making the change, this situation is asymmetric. It’s on the affected to convince the “affectors,” if you will, that what they are doing is a bad idea. Whether they convince us or not is irrelevant.
- blairanderson 7y agoTL;DR https://news.ycombinator.com/item?id=19821813 https://news.ycombinator.com/item?id=19821813
- RocketSyntax 7y agoBoo. Now old packages won't work.
- segmondy 7y agoSo much for customer obsession.
- phlakaton 7y agoHm. I had a local testing setup using an S3 standin service from localstack and a Docker Compose cluster, and path-style addressing made that pretty easy to set up. Anyone else in that "bucket?" Suggestions on the best workaround?
- yeahitslikethat 7y ago"In our effort to continuously improve customer experience, the path-style naming convention is being retired in favor of virtual-hosted style request format. Customers should update their applications" How does forcing customers to rewrite their code to confirm to this change, improve customer experience?
- blantonl 7y agoMaybe as the technical debt continues to come due with the current architecture, it's time to make the hard choices to keep a good customer experience?
- yeahitslikethat 7y agoIt's on amazon to pay off their technical debt. Not their customers. They are turning off a feature at their customers expense. That's the exact opposite of good customer service.
- bagels 7y agoFun fact: The s3 console as of right now still shows v1 urls when you look at the overview page for a key/file.
- geekrax 7y agoThere are millions of results for "https://s3.amazonaws.com/" https://s3.amazonaws.com/" on GitHub: http://bit.ly/2GUVjDi http://bit.ly/2GUVjDi
- BillinghamJ 7y agoGitHub search is really poor. It is also including the uses of the subdomain style.
- geekrax 7y agoAgreed. The search could use some love for doing exact match. The scale at which different libraries, tools, and systems depending on hard-coded S3 urls will break by this change is insane.
- caseymarquis 7y agoI'm so glad I saw this. I would have been very confused when this went live had I not seen this post today. I wish I could upvote this more.
- blantonl 7y agoI'm kind of shocked at some of the responses here... everything from outrage, to expressing dismay at how many things could break, to how hard this is to fix, to accusing Amazon of all kinds of nefarious things. How hard is it for 99% of the developers and technical leaders here to search your codebase for s3.amazonaws.com and update your links in the next 18 months?
- simonw 7y agoYou're assuming that most code is actively maintained. It isn't. This is going to break so much older stuff.
- jhall1468 7y agoWhile true, I don't think backwards compatibility forever is the right solution to the old code problem.
- peterwwillis 7y agoIt's a reasonable timeframe, but not all codebases are actively maintained. In addition, it's concievable that there's some hidden custom library somewhere that crafts S3 URIs, making it near impossible to simply grep for a certain URI type in the codebase. So people may have to scour codebases they don't even maintain to look for random code which may craft an S3 URI in a certain way, then fork that project, fix the functionality, publish it, and use the fork. Then they may need to fork every other project that uses that original project, and do the same thing ad infinitum. If this is a private company, they have to do all that within some corp-wide globally available private repo, which either means (1) making this repo public on the internet, or (2) adding it to every security group they have that pulls code. It may even require adding direct connect or privatelink. So that means a long research project, followed by a project of fixing, testing, and releasing new code, followed by a project get network access to the custom repos and change software to use them. So, surprisingly hard, but doable. And from the customer's perspective, a huge pain in the ass, just to save Amazon some pennies on bandwidth.
- fpgaminer 7y ago> How hard is it for 99% of the developers and technical leaders here to search your codebase for s3.amazonaws.com and update your links in the next 18 months? I've got a number of hobby projects, some hosted on AWS, that I built ages ago. I have no idea how this change will effect those projects because ... I just frankly don't remember the codebases. I built them on a weekend, set them up, and now just use them. It isn't the end of the world. But I'm not really excited about having to dig up old code, re-grok it, and fix anything that changes like these might break. I suppose that's just the nature of a developer's life. But I think many of us long for a "write once, run forever" world. Horror stories about legacy software aside, it was nice to be able to write software for Windows and then have it work a decade later.
- sly010 7y agoIs there a deprecation announcement that does not include the phrase "In our effort to continuously improve customer experience"? Edit: autotypo
- massung 7y agoAnyone know if this will affect the internal use (e.g. EMR) s3 schema: s3://bucket/path/key?
- BillinghamJ 7y agoNo it won't
- electrum 7y agoNo. The file system implementation uses the AWS S3 client, which will automatically use virtual-host style when possible (if the bucket name supports it).
- merb 7y agoI see a problem when using the s3 library to other services that support s3 but only have some kind of path style access like minio or ceph with no subdomains enabled. it will break once their java api removes the old code.
- euank 7y agoOne of the weird peculiarities of path-style API requests was that it meant CORS headers meant nothing for any bucket pretty much. I wrote a post about this a bit ago [0]. I guess after this change, the cors configuration will finally do something! On the flip side, anyone who wants to list buckets entirely from the client-side javascript sdk won't be able to anymore unless Amazon also modifies cors headers on the API endpoint further after disabling path-style requests. [0]: https://euank.com/2018/11/12/s3-cors-pfffff.html https://euank.com/2018/11/12/s3-cors-pfffff.html
- ajcodez 7y agoI switched to MinIO for anything new. Happy user - https://min.io/ https://min.io/
- sl1ck731 7y agoDoes the "you are no longer logged in" screen not infuriate anyone besides me? There doesn't seem any purpose to it just redirecting you to the landing page when you were trying to access a forum post that doesn't even require you be logged in. Absolutely mind boggling with as much as they pay people they do something so stupid and haven't changed it after so long.
- pulkitsh1234 7y agoag -o 'https?://s3.amazonaws.com.*?\/.*?\/'| awk -F':' '{print $1, $4}' | sort | uniq | cut -d'/' -f 1 | sort | uniq -c | gsort -h -rk1,1 For anyone interesting in finding out the occurrences in their codebase. (Mac)
- lazyant 7y ago`In our effort to continuously improve customer experience` , what's the actual driver here, I don't see how going from two to one option and forcing you to change if you are in the wrong one improves my experience.
- a13n 7y agoReduce complexity for future customers
- thaumasiotes 7y agohttp://chainsawsuit.com/comic/2017/12/07/improvements/ http://chainsawsuit.com/comic/2017/12/07/improvements/ > We asked our investors and they said you're very excited about it being less good, which is great news for you!
- btown 7y agoWhat kind of company deprecates a URL format that's still recommended by the Object URL in the S3 Management Console? https://www.dropbox.com/s/zzr3r1nvmx6ekct/Screenshot%202019-05-03%2019.32.48.png?dl=0 https://www.dropbox.com/s/zzr3r1nvmx6ekct/Screenshot%202019-... There are so, SO many teams that use S3 for static assets, make sure it's public, and copy that Object URL. We've done this at my company, and I've seen these types of links in many of our partners' CSS files. These links may also be stored deep in databases, or even embedded in Markdown in databases. This will quite literally cause a Y2K-level event, and since all that traffic will still head to S3's servers, it won't even solve any of their routing problems. Set it as a policy for new buckets, if you must, if you change the Object URL output and have a giant disclaimer. But don't. Freaking. Break. The. Web.
- EugeneOZ 7y agoAlso in millions of manuals, generated PDFs, sent emails... Some things you just can't "update" anymore.. It's really disastrous change for the web data integrity.
- hueving 7y agoOne of the magicians in Las Vegas (the one at the MGM) even used s3 image links in emails to send emails to everyone "predicting" the contents of something that hadn't happened.
- paulddraper 7y agoDavid Copperfield does that.
- deleted 7y ago[deleted]
- parktheredcar 7y agoI think recent years have proven that despite all the memeing about things like rest, "don't break the web" is not a value shared by all of the parties involved. The takeaway is that for those of us that do still wish to uphold those values, we can let this serve as a lesson that we should not publish assets behind urls we don't control.
- xyzzy_plugh 7y agoThis will break software updates for so many systems, probably even some Amazon devices.
- tomc1985 7y agoPlay stupid cloud provider games, win stupid cloud provider prizes Amazon waving its dictatorial wand about is exactly why it's better to take ownership of your infra. It makes me so sad to watch the tech world become Amazon's bitch
- pronoiac 7y agoFor other folks looking for announcement feeds, see https://forums.aws.amazon.com/rss.jspa https://forums.aws.amazon.com/rss.jspa - announcements are the asterisks.
- etxm 7y agoThis is going to be the Y2K of September 2020.
- reilly3000 7y agoI was already planning a move to GCP, but this certainly helps. Now that cloud is beating retail in earnings, the ‘optimizations’ come along with it. That and BigQuery is an amazing tool. It’s not like I’m super outraged that they would change their API, the reasoning seems sound. It’s just that if I have to touch S3 paths everywhere I may as well move them elsewhere to gain some synergies with GCP services. I would think twice if I were heavy up on IAM roles and S3 Lambda triggers, but that isn’t the case.
- chillaxtian 7y agoA similar removal is coming in just 2 months for V2 signatures: https://forums.aws.amazon.com/ann.jspa?annID=5816 https://forums.aws.amazon.com/ann.jspa?annID=5816 This could be just as disruptive. Difficult to say that they will actually follow through, as the only mention of this date is in the random forum post I linked.
- scrollaway 7y agoDoubtful, sigv2 is not supported in all regions. So all current software that wants to be compatible with more than a portion of regions has to be compatible. This is a great way of introducing breaking changes. Imagine that for example, ipv6 would be at near 100% adoption if "new websites" were only available over v6.
- miguelmota 7y agoAlways confused me how they had two different ways of retrieving the same object. Glad that they're sticking to the subdomain option. Sucks to go back and check for old urls though. This change might break a good chunk of the web.
- ec109685 7y agoAmazon is proud that they never break backwards compatibility like this. Quotes like the container you are running on Fargate will keep running 10 years from now. Something weird is going on if they don’t keep path style domains working for existing buckets.
- quickthrower2 7y agoOnly 10 years. Shame that that is a boast. 100 years would be better.
- swiley 7y agoCommercial platform breaks things people have built on it for "the sake of continuously improving customer experience. " Also: see photos of your favorite celebrity walking their dog and other news at 11.
- niyazpk 7y agoOne way to do this without breaking existing applications would be to charge more for the path style requests for a while. Then deprecate once enough people have moved away from it, so that less people are outraged by the change.
- deleted 7y ago[deleted]
- interfixus 7y ago> In our effort to continuously improve customer experience, [feature x] is being retired In this case, the most highly improved experience I can think of eould be that of sundry nefarious entities monitoring internet traffic.
- iamgopal 7y agoThey should produce a free redirect service at least.
- jasonkester 7y agoI wonder how they’ll handle capitalized bucket names. This seems like it will break that. S3 has been around a long time, and they made some decisions early on that they realised wouldn’t scale, so they reversed them. This v1 vs v2 url thing is one of them. But another was letting you have “BucketName” and “bucketname” as two distinct buckets. You can’t name them like that today, but you could at first, and they still work (and are in conflict under v2 naming). Amazons own docs explain that you still need to use the old v1 scheme for capitalized names, as well as names containing certain special characters. It’d be a shame if they just tossed all those old buckets in the trash by leaving them inaccessible. All in, this seems like another silly, unnecessary, depreciation of an API that was working perfectly well. A trend I’m noticing more often these days. Shame.
- Roark66 7y agoAWS API is an inconsistent mess. If you don't believe me try writing a script to tag resources. Every resource type requires using different way to identify it, different way to pass the tags etc. You're pretty much required to write different code to handle each resource type.
- cs02rm0 7y agoI didn't know path style was possible. I'd have found it really useful. :-/
- rynop 7y agoHow does this impact CloudFront origin domain names? I have an s3 bucket as a CF origin and the format the AWS CF Console auto-completes to is: <bucket>.s3.amazonaws.com Do I need to change my origin to be, Origin domain name: s3.amazonaws.com, Origin Path: <bucket> This is a sneaky one that will bite lots of folks as it is NOT clear.
- bagels 7y agoI think you have this backwards. <bucket>.s3.amazonaws.com is the V2 url formula.
- gigatexal 7y agoHmm I don’t understand why this change is happening. What does this gain? Removal of tech debt?
- tckr 7y agoThe title is misleading. Path style request "/foo/bar/file.ext" are still supported. What changes is that the bucket name must be in the hostname.
- k__ 7y agoPath style can be used in hostnames?
- orf 7y agohttps://github.com/search?q=%22https%3A%2F%2Fs3.amazonaws.com%2F%22&type=Code https://github.com/search?q=%22https%3A%2F%2Fs3.amazonaws.co... Over a million results (+250k http). This is going to be painful.
- jvarsanik 7y agoDoes anyone know if this will affect uploads? We are getting an upload URL using s3.createPresignedPost and this returns (at least currently) a path-style url...
- abra559 7y agothis looks to be largely resolved: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...