13 ms·

Amazon S3 Path Deprecation Plan – The Rest of the Story

604 points by jeffbarr 7y ago

DVassallo 7y ago
Thank you for listening! The original plan was insane. The new one is sane. As I pointed out here https://twitter.com/dvassallo/status/1125549694778691584 https://twitter.com/dvassallo/status/1125549694778691584 thousands of printed books had references to V1 S3 URLs. Breaking them would have been a huge loss. Thank you!
- staticassertion 7y ago
  There's a book out there that points to my website. I wish they had just mirrored it. My website has been dead for a few months, and some O'Reilly book has a dead link.
  - gumby 7y ago
    I just now realised that if the web required two-way links there would be no way to put them into books! Some context: Back when the www was first released the most common criticism was the lack of back links. This is such a stupid and obvious deficiency that it really wasn't worth even looking into a system that was an obvious stillbirth. It wasn't just the "experts" saying this, but so many of them did because it was just so dumb. So you've probably never heard of this "world wide web" thing -- not only were there only one-way links, but it had its own homegrown markup language dialect and instead of using an ordinary protocol like FTP or even gopher it pointlessly used its own http protocol. (Also back then there was this research protocol called TCP/IP, which was another waste of time given that the OSI protocol stack was poised to dominate the networks just as soon as a working one was written. I wonder what the modern equivalents are).
    - omeid2 7y ago
      It is a shame that OSI didn't happen though and instead we have an ad-hoc mess of layered cake without clear boundaries as demonstrated by OSI Model.
      - greglindahl 7y ago
        If you want low performance, that’s one way to get it.
      - gumby 7y ago
        I don't agree: the OSI protocols were the classic camel-is-a-horse-designed-by-committee: heavyweight and looked like a pain to use. Looked like, as I never saw a working stack. The IETF/RFC/Working Code/Interop(RIP) approach has given V4 incredibly long legs. At least the OSI model itself kinda survived.
        rjsw 7y ago
        ISODE [1] worked, and it was usable before there were commercial ISPs offering TCP/IP services. [1] https://en.wikipedia.org/wiki/ISO_Development_Environment https://en.wikipedia.org/wiki/ISO_Development_Environment
    - DonHopkins 7y ago
      Pfff, TCP/IP will never succeed. It doesn't have enough layers! /s https://archive.org/details/elementsofnetwor00padl https://archive.org/details/elementsofnetwor00padl "The Book": The Elements of Networking Style: And Other Essays & Animadversions of the Art of Intercomputer Networking, by M. A. Padlipsky (1985) The World's Only Know Constructively Snotty Computer Science Book: historically, its polemics for TCP/IP and against the international standardsmongers' "OSI" helped the Internet happen; currently, its principles of technoaesthetic criticism are still eminently applicable to the States of most (probably all) technical Arts-all this and Cover Cartoons, too but it's not for those who can't deal with real sentences. Standards: Threat or Menace, p. 193 A final preliminary: Because ISORM is more widely touted than TCP/IP, and hence the clearer present danger, it seems only fair that it should be the target of the nastier of the questions. This is in the spirit of our title, for in my humble but dogmatic opinion even a good proposed Standard is a prima facie threat to further advance in the state of the art, but a sufficiently flawed standard is a menace even to maintaining the art in its present state, so if the ISORM school is wrong and isn't exposed the consequences could be extremely unfortunate. At least, the threat / menace paradigm applies, I submit in all seriousness, to protocol standards; that is, I wouldn't think of being gratuitously snotty to the developers of physical standards -- I like to be able to use the same cap to reclose sodapop bottles and beer bottles (though I suspect somebody as it were screwed up when it came to those damn "twist off" caps) -- but I find it difficult to be civil to advocates of "final," "ultimate" standards when they're dealing with logical constructs rather than physical ones. After all, as I understand it, a fundamental property of the stored program computer is its ability to be reprogrammed. Yes, I understand that to do so costs money and yes, I've heard of ROM, and no I'm not saying that I insist on some idealistic notion of optimality, but definitely I don't think it makes much sense to keep trudging to an outhouse if I can get indoor plumbing . . . even if the moon in the door is exactly like the one in my neighbor's. Appendix 3, The Self-Framed Slogans Suitable for Mounting https://donhopkins.com/home/Layers.png https://donhopkins.com/home/Layers.png IF YOU KNOW WHAT YOU'RE DOING, THREE LAYERS IS ENOUGH; IF YOU DON'T, EVEN SEVENTEEN LEVELS WON'T HELP https://en.wikipedia.org/wiki/Michael_A._Padlipsky https://en.wikipedia.org/wiki/Michael_A._Padlipsky On the occasion of The Book's reissuance, Peter Salus wrote a review in Cisco's Internet Protocol Journal which included the following observations: Padlipsky brought together several strands that managed to result in the perfect chord for me over 15 years ago. I reread this slim volume (made up of a Foreword, 11 chapters (each a separate arrow from Padlipsky's quiver) and three appendixes (made up of half a dozen darts of various lengths and a sheaf of cartoons and slogans) several months ago, and have concluded that it is as acerbic and as important now as it was 15 years ago. [Emphasis added] The instruments Padlipsky employs are a sharp wit (and a deep admiration for François Marie Arouet), a sincere detestation for the ISO Reference Model, a deep knowledge of the Advanced Research Projects Agency Network (ARPANET)/Internet, and wide reading in classic science fiction. In a lighter vein, The Book has been called "... beyond doubt the funniest technical book ever written."
    - scoot 7y ago
      Your timeline’s all wrong. TCP/IP the Internet built on it were already well established by the time the web was born. There was nothing particularly special about HTTP or HTML, or even the concept of the web. What made it a success was the availability of a server reference architecture, and, more importantly, a browser. It was easy to try it out, see the value, and get up and running with your own server if you had something to publish. Discoverability was a problem in the early days. There were printed catalogs of wesites! Backlinks might have helped, but clearly were not a fundamental requirement for the web’s success.
      - gumby 7y ago
        The fact the tcp (and it’s own institutional infrastructure) were already established is what made the whole OSI network effort even more enjoyably absurd. It was the last gasp of Big IT trying to take over the crazies. Most amusingly to me, it seemed only to be discussed in enterprise contexts and Very Important IT Journals. Such people were officially committed to deployment, while their own people were busy getting stuff done. IIRC the first nail in the coffin was the US military ignoring the naked emperor and officially deciding to stick with TCP. But by that time most people with real work to be done had ignore the whole OSI effort. > Discoverability was a problem in the early days. Yeah, I remember being at a conference in which a smart person (actually a smart person, no snark) said that discoverability would be over, as indexing the web would require keeping a copy of everything which, of course, is completely impossible. And we all nodded, because indeed, that did make sense. And about six months later, when altavista launched, it seemed only to confirm this belief.
        no_identd 7y ago
        You both get so much of this story so utterly… /not even/ __quite__ wrong, but more importantly, leave so much detail out that, if I didn't presume better(which I do! Would seem rather paranoid if I didn't.), I'd suspect lying by omission. All of this, which includes the story to follow, makes me—and I don't say this for exaggeration purposes, it really does have an emotional impact—very sad, although it doesn't surprise me, barely anyone realizes the true technological horror lurking deep in the history of the 'Internet'. Please, consider looking A BIT more at the history of TCP/IP: http://rina.tssg.org/docs/DublinLostLayer140109.pdf http://rina.tssg.org/docs/DublinLostLayer140109.pdf (Slides!) http://rina.tssg.org/docs/How_in_the_Heck_do_you_lose_a_layer.pdf http://rina.tssg.org/docs/How_in_the_Heck_do_you_lose_a_laye... Day, John - How in the Heck Do You Lose a Layer!? (2012) Abstract: "Recently, Alex McKenzie published an anecdote in the IEEE Annals of the History of Computing on the creation of INWG 96, a proposal by IFIP WG6.1 for an international transport protocol. McKenzie concentrates on the differences between the proposals that lead to INWG 96. However, it is the similarities that are much more interesting. This has lead to some rather surprising insights into not only the subsequent course of events, but also the origins of many current problems, and where the solutions must be found. The results are more than a little surprising." And here, a rather lengthy excerpt from later in the paper, as I suspect a lot of people might presume that the paper would go for some points it definitely DOESN'T go for: "[…] This does not mean that we should be doing OSI. Good grief, no. This only implies that the data OSI had to work with brought them to the same structure INWG had come to.[10] OSI would have brought along a different can of worms. OSI was the state of understanding in the early 80s. We have learned a lot more since.[11] There was much unnecessary complexity in OSI and recent insights allow considerable simplification over even current practice. OSI also split the addresses from the error and flow control protocol. This creates other problems. But the Internet’s course is definitely curious. Everyone else came up with an internet architecture for an internet, except them. These were the people who were continually stressing that they were building an Internet. Even more ironic is that the Internet ceased to be an Internet on the day most people would mark as the birth of the Internet, i.e. on the flag day January 1, 1983 when NCP was turned off and it became one large network. It was well understood at the time that two levels of addressing were required. This had been realized in the ARPANET when the first host with redundant network connections was deployed in 1972. The INWG structure provided the perfect solution. It is clear why the Internet kept the name, but less clear why they dropped the Network Layer. Or perhaps more precisely, why they renamed the Network Layer, the Internet Layer. Did they think, just calling it something different made it different? […] ———— […] [10] There was little or no overlap between SC6/WG2 and INWG. [11] And if the politics had not been so intense and research had continued to develop better understanding, we would have learned it a bit sooner. […] [22] Someone will ask, What about IPv6? It does nothing for these problems but make them worse and the problem it does solve is not a problem. […]" http://csr.bu.edu/rina/KoreaNamingFund100218.pdf http://csr.bu.edu/rina/KoreaNamingFund100218.pdf more slides! And much more, here: http://rina.tssg.org/ http://rina.tssg.org/ (I find it rather very strange the RINA folks and the GNUnet folks seem to each pull their own thing instead of working together, it very much seems like a—hopefully NOT inevitable—repeat of the very thing John Day describes in the slides & articles above…) —— Addendum #1: See also, for a network security perspective: http://www.toad.com/gnu/netcrypt.html http://www.toad.com/gnu/netcrypt.html http://bitsavers.informatik.uni-stuttgart.de/pdf/bbn/imp/BBN1822_Jan1976.pdf http://bitsavers.informatik.uni-stuttgart.de/pdf/bbn/imp/BBN... see also Appendix H here, starting on PDF page 180 ——— Addendum #2: Something in the back of my mind & the depths of my guts tells me I should link the following here, albeit I remain completely clueless as to why, or how it could seem relevant to—& topical for—any of the above, so, I'll just drop it here without explanation: https://en.wikipedia.org/wiki/Managed_Trusted_Internet_Protocol_Service https://en.wikipedia.org/wiki/Managed_Trusted_Internet_Proto... (Interesting standards compliance section there, by the way.)
  - maaaats 7y ago
    A book (on Libgdx) uses one of my repos as a starting point, telling the readers to clone my repo and do certain changes. I've left the repo alone, bugs and all, as I think it's cool that people uses my code. But the authors never reached out or anything, I only discovered it by chance. I could easily by accident have invalidated their whole chapter.
- inopinatus 7y ago
  If we're talking textbooks, well then. This is a textbook case for the 301 HTTP response code.
  - jasonkester 7y ago
    The old REST-style S3 URLs are specifically excluded from being able to redirect: https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-... You can create a new bucket or switch your existing one to "Static Website Hosting" mode to enable the ability to 301 your content going forward. But the URL for the "website" version isn't the same as the REST URL. And again, there's no way to redirect from the old naming scheme to the new one. If you have content that you've ever linked with one of those URLs, it's stuck there forever.
    - codetrotter 7y ago
      > And again, there's no way to redirect from the old naming scheme to the new one. For customers, no. For Amazon itself, yes. And I think that is what parent commenter meant. That Amazon should 301 all requests that are using old paths.
      - deleted 7y ago
        [deleted]
      - xchaotic 7y ago
        It's not that simple, unfortunately - it won't work for the old dotted addresses and S3 is not HTTP
    - inopinatus 7y ago
      This isn't a restriction if you're AWS and looking to give more customers a soft landing over an extended deprecation timeframe. The certificate concern some are raising is also a furphy.
  - masklinn 7y ago
    Except for all the dotted bucket names which can't be redirected because the result will always trigger a certificate error.
- cm2187 7y ago
  Stupid question but can’t amazon simply do a redirection?
  - masklinn 7y ago
    It'd break working paths, because bucket names can contain dots which work fine in "old style" paths but breaks using vhost style (a cert can only have a single wildcard, and the wildcard does not match dots, so ".s3.amazonraws.com" will not match "foo.bar.s3.amazonraws.com", and it's not possible to create a ".*.s3.amazonraws.com" cert).
    - cm2187 7y ago
      But if it is an auto-redirect, can’t aws redirect to an alias of the bucket that replaces dots by another character?
      - masklinn 7y ago
        How do you handle conflicts when an other bucket name legitimately uses whichever replacement character you picked? Historically bucket names are supersets of DNS names. In fact, early 2018 Amazon modified the bucket naming rules in one of the older regions as the names that region allowed were completely inaccessible in vhost style (not just cert mismatch, the names were literally not expressible): > The legacy rules for bucket names in the US East (N. Virginia) Region allowed bucket names to be as long as 255 characters, and bucket names could contain any combination of uppercase letters, lowercase letters, numbers, periods (.), hyphens (-), and underscores (_).
        cm2187 7y ago
        Or use an md5 of the bucket name with a prefix and make bucket names with that prefix followed by a bunch of hex chars illegal bucket names. It doesn’t need to be pretty, it’s only an alias that will never be typed or viewed by a human. [edit]: I assume that new bucket names that break subdomains have been made illegal now so they work with a finite and static list of names. I am sure they can come up with a substitution that doesn't create collisions and with a reserved prefix you can avoid future collisions.
- ethagnawl 7y ago
  This announcement comes as a relief. I was already drafting the email I'd need to send to current/former clients, letting them know that they'd have to hire me (or someone else) to write/run a migration to update asset paths hardcoded in static HTML pages or risk broken assets going forward. IIRC, an older version of the Froala WYSIWYG editor didn't support uploads using "nested" object paths (e.g. bucket/post-1/photo.png) and VH paths, which is why I leaned on the path-style feature for a few projects. So, not only would the fix (for the projects using Froala) involve migrating the S3 objects, I'd also have to change application code and/or upgrade the Froala WYSIWYG editor. Hindsight is 20/20, of course, but I'm (unpleasantly) surprised they didn't take the thousands of ways a hard cut-over would break the web before drafting and (softly) announcing the initial plan.
raiyu 7y ago
It's nice to see that instead of deprecation support for the old paths will continue for all buckets created on or before the cut-off date of Sept 30, 2020. So if you don't want to change, you can continue using the old paths. Just might limit access to some new features coming later that are dependent on the virtual host sub domains.
tanilama 7y ago
Grandfathering is a good idea. GJ AWS.
- andybak 7y ago
  "GJ"?
  - rekshaw 7y ago
    Good job
deleted 7y ago
[deleted]
luhn 7y ago
> Bucket Names with Dots – It is important to note that bucket names with “.” characters are perfectly valid for website hosting and other use cases. However, there are some known issues with TLS and with SSL certificates. We are hard at work on a plan to support virtual-host requests to these buckets, and will share the details well ahead of September 30, 2020. I’m mystified how they’re planning on doing this. Anybody care to speculate?
- regecks 7y ago
  They're already a CA, could they reasonably just issue a certificate for every bucket? I have no idea how many buckets there are in total. ~They probably couldn't take the Cloudflare approach of jamming 100 customer domains onto each certificate, since that would leak bucket names too easily.~
  - mmastrac 7y ago
    Isn't it technically valid to issue a cert with (star).domain.com, (star).(star).domain.com, (star).(star).(star).domain.com, etc...?
    - judge2020 7y ago
      You can technically create them, but IIRC browsers don't trust them.
      - oasisbob 7y ago
        And creating/signing them is a violation of the CAB Forum Baseline requirements.
    - koolba 7y ago
      No you can only have a single wildcard per domain listed in a cert.
      - deathanatos 7y ago
chipperyman573 7y ago
Still doesn't help with domain censorship. This was discussed in-depth in the other thread from yesterday, but TLDR, it's a lot harder to block https://s3.amazonaws.com/tiananmen-square-facts https://s3.amazonaws.com/tiananmen-square-facts than https://tiananmen-square-facts.s3.amazonaws.com https://tiananmen-square-facts.s3.amazonaws.com because DNS lookups are made before HTTPS kicks in.
- erikpukinskis 7y ago
  What could Amazon do that would "help"?
  - chipperyman573 7y ago
    Still let people use the old path system, even on new servers. This is an often-used trick to get around government censorship that will be destroyed with this change.
    - est 7y ago
      Why S3 must carry the burden of getting around censorship? If you are talking about China, yeah, Google used to carry that burden. Now GAE, GCloud, Youtube, Gmail are all gone. The whole IP range was blacklisted. Now what? Just because something accidentally works does not mean it will last forever.
      - dserodio 7y ago
        They don't "must", but it would be very nice of them.
      - realusername 7y ago
        China isn't the only repressive country in the world, there's plenty of people using domain fronting right now in other parts of the world.
    - hn_throwaway_99 7y ago
ryanbigg 7y ago
This is a great step forward. Particularly changing the rules a little so that old buckets won’t break after a certain date. Thank you for taking the time to write this up Jeff.
- jeffbarr 7y ago
  You are welcome. And now, back to my stay-cation.
gundmc 7y ago
Props to Amazon for listening to feedback and altering course.
whoevercares 7y ago
S3 comes from the vision of Bezos?! Wow
valgaze 7y ago
Malloc for the internet: "We launched S3 in early 2006. Jeff Bezos’ original spec for S3 was very succinct – he wanted malloc (a key memory allocation function for C programs) for the Internet. From that starting point, S3 has grown to the point where it now stores many trillions of objects and processes millions of requests per second for them. Over the intervening 13 years, we have added many new storage options, features, and security controls to S3."
kenhwang 7y ago
For anyone still confused to why AWS dominates the cloud market, it's because they're willing to grandfather features with a reasonable sunset horizon.
- rsanek 7y ago
  I'd say the momentum from having no competition in the space for nearly a decade is far more impactful.
  - phh 7y ago
    I'd say it's a combination of both. Not having competition means they got a lot of users, but having long-term feature support means people didn't run away.
blaisio 7y ago
This is interesting for a few reasons. IMHO, the original deprecation plan was reasonable. Not generous, but reasonable. Especially compared to what other cloud providers (eg. Google Cloud) have done. It did seem like a diversion from their normal practice of obsessively supporting old stuff for as long as possible, but it really wasn't too bad. Responding to feedback, publicly, and explaining what they were trying to do and why they needed to do it, is incredibly refreshing. This seems like a big PR win for AWS. I'm left trusting and liking them more, not less.
- DVassallo 7y ago
  How was the original plan “reasonable”? S3’s FAQ talks about durability in terms of tens of thousands of years https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/. To honor that claim, AWS has the burden to support v1 URLs until the end of the internet.
  - whoisjuan 7y ago
    Storage durability has nothing to do with this. Changing how you access data saved in storage is a reasonable change to keep up with the evolution of networking technologies. The change here was deprecating an access pattern, not destroying data or anything remotely similar.
    - mantap 7y ago
      The distinction is academic if you don't have the ability to change the URL.
    - DVassallo 7y ago
      Tell that to the author of “The Peace Corps and Latin America”, who used S3 v1 URLs dozens of times in the book, with the assumption that they’d be available forever: https://books.google.com/books?id=Q312DwAAQBAJ&pg=PA135&dq=https://s3.amazonaws.com&hl=en&sa=X&ved=0ahUKEwiA2tek3o3iAhWYHjQIHVOjDSwQ6AEIKjAB#v=onepage&q=https%3A%2F%2Fs3.amazonaws.com&f=false https://books.google.com/books?id=Q312DwAAQBAJ&pg=PA135&dq=h... And thousands of other books like that.
el_benhameen 7y ago
Kind of tangential, but is Bezos a programmer type? I thought he came from banking or the big 4. I’m curious if the “malloc for the internet” bit is verbatim.
- hbosch 7y ago
  I don't think Jeff is a programmer, but I think he is very smart and his success shows that he can understand and create businesses around lots of different types of concepts and ideas. I imagine in reality there is some amount of collaboration that happens before Bezos spouts off something like "create a malloc for the internet", but in any case, it's very strong for his brand of leadership that the lore states it came straight from him.
  - fgonzag 7y ago
    1.) He graduated from CS at Princeton. 2.) Only an actual programmer would know what malloc is and how it works (he knows what it is exactly since he has the idea "malloc for the web")
    - hbosch 7y ago
      Point taken, I didn’t know Bezos had an engineering background! Amazing.
  - eclipxe 7y ago
    He's literally an EE/CS Major. He's a programmer at heart. His technical depth is key to a lot of Amazon's biggest successes.
    - hbosch 7y ago
      Wow! News to me. Thanks for letting me know.
- tybit 7y ago
  Great quite. I had thought the same about his background, from https://en.wikipedia.org/wiki/Jeff_Bezos https://en.wikipedia.org/wiki/Jeff_Bezos looks like he was technical at college and the first couple of years (or less) of his career. > He graduated from Princeton University in 1986 with degrees in electrical engineering and computer science. and > After Bezos graduated from Princeton in 1986,... He first worked at Fitel, a fintech telecommunications start-up, where he was tasked with building a network for international trade.[28] Bezos was promoted to head of development and director of customer service thereafter.[29] He transitioned into the banking industry when he became a product manager at Bankers Trust; he worked there from 1988 ...
gigatexal 7y ago
Yeah pushing the new way is fine but not removing the logic to resolve the old way is better.
- jasonhansel 7y ago
  Agreed! I think Amazon definitely made the right call here.
nullecksor 7y ago
This was a DOA when I first read it. S3 or AWS wouldn't break a single customer before changing anything.
ZiiS 7y ago
It seems to me that adding a 301 redirect from the old URL to the new would not unresonably stress the resources of AWS? It seems perfectly resonable to update the library access, but breaking old URLs seems unessesary. They could even add a second of latency to incentivise people who can update their links.
- joemag 7y ago
  Some HTTP clients don’t support redirects, or at least require an explicit configuration to enable them. So this would still be a breaking change for some applications.
ahmhn 7y ago
Does this change affect S3 access via the various AWS SDKs, or just the format of URLs?
faithteams 7y ago
Faith Teams is easy-to-use and affordable church management & database software with online giving, child check in, volunteer management, events, mailings, email campaigns, mobile blast & more. 14 days free trial. No credit card required. Sign up now!! https://faithteams.com/ https://faithteams.com/
faithteams 7y ago
Faith Teams is easy-to-use and affordable church management & database software with online giving, child check in, volunteer management, events, mailings, email campaigns, mobile blast & more. 14 days free trial. No credit card required. Sign up now at: https://faithteams.com/ https://faithteams.com/
nik736 7y ago
"In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /jeffbarr-public/classic_amazon_door_desk.png are object keys." I think this should be: "In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /classic_amazon_door_desk.png are object keys."
- jeffbarr 7y ago
  You are correct; thanks for spotting this! All fixed.
freeasindave 7y ago
Pre-signed urls still come back from the S3 SDK as a V1 path style. I'm assuming this either changes at some point, or that will continue to work?
parliament32 7y ago
I still don't get why there was such an uproar about this: Amazon should just issue a "301 Moved Permanently" and be done with it. If your app for some arcane reason doesn't understand an HTTP status code that's been around for 20 years... your code is bad and you should feel bad.
bilater 7y ago
Okay probably a dumb question but why can't they just have an automatic redirect from the path style to the virtual hosted ones under the hood? People get both options up front while they can work with the one they like.

604 points by jeffbarr 7y ago

Thank you for listening! The original plan was insane. The new one is sane. As I pointed out here https://twitter.com/dvassallo/status/1125549694778691584 https://twitter.com/dvassallo/status/1125549694778691584 thousands of printed books had references to V1 S3 URLs. Breaking them would have been a huge loss. Thank you!

staticassertion 7y ago
There's a book out there that points to my website. I wish they had just mirrored it. My website has been dead for a few months, and some O'Reilly book has a dead link.
- gumby 7y ago
  I just now realised that if the web required two-way links there would be no way to put them into books! Some context: Back when the www was first released the most common criticism was the lack of back links. This is such a stupid and obvious deficiency that it really wasn't worth even looking into a system that was an obvious stillbirth. It wasn't just the "experts" saying this, but so many of them did because it was just so dumb. So you've probably never heard of this "world wide web" thing -- not only were there only one-way links, but it had its own homegrown markup language dialect and instead of using an ordinary protocol like FTP or even gopher it pointlessly used its own http protocol. (Also back then there was this research protocol called TCP/IP, which was another waste of time given that the OSI protocol stack was poised to dominate the networks just as soon as a working one was written. I wonder what the modern equivalents are).
  - omeid2 7y ago
    It is a shame that OSI didn't happen though and instead we have an ad-hoc mess of layered cake without clear boundaries as demonstrated by OSI Model.
    - greglindahl 7y ago
      If you want low performance, that’s one way to get it.
    - gumby 7y ago
      I don't agree: the OSI protocols were the classic camel-is-a-horse-designed-by-committee: heavyweight and looked like a pain to use. Looked like, as I never saw a working stack. The IETF/RFC/Working Code/Interop(RIP) approach has given V4 incredibly long legs. At least the OSI model itself kinda survived.
      - rjsw 7y ago
        ISODE [1] worked, and it was usable before there were commercial ISPs offering TCP/IP services. [1] https://en.wikipedia.org/wiki/ISO_Development_Environment https://en.wikipedia.org/wiki/ISO_Development_Environment
  - DonHopkins 7y ago
    Pfff, TCP/IP will never succeed. It doesn't have enough layers! /s https://archive.org/details/elementsofnetwor00padl https://archive.org/details/elementsofnetwor00padl "The Book": The Elements of Networking Style: And Other Essays & Animadversions of the Art of Intercomputer Networking, by M. A. Padlipsky (1985) The World's Only Know Constructively Snotty Computer Science Book: historically, its polemics for TCP/IP and against the international standardsmongers' "OSI" helped the Internet happen; currently, its principles of technoaesthetic criticism are still eminently applicable to the States of most (probably all) technical Arts-all this and Cover Cartoons, too but it's not for those who can't deal with real sentences. Standards: Threat or Menace, p. 193 A final preliminary: Because ISORM is more widely touted than TCP/IP, and hence the clearer present danger, it seems only fair that it should be the target of the nastier of the questions. This is in the spirit of our title, for in my humble but dogmatic opinion even a good proposed Standard is a prima facie threat to further advance in the state of the art, but a sufficiently flawed standard is a menace even to maintaining the art in its present state, so if the ISORM school is wrong and isn't exposed the consequences could be extremely unfortunate. At least, the threat / menace paradigm applies, I submit in all seriousness, to protocol standards; that is, I wouldn't think of being gratuitously snotty to the developers of physical standards -- I like to be able to use the same cap to reclose sodapop bottles and beer bottles (though I suspect somebody as it were screwed up when it came to those damn "twist off" caps) -- but I find it difficult to be civil to advocates of "final," "ultimate" standards when they're dealing with logical constructs rather than physical ones. After all, as I understand it, a fundamental property of the stored program computer is its ability to be reprogrammed. Yes, I understand that to do so costs money and yes, I've heard of ROM, and no I'm not saying that I insist on some idealistic notion of optimality, but definitely I don't think it makes much sense to keep trudging to an outhouse if I can get indoor plumbing . . . even if the moon in the door is exactly like the one in my neighbor's. Appendix 3, The Self-Framed Slogans Suitable for Mounting https://donhopkins.com/home/Layers.png https://donhopkins.com/home/Layers.png IF YOU KNOW WHAT YOU'RE DOING, THREE LAYERS IS ENOUGH; IF YOU DON'T, EVEN SEVENTEEN LEVELS WON'T HELP https://en.wikipedia.org/wiki/Michael_A._Padlipsky https://en.wikipedia.org/wiki/Michael_A._Padlipsky On the occasion of The Book's reissuance, Peter Salus wrote a review in Cisco's Internet Protocol Journal which included the following observations: Padlipsky brought together several strands that managed to result in the perfect chord for me over 15 years ago. I reread this slim volume (made up of a Foreword, 11 chapters (each a separate arrow from Padlipsky's quiver) and three appendixes (made up of half a dozen darts of various lengths and a sheaf of cartoons and slogans) several months ago, and have concluded that it is as acerbic and as important now as it was 15 years ago. [Emphasis added] The instruments Padlipsky employs are a sharp wit (and a deep admiration for François Marie Arouet), a sincere detestation for the ISO Reference Model, a deep knowledge of the Advanced Research Projects Agency Network (ARPANET)/Internet, and wide reading in classic science fiction. In a lighter vein, The Book has been called "... beyond doubt the funniest technical book ever written."
  - scoot 7y ago
    Your timeline’s all wrong. TCP/IP the Internet built on it were already well established by the time the web was born. There was nothing particularly special about HTTP or HTML, or even the concept of the web. What made it a success was the availability of a server reference architecture, and, more importantly, a browser. It was easy to try it out, see the value, and get up and running with your own server if you had something to publish. Discoverability was a problem in the early days. There were printed catalogs of wesites! Backlinks might have helped, but clearly were not a fundamental requirement for the web’s success.
    - gumby 7y ago
      The fact the tcp (and it’s own institutional infrastructure) were already established is what made the whole OSI network effort even more enjoyably absurd. It was the last gasp of Big IT trying to take over the crazies. Most amusingly to me, it seemed only to be discussed in enterprise contexts and Very Important IT Journals. Such people were officially committed to deployment, while their own people were busy getting stuff done. IIRC the first nail in the coffin was the US military ignoring the naked emperor and officially deciding to stick with TCP. But by that time most people with real work to be done had ignore the whole OSI effort. > Discoverability was a problem in the early days. Yeah, I remember being at a conference in which a smart person (actually a smart person, no snark) said that discoverability would be over, as indexing the web would require keeping a copy of everything which, of course, is completely impossible. And we all nodded, because indeed, that did make sense. And about six months later, when altavista launched, it seemed only to confirm this belief.
      - no_identd 7y ago
        You both get so much of this story so utterly… /not even/ __quite__ wrong, but more importantly, leave so much detail out that, if I didn't presume better(which I do! Would seem rather paranoid if I didn't.), I'd suspect lying by omission. All of this, which includes the story to follow, makes me—and I don't say this for exaggeration purposes, it really does have an emotional impact—very sad, although it doesn't surprise me, barely anyone realizes the true technological horror lurking deep in the history of the 'Internet'. Please, consider looking A BIT more at the history of TCP/IP: http://rina.tssg.org/docs/DublinLostLayer140109.pdf http://rina.tssg.org/docs/DublinLostLayer140109.pdf (Slides!) http://rina.tssg.org/docs/How_in_the_Heck_do_you_lose_a_layer.pdf http://rina.tssg.org/docs/How_in_the_Heck_do_you_lose_a_laye... Day, John - How in the Heck Do You Lose a Layer!? (2012) Abstract: "Recently, Alex McKenzie published an anecdote in the IEEE Annals of the History of Computing on the creation of INWG 96, a proposal by IFIP WG6.1 for an international transport protocol. McKenzie concentrates on the differences between the proposals that lead to INWG 96. However, it is the similarities that are much more interesting. This has lead to some rather surprising insights into not only the subsequent course of events, but also the origins of many current problems, and where the solutions must be found. The results are more than a little surprising." And here, a rather lengthy excerpt from later in the paper, as I suspect a lot of people might presume that the paper would go for some points it definitely DOESN'T go for: "[…] This does not mean that we should be doing OSI. Good grief, no. This only implies that the data OSI had to work with brought them to the same structure INWG had come to.[10] OSI would have brought along a different can of worms. OSI was the state of understanding in the early 80s. We have learned a lot more since.[11] There was much unnecessary complexity in OSI and recent insights allow considerable simplification over even current practice. OSI also split the addresses from the error and flow control protocol. This creates other problems. But the Internet’s course is definitely curious. Everyone else came up with an internet architecture for an internet, except them. These were the people who were continually stressing that they were building an Internet. Even more ironic is that the Internet ceased to be an Internet on the day most people would mark as the birth of the Internet, i.e. on the flag day January 1, 1983 when NCP was turned off and it became one large network. It was well understood at the time that two levels of addressing were required. This had been realized in the ARPANET when the first host with redundant network connections was deployed in 1972. The INWG structure provided the perfect solution. It is clear why the Internet kept the name, but less clear why they dropped the Network Layer. Or perhaps more precisely, why they renamed the Network Layer, the Internet Layer. Did they think, just calling it something different made it different? […] ———— […] [10] There was little or no overlap between SC6/WG2 and INWG. [11] And if the politics had not been so intense and research had continued to develop better understanding, we would have learned it a bit sooner. […] [22] Someone will ask, What about IPv6? It does nothing for these problems but make them worse and the problem it does solve is not a problem. […]" http://csr.bu.edu/rina/KoreaNamingFund100218.pdf http://csr.bu.edu/rina/KoreaNamingFund100218.pdf more slides! And much more, here: http://rina.tssg.org/ http://rina.tssg.org/ (I find it rather very strange the RINA folks and the GNUnet folks seem to each pull their own thing instead of working together, it very much seems like a—hopefully NOT inevitable—repeat of the very thing John Day describes in the slides & articles above…) —— Addendum #1: See also, for a network security perspective: http://www.toad.com/gnu/netcrypt.html http://www.toad.com/gnu/netcrypt.html http://bitsavers.informatik.uni-stuttgart.de/pdf/bbn/imp/BBN1822_Jan1976.pdf http://bitsavers.informatik.uni-stuttgart.de/pdf/bbn/imp/BBN... see also Appendix H here, starting on PDF page 180 ——— Addendum #2: Something in the back of my mind & the depths of my guts tells me I should link the following here, albeit I remain completely clueless as to why, or how it could seem relevant to—& topical for—any of the above, so, I'll just drop it here without explanation: https://en.wikipedia.org/wiki/Managed_Trusted_Internet_Protocol_Service https://en.wikipedia.org/wiki/Managed_Trusted_Internet_Proto... (Interesting standards compliance section there, by the way.)
- maaaats 7y ago
  A book (on Libgdx) uses one of my repos as a starting point, telling the readers to clone my repo and do certain changes. I've left the repo alone, bugs and all, as I think it's cool that people uses my code. But the authors never reached out or anything, I only discovered it by chance. I could easily by accident have invalidated their whole chapter.
inopinatus 7y ago
If we're talking textbooks, well then. This is a textbook case for the 301 HTTP response code.
- jasonkester 7y ago
  The old REST-style S3 URLs are specifically excluded from being able to redirect: https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-... You can create a new bucket or switch your existing one to "Static Website Hosting" mode to enable the ability to 301 your content going forward. But the URL for the "website" version isn't the same as the REST URL. And again, there's no way to redirect from the old naming scheme to the new one. If you have content that you've ever linked with one of those URLs, it's stuck there forever.
  - codetrotter 7y ago
    > And again, there's no way to redirect from the old naming scheme to the new one. For customers, no. For Amazon itself, yes. And I think that is what parent commenter meant. That Amazon should 301 all requests that are using old paths.
    - deleted 7y ago
      [deleted]
    - xchaotic 7y ago
      It's not that simple, unfortunately - it won't work for the old dotted addresses and S3 is not HTTP
  - inopinatus 7y ago
    This isn't a restriction if you're AWS and looking to give more customers a soft landing over an extended deprecation timeframe. The certificate concern some are raising is also a furphy.
- masklinn 7y ago
  Except for all the dotted bucket names which can't be redirected because the result will always trigger a certificate error.
cm2187 7y ago
Stupid question but can’t amazon simply do a redirection?
- masklinn 7y ago
  It'd break working paths, because bucket names can contain dots which work fine in "old style" paths but breaks using vhost style (a cert can only have a single wildcard, and the wildcard does not match dots, so ".s3.amazonraws.com" will not match "foo.bar.s3.amazonraws.com", and it's not possible to create a ".*.s3.amazonraws.com" cert).
  - cm2187 7y ago
    But if it is an auto-redirect, can’t aws redirect to an alias of the bucket that replaces dots by another character?
    - masklinn 7y ago
      How do you handle conflicts when an other bucket name legitimately uses whichever replacement character you picked? Historically bucket names are supersets of DNS names. In fact, early 2018 Amazon modified the bucket naming rules in one of the older regions as the names that region allowed were completely inaccessible in vhost style (not just cert mismatch, the names were literally not expressible): > The legacy rules for bucket names in the US East (N. Virginia) Region allowed bucket names to be as long as 255 characters, and bucket names could contain any combination of uppercase letters, lowercase letters, numbers, periods (.), hyphens (-), and underscores (_).
      - cm2187 7y ago
        Or use an md5 of the bucket name with a prefix and make bucket names with that prefix followed by a bunch of hex chars illegal bucket names. It doesn’t need to be pretty, it’s only an alias that will never be typed or viewed by a human. [edit]: I assume that new bucket names that break subdomains have been made illegal now so they work with a finite and static list of names. I am sure they can come up with a substitution that doesn't create collisions and with a reserved prefix you can avoid future collisions.
ethagnawl 7y ago
This announcement comes as a relief. I was already drafting the email I'd need to send to current/former clients, letting them know that they'd have to hire me (or someone else) to write/run a migration to update asset paths hardcoded in static HTML pages or risk broken assets going forward. IIRC, an older version of the Froala WYSIWYG editor didn't support uploads using "nested" object paths (e.g. bucket/post-1/photo.png) and VH paths, which is why I leaned on the path-style feature for a few projects. So, not only would the fix (for the projects using Froala) involve migrating the S3 objects, I'd also have to change application code and/or upgrade the Froala WYSIWYG editor. Hindsight is 20/20, of course, but I'm (unpleasantly) surprised they didn't take the thousands of ways a hard cut-over would break the web before drafting and (softly) announcing the initial plan.

raiyu 7y ago

It's nice to see that instead of deprecation support for the old paths will continue for all buckets created on or before the cut-off date of Sept 30, 2020. So if you don't want to change, you can continue using the old paths. Just might limit access to some new features coming later that are dependent on the virtual host sub domains.

tanilama 7y ago

Grandfathering is a good idea. GJ AWS.

andybak 7y ago
"GJ"?
- rekshaw 7y ago
  Good job

deleted 7y ago

[deleted]

luhn 7y ago

> Bucket Names with Dots – It is important to note that bucket names with “.” characters are perfectly valid for website hosting and other use cases. However, there are some known issues with TLS and with SSL certificates. We are hard at work on a plan to support virtual-host requests to these buckets, and will share the details well ahead of September 30, 2020. I’m mystified how they’re planning on doing this. Anybody care to speculate?

regecks 7y ago
They're already a CA, could they reasonably just issue a certificate for every bucket? I have no idea how many buckets there are in total. ~They probably couldn't take the Cloudflare approach of jamming 100 customer domains onto each certificate, since that would leak bucket names too easily.~
- mmastrac 7y ago
  Isn't it technically valid to issue a cert with (star).domain.com, (star).(star).domain.com, (star).(star).(star).domain.com, etc...?
  - judge2020 7y ago
    You can technically create them, but IIRC browsers don't trust them.
    - oasisbob 7y ago
      And creating/signing them is a violation of the CAB Forum Baseline requirements.
  - koolba 7y ago
    No you can only have a single wildcard per domain listed in a cert.
    - deathanatos 7y ago

chipperyman573 7y ago

Still doesn't help with domain censorship. This was discussed in-depth in the other thread from yesterday, but TLDR, it's a lot harder to block https://s3.amazonaws.com/tiananmen-square-facts https://s3.amazonaws.com/tiananmen-square-facts than https://tiananmen-square-facts.s3.amazonaws.com https://tiananmen-square-facts.s3.amazonaws.com because DNS lookups are made before HTTPS kicks in.

erikpukinskis 7y ago
What could Amazon do that would "help"?
- chipperyman573 7y ago
  Still let people use the old path system, even on new servers. This is an often-used trick to get around government censorship that will be destroyed with this change.
  - est 7y ago
    Why S3 must carry the burden of getting around censorship? If you are talking about China, yeah, Google used to carry that burden. Now GAE, GCloud, Youtube, Gmail are all gone. The whole IP range was blacklisted. Now what? Just because something accidentally works does not mean it will last forever.
    - dserodio 7y ago
      They don't "must", but it would be very nice of them.
    - realusername 7y ago
      China isn't the only repressive country in the world, there's plenty of people using domain fronting right now in other parts of the world.
  - hn_throwaway_99 7y ago

ryanbigg 7y ago

This is a great step forward. Particularly changing the rules a little so that old buckets won’t break after a certain date. Thank you for taking the time to write this up Jeff.

jeffbarr 7y ago
You are welcome. And now, back to my stay-cation.

gundmc 7y ago

Props to Amazon for listening to feedback and altering course.

whoevercares 7y ago

S3 comes from the vision of Bezos?! Wow

valgaze 7y ago

Malloc for the internet: "We launched S3 in early 2006. Jeff Bezos’ original spec for S3 was very succinct – he wanted malloc (a key memory allocation function for C programs) for the Internet. From that starting point, S3 has grown to the point where it now stores many trillions of objects and processes millions of requests per second for them. Over the intervening 13 years, we have added many new storage options, features, and security controls to S3."

kenhwang 7y ago

For anyone still confused to why AWS dominates the cloud market, it's because they're willing to grandfather features with a reasonable sunset horizon.

rsanek 7y ago
I'd say the momentum from having no competition in the space for nearly a decade is far more impactful.
- phh 7y ago
  I'd say it's a combination of both. Not having competition means they got a lot of users, but having long-term feature support means people didn't run away.

blaisio 7y ago

This is interesting for a few reasons. IMHO, the original deprecation plan was reasonable. Not generous, but reasonable. Especially compared to what other cloud providers (eg. Google Cloud) have done. It did seem like a diversion from their normal practice of obsessively supporting old stuff for as long as possible, but it really wasn't too bad. Responding to feedback, publicly, and explaining what they were trying to do and why they needed to do it, is incredibly refreshing. This seems like a big PR win for AWS. I'm left trusting and liking them more, not less.

DVassallo 7y ago
How was the original plan “reasonable”? S3’s FAQ talks about durability in terms of tens of thousands of years https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/. To honor that claim, AWS has the burden to support v1 URLs until the end of the internet.
- whoisjuan 7y ago
  Storage durability has nothing to do with this. Changing how you access data saved in storage is a reasonable change to keep up with the evolution of networking technologies. The change here was deprecating an access pattern, not destroying data or anything remotely similar.
  - mantap 7y ago
    The distinction is academic if you don't have the ability to change the URL.
  - DVassallo 7y ago
    Tell that to the author of “The Peace Corps and Latin America”, who used S3 v1 URLs dozens of times in the book, with the assumption that they’d be available forever: https://books.google.com/books?id=Q312DwAAQBAJ&pg=PA135&dq=https://s3.amazonaws.com&hl=en&sa=X&ved=0ahUKEwiA2tek3o3iAhWYHjQIHVOjDSwQ6AEIKjAB#v=onepage&q=https%3A%2F%2Fs3.amazonaws.com&f=false https://books.google.com/books?id=Q312DwAAQBAJ&pg=PA135&dq=h... And thousands of other books like that.

el_benhameen 7y ago

Kind of tangential, but is Bezos a programmer type? I thought he came from banking or the big 4. I’m curious if the “malloc for the internet” bit is verbatim.

hbosch 7y ago
I don't think Jeff is a programmer, but I think he is very smart and his success shows that he can understand and create businesses around lots of different types of concepts and ideas. I imagine in reality there is some amount of collaboration that happens before Bezos spouts off something like "create a malloc for the internet", but in any case, it's very strong for his brand of leadership that the lore states it came straight from him.
- fgonzag 7y ago
  1.) He graduated from CS at Princeton. 2.) Only an actual programmer would know what malloc is and how it works (he knows what it is exactly since he has the idea "malloc for the web")
  - hbosch 7y ago
    Point taken, I didn’t know Bezos had an engineering background! Amazing.
- eclipxe 7y ago
  He's literally an EE/CS Major. He's a programmer at heart. His technical depth is key to a lot of Amazon's biggest successes.
  - hbosch 7y ago
    Wow! News to me. Thanks for letting me know.
tybit 7y ago
Great quite. I had thought the same about his background, from https://en.wikipedia.org/wiki/Jeff_Bezos https://en.wikipedia.org/wiki/Jeff_Bezos looks like he was technical at college and the first couple of years (or less) of his career. > He graduated from Princeton University in 1986 with degrees in electrical engineering and computer science. and > After Bezos graduated from Princeton in 1986,... He first worked at Fitel, a fintech telecommunications start-up, where he was tasked with building a network for international trade.[28] Bezos was promoted to head of development and director of customer service thereafter.[29] He transitioned into the banking industry when he became a product manager at Bankers Trust; he worked there from 1988 ...

gigatexal 7y ago

Yeah pushing the new way is fine but not removing the logic to resolve the old way is better.

jasonhansel 7y ago
Agreed! I think Amazon definitely made the right call here.

nullecksor 7y ago

This was a DOA when I first read it. S3 or AWS wouldn't break a single customer before changing anything.

ZiiS 7y ago

It seems to me that adding a 301 redirect from the old URL to the new would not unresonably stress the resources of AWS? It seems perfectly resonable to update the library access, but breaking old URLs seems unessesary. They could even add a second of latency to incentivise people who can update their links.

joemag 7y ago
Some HTTP clients don’t support redirects, or at least require an explicit configuration to enable them. So this would still be a breaking change for some applications.

ahmhn 7y ago

Does this change affect S3 access via the various AWS SDKs, or just the format of URLs?

faithteams 7y ago

Faith Teams is easy-to-use and affordable church management & database software with online giving, child check in, volunteer management, events, mailings, email campaigns, mobile blast & more. 14 days free trial. No credit card required. Sign up now!! https://faithteams.com/ https://faithteams.com/

faithteams 7y ago

nik736 7y ago

"In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /jeffbarr-public/classic_amazon_door_desk.png are object keys." I think this should be: "In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /classic_amazon_door_desk.png are object keys."

jeffbarr 7y ago
You are correct; thanks for spotting this! All fixed.

freeasindave 7y ago

Pre-signed urls still come back from the S3 SDK as a V1 path style. I'm assuming this either changes at some point, or that will continue to work?

parliament32 7y ago

I still don't get why there was such an uproar about this: Amazon should just issue a "301 Moved Permanently" and be done with it. If your app for some arcane reason doesn't understand an HTTP status code that's been around for 20 years... your code is bad and you should feel bad.

bilater 7y ago

Okay probably a dumb question but why can't they just have an automatic redirect from the path style to the virtual hosted ones under the hood? People get both options up front while they can work with the one they like.