4 ms·

Agree too. Just because you can get something to work doesn’t mean it’s supported. Using an Amazon S3 library on a non-Amazon service is “works but isn’t suppo

by tabony 2y ago

Agree too. Just because you can get something to work doesn’t mean it’s supported. Using an Amazon S3 library on a non-Amazon service is “works but isn’t supported.” Stick to only supported things if you want reliability.

genewitch 2y ago
a little less charitable is amazon is throwing its weight around to quash competition before they can get started; and shove tech debt onto third parties. I stopped giving amazon the benefit of the doubt about any aspect of their operations about 8 years ago.
- akerl_ 2y ago
  Did Amazon recommend that other 3rd party products use their SDK as their own client?
- jameslars 2y ago
  Less charitable or More cynical? How is Amazon supposed to track a 3rd party pulling their SDK and then reverse-engineering their own service side to work with the SDK? Assuming we're all okay with that premise to begin with, all sorts of other questions start popping up. Do these 3rd parties get veto power over a feature they can't support? Can they delay a launch if they need more time to make their reverse-engineered effort compatible again? It seems a hard to defend position that this is at all Amazon's problem. The OP even links to the blog post announcing this change months ago. If users pay you for your service to remain S3-compatible that seems like its on you to make sure you live up to that promise, not Amazon. Clicking through to the actual git issues, it definitely seems like the maintainers of Iceberg have the right mental model here too. This is their problem to fix. After re-reading this post this mostly feels like a click-baity way to advertise OpenDAL, which the author appears to be heavily involved in.
  - julik 2y ago
    Requiring a header "just because you sniffed it to usually be there" is not Amazon being cynical, it's creatively-developing-overly-strict-checks. And it happens on the side of the S3-compatible service. If your service no longer works with the AWS SDK because you crash at `headers["content-md5"]` just because "it seemed a good way to make things more correct" - it is on you to fix it, IMO. Like, this changeset https://github.com/minio/minio/pull/20855/files#diff-be8383605d637cd937d113f6baf42a205379935d376ecf1afdd1b631325d9e4aL432 https://github.com/minio/minio/pull/20855/files#diff-be83836... Why does Minio mandate the presence of Content-MD5? Is it in the docs somewhere for the S3 "protocol"? No, it's not. It's someone wanting to "be extra correct with validating user input" and thus creating a subtle extra restriction on the interface they do not control.
    - jameslars 2y ago
      I think you misread my response. I think assuming Amazon did this to hurt “s3 compatible” services is cynical. Amazon implemented a feature, well within their rights. Writing a blog post saying they “broke backwards compatibility” is cynical and disingenuous. Amazon never committed to supporting any random use of their SDK.
      - julik 2y ago
        I did, mea culpa!
- WatchDog 2y ago
  It's one thing when the changes are obviously designed to damage competition, like Microsoft's embrace extend extinguish strategy, but in this case, the breaking changes seem to be pretty obviously motivated by a real need, and there isn't anything preventing so called "S3-compatible" storage services from implementing this new feature.
julik 2y ago
The interesting part here is that if AWS docs state "We expect header X with request Y" the "compatible storage" implementors tend to add validations for presence of header X. In that sense it is tricky for them, but I would still argue that from Postel's law perspective they should not validate things that strictly. There are also ways to determine whether a header is supplied. The AWSv4 signature format adds signed headers, and (AFAIK) the checksum headers usually get signed. The URL signature can be decoded and if the headers are present in the list of signed headers you can go and validate the presence of the said header. The SDK would never sign a header which it doesn't supply.

by tabony 2y ago

genewitch 2y ago

a little less charitable is amazon is throwing its weight around to quash competition before they can get started; and shove tech debt onto third parties. I stopped giving amazon the benefit of the doubt about any aspect of their operations about 8 years ago.

akerl_ 2y ago
Did Amazon recommend that other 3rd party products use their SDK as their own client?
jameslars 2y ago
Less charitable or More cynical? How is Amazon supposed to track a 3rd party pulling their SDK and then reverse-engineering their own service side to work with the SDK? Assuming we're all okay with that premise to begin with, all sorts of other questions start popping up. Do these 3rd parties get veto power over a feature they can't support? Can they delay a launch if they need more time to make their reverse-engineered effort compatible again? It seems a hard to defend position that this is at all Amazon's problem. The OP even links to the blog post announcing this change months ago. If users pay you for your service to remain S3-compatible that seems like its on you to make sure you live up to that promise, not Amazon. Clicking through to the actual git issues, it definitely seems like the maintainers of Iceberg have the right mental model here too. This is their problem to fix. After re-reading this post this mostly feels like a click-baity way to advertise OpenDAL, which the author appears to be heavily involved in.
- julik 2y ago
  Requiring a header "just because you sniffed it to usually be there" is not Amazon being cynical, it's creatively-developing-overly-strict-checks. And it happens on the side of the S3-compatible service. If your service no longer works with the AWS SDK because you crash at `headers["content-md5"]` just because "it seemed a good way to make things more correct" - it is on you to fix it, IMO. Like, this changeset https://github.com/minio/minio/pull/20855/files#diff-be8383605d637cd937d113f6baf42a205379935d376ecf1afdd1b631325d9e4aL432 https://github.com/minio/minio/pull/20855/files#diff-be83836... Why does Minio mandate the presence of Content-MD5? Is it in the docs somewhere for the S3 "protocol"? No, it's not. It's someone wanting to "be extra correct with validating user input" and thus creating a subtle extra restriction on the interface they do not control.
  - jameslars 2y ago
    I think you misread my response. I think assuming Amazon did this to hurt “s3 compatible” services is cynical. Amazon implemented a feature, well within their rights. Writing a blog post saying they “broke backwards compatibility” is cynical and disingenuous. Amazon never committed to supporting any random use of their SDK.
    - julik 2y ago
      I did, mea culpa!
WatchDog 2y ago
It's one thing when the changes are obviously designed to damage competition, like Microsoft's embrace extend extinguish strategy, but in this case, the breaking changes seem to be pretty obviously motivated by a real need, and there isn't anything preventing so called "S3-compatible" storage services from implementing this new feature.

julik 2y ago

The interesting part here is that if AWS docs state "We expect header X with request Y" the "compatible storage" implementors tend to add validations for presence of header X. In that sense it is tricky for them, but I would still argue that from Postel's law perspective they should not validate things that strictly. There are also ways to determine whether a header is supplied. The AWSv4 signature format adds signed headers, and (AFAIK) the checksum headers usually get signed. The URL signature can be decoded and if the headers are present in the list of signed headers you can go and validate the presence of the said header. The SDK would never sign a header which it doesn't supply.