3 ms·

Curl native support for –aws-sigv4 (2023)

20 points by paulddraper 2y ago

paulddraper 2y ago
This is helpful for calling AWS Gateway API with AWS_IAM authorization. Or for calling AWS APIs in containers, etc. without installing heavyweight AWS CLI.
Galanwe 2y ago
Happy to see that, though it would have been nice if it retrieved the credentials automatically instead of explicitly requiring them.
lopkeny12ko 2y ago
Why are we celebrating proprietary, vendor-specific-extensions in our open source tools that are, by nature, supposed to operate in a vendor-agnostic manner? This flag makes zero sense for anyone who isn't an AWS customer. This is even more insulting considering that "being an AWS customer" is not free (both as in beer and freedom). It really surprises me that this made it into curl. I've always been a fan of Terraform's provider architecture--all the vendor-specific bits are implemented as third-party plugins that implement a standard API and communicate with Terraform itself in a client-server model.
- aeyes 2y ago
  Is it really proprietary? We now have a lot of S3 compatible servers, MinIO for example requires this type of signature for requests.
  - alexhornbake 2y ago
    ^ this. The AWS v4 signing method is a spec for how to do HMAC request signing. It is not limited to use only with AWS services. You can use it to sign requests on any service, and there are many implementations. Here is an example of one I’ve used in the past: https://github.com/auroratechnologies/vangoh https://github.com/auroratechnologies/vangoh Or as mentioned above: https://github.com/minio/minio/blob/master/cmd/signature-v4.go https://github.com/minio/minio/blob/master/cmd/signature-v4....
- gsk22 2y ago
  Isn't that the beauty of open source? That we are empowered to make tools that fit our needs and the needs of our users? I don't see why open source tools should be intentionally crippled in the name of some sort of "purity". Implementing this feature costs nothing (it is an open standard, unencumbered by licenses or patents), and you can continue to use the tool as if the vendor-specific option doesn't exist.
  - yjftsjthsd-h 2y ago
    It's less about purity and more about overhead. Adding this feature adds friction to refactoring the code, adds surface area for vulnerabilities (and the user ignoring it won't make that go away), and makes the binary bigger. The cost/benefit ratio might well be favorable, but implementing this feature absolutely costs something.
- akira2501 2y ago
  > celebrating proprietary, vendor-specific-extensions in our open source tools It's not proprietary nor vendor specific. I'm not sure what makes cURL "our" tool? > by nature, supposed to operate in a vendor-agnostic manner? Does the inclusion of this feature mean a different feature cannot coexist with it? Was it so absurdly complicated to develop that it took time away from other integrations? > This flag makes zero sense for anyone who isn't an AWS customer. Do you also dislike QUIC or RTMP? How about NTLM? > that "being an AWS customer" is not free Yea but "using AWS protocols outside of AWS" _is_ free. > It really surprises me that this made it into curl. They should turn down useful but impure contributions? The lead developer of cURL still has a day job.
- kelnos 2y ago
  Because some people believe tools exist to make lives easier, not to evangelize some sort of ideology. A lot of people use AWS; adding helpers for it into commonly-used tools can make people's lives easier. And beyond that, there a bunch of clones of various AWS APIs, both in other proprietary products, and in open source alternatives. I bet this functionality works with those, too.
  - Scaevolus 2y ago
    It's quite a clean extension that can be disabled at compile time. It doesn't complicate the rest of curl, and provides a useful way to invoke any AWS API from the CLI without needing a giant AWS library dependency. It's well worth a thousand lines of code. https://github.com/curl/curl/blob/b9fea261e632b9493ea6038bfb3da6075909cc39/lib/http_aws_sigv4.c https://github.com/curl/curl/blob/b9fea261e632b9493ea6038bfb...
    - lopkeny12ko 2y ago
      As an open source maintainer, this is what I would consider a slippery slope. Sure, this one is well-specified and "not that complex" until N years later there are 30 compile-time flags for all sorts of vendor-specific features and a new CVE every week from the massive increase in highly-use-case-specific code to maintain.
    - matt-attack 2y ago
      I’m not that familiar with AWS innards. Can you explain how this one flag can access any API?
      - Scaevolus 2y ago
        AWS uses one algorithm to authenticate requests against all of its APIs. Many of its APIs accept simple query parameters or JSON, making them easy to call with curl. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-signing.html https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_a...
- akdev1l 2y ago
  As some other people pointed out. AWS SigV4 is just a spec on how to apply HMAC to validate/authenticate requests. You can implement quite easily in any language without touching any Amazon tooling or infrastructure.
thomastaylor 2y ago
I appreciate the shout! To accommodate the influx of people coming to the blog, I updated the post to include IAM role assumption credentials as well. I hope you all enjoy!
NanoWar 2y ago
I had fun recently to make such a request in C#. In the end I ported the code from the aws typescript implementation to C# which was surprisingly easy: https://gist.github.com/NanoWar/788860d74c49f337f4c3dab6dd24871a https://gist.github.com/NanoWar/788860d74c49f337f4c3dab6dd24...
dmart 2y ago
Just mentioning that I love AWS' signed request model. It's so much easier than dealing with OAuth for APIs and I wish it would catch on more broadly. It's inspired an IETF RFC (https://datatracker.ietf.org/doc/html/rfc9421 https://datatracker.ietf.org/doc/html/rfc9421), so we'll see.

20 points by paulddraper 2y ago

This is helpful for calling AWS Gateway API with AWS_IAM authorization. Or for calling AWS APIs in containers, etc. without installing heavyweight AWS CLI.

Galanwe 2y ago

Happy to see that, though it would have been nice if it retrieved the credentials automatically instead of explicitly requiring them.

lopkeny12ko 2y ago

Why are we celebrating proprietary, vendor-specific-extensions in our open source tools that are, by nature, supposed to operate in a vendor-agnostic manner? This flag makes zero sense for anyone who isn't an AWS customer. This is even more insulting considering that "being an AWS customer" is not free (both as in beer and freedom). It really surprises me that this made it into curl. I've always been a fan of Terraform's provider architecture--all the vendor-specific bits are implemented as third-party plugins that implement a standard API and communicate with Terraform itself in a client-server model.

aeyes 2y ago
Is it really proprietary? We now have a lot of S3 compatible servers, MinIO for example requires this type of signature for requests.
- alexhornbake 2y ago
  ^ this. The AWS v4 signing method is a spec for how to do HMAC request signing. It is not limited to use only with AWS services. You can use it to sign requests on any service, and there are many implementations. Here is an example of one I’ve used in the past: https://github.com/auroratechnologies/vangoh https://github.com/auroratechnologies/vangoh Or as mentioned above: https://github.com/minio/minio/blob/master/cmd/signature-v4.go https://github.com/minio/minio/blob/master/cmd/signature-v4....
gsk22 2y ago
Isn't that the beauty of open source? That we are empowered to make tools that fit our needs and the needs of our users? I don't see why open source tools should be intentionally crippled in the name of some sort of "purity". Implementing this feature costs nothing (it is an open standard, unencumbered by licenses or patents), and you can continue to use the tool as if the vendor-specific option doesn't exist.
- yjftsjthsd-h 2y ago
  It's less about purity and more about overhead. Adding this feature adds friction to refactoring the code, adds surface area for vulnerabilities (and the user ignoring it won't make that go away), and makes the binary bigger. The cost/benefit ratio might well be favorable, but implementing this feature absolutely costs something.
akira2501 2y ago
> celebrating proprietary, vendor-specific-extensions in our open source tools It's not proprietary nor vendor specific. I'm not sure what makes cURL "our" tool? > by nature, supposed to operate in a vendor-agnostic manner? Does the inclusion of this feature mean a different feature cannot coexist with it? Was it so absurdly complicated to develop that it took time away from other integrations? > This flag makes zero sense for anyone who isn't an AWS customer. Do you also dislike QUIC or RTMP? How about NTLM? > that "being an AWS customer" is not free Yea but "using AWS protocols outside of AWS" _is_ free. > It really surprises me that this made it into curl. They should turn down useful but impure contributions? The lead developer of cURL still has a day job.
kelnos 2y ago
Because some people believe tools exist to make lives easier, not to evangelize some sort of ideology. A lot of people use AWS; adding helpers for it into commonly-used tools can make people's lives easier. And beyond that, there a bunch of clones of various AWS APIs, both in other proprietary products, and in open source alternatives. I bet this functionality works with those, too.
- Scaevolus 2y ago
  It's quite a clean extension that can be disabled at compile time. It doesn't complicate the rest of curl, and provides a useful way to invoke any AWS API from the CLI without needing a giant AWS library dependency. It's well worth a thousand lines of code. https://github.com/curl/curl/blob/b9fea261e632b9493ea6038bfb3da6075909cc39/lib/http_aws_sigv4.c https://github.com/curl/curl/blob/b9fea261e632b9493ea6038bfb...
  - lopkeny12ko 2y ago
    As an open source maintainer, this is what I would consider a slippery slope. Sure, this one is well-specified and "not that complex" until N years later there are 30 compile-time flags for all sorts of vendor-specific features and a new CVE every week from the massive increase in highly-use-case-specific code to maintain.
  - matt-attack 2y ago
    I’m not that familiar with AWS innards. Can you explain how this one flag can access any API?
    - Scaevolus 2y ago
      AWS uses one algorithm to authenticate requests against all of its APIs. Many of its APIs accept simple query parameters or JSON, making them easy to call with curl. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-signing.html https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_a...
akdev1l 2y ago
As some other people pointed out. AWS SigV4 is just a spec on how to apply HMAC to validate/authenticate requests. You can implement quite easily in any language without touching any Amazon tooling or infrastructure.

thomastaylor 2y ago

I appreciate the shout! To accommodate the influx of people coming to the blog, I updated the post to include IAM role assumption credentials as well. I hope you all enjoy!

NanoWar 2y ago

I had fun recently to make such a request in C#. In the end I ported the code from the aws typescript implementation to C# which was surprisingly easy: https://gist.github.com/NanoWar/788860d74c49f337f4c3dab6dd24871a https://gist.github.com/NanoWar/788860d74c49f337f4c3dab6dd24...

dmart 2y ago

Just mentioning that I love AWS' signed request model. It's so much easier than dealing with OAuth for APIs and I wish it would catch on more broadly. It's inspired an IETF RFC (https://datatracker.ietf.org/doc/html/rfc9421 https://datatracker.ietf.org/doc/html/rfc9421), so we'll see.