8 ms·
AWS SIGv4 and SIGv4A – How AWS signs and verifies API requests
- deanCommie 5y agoSuper interesting! I wonder - have any other services or companies (SaaS or otherwise) adopted SigV4 other than AWS? Seems like a solid protocol (though a lot to implement in a client, that can be encapsulated in an SDK)
- jaytaylor 5y agoIs there a benefit to companies all using the exact same web service request signature scheme? One downside is that it would make widespread web service API abuse even easier than it already is. One would need to understand even less about the mechanics and workings of same-scheme services across companies. Yes, this is security through obscurity, but why leave more surface area open than necessary? Malicious actors will certainly appreciate it!
- whartung 5y ago> Is there a benefit to companies all using the exact same web service request signature scheme? Well, there's some things to consider. First, Amazon AWS is a very rich target. It hosts a very large quantity of services that would be ripe for exploitation. I'm sure attackers would love to find holes in the AWS mechanism. Second, because of number 1, I'm sure the AWS services are not only a prime target for attack, but also actively being attacked. By both nefarious black hat ne'er do wells, to state level agencies. I would fully expect US, Russian, Chinese and other state intelligence agencies to be very interested in an exploit of something as ripe as the AWS system. Third, Amazon has the motivation, due to number 1, to keep on top of and ensure that its technique is sound and robust. Not only that, it has the resources to do it. Four, the technique is open, and documented, and available to all. No skullduggery is required at an algorithm level to analyze it and understand it. Thesis seeking white hat PhD students can have at it and advance the field. So, if there were to be "one algorithm", Amazon and AWS have the experience and know how to hold theirs up high on a pedestal labeled "You could do a lot worse".
- edoceo 5y agoI'm using signatures, influenced by AWS and ActivityPub specs for some of our APIs. But I'm using Sodium for the hashing/signing.
- et1337 5y agoWe use a flavor of SigV4 for our REST and gRPC APIs at StrongDM[1] [1] https://www.strongdm.com/ https://www.strongdm.com/
- phnofive 5y agoWasn't clear to me who wrote this or why, so: > I’m Colm MacCárthaigh, a VP and Distinguished Engineer at Amazon Web Services, and this website is my blog.
- capableweb 5y ago> a VP and Distinguished Engineer at Amazon Web Services Any further translation on what VP and "Distinguished Engineer" means here? It sounds like both of them are honorary and doesn't really mean much, but then "Distinguished" makes it seem they received some sort of Nobal price of computing as well. And if neither of them really mean anything, then this guy is really "just" a developer right?
- Thaxll 5y ago> And if neither of them really mean anything, then this guy is really "just" a developer right? Probably yeah, he went to a coding boodcamp and was hired after that.
- BikiniPrince 5y agoColm is a fairly badass engineer. He was a senior principle when I left. Sounds like they made another level since then and he jumped into VP. Somewhat makes sense as he leads the business in both a technical and business sense. Amazon’s traditional issue is they only know how to level managers. Turns out when the company starts accruing some really senior people you have to accommodate them. This is a fairly fluffy piece of writing. His internal writings were much deeper.
- bbgm 5y agoAmazon has long had a VP-level engineer track (Peter Vosshall was the first IIRC). Colm is very rightly one.
- BikiniPrince 5y ago
- psanford 5y agoThis is neat, although it throws a wrench into my recent move encrypt my AWS secret access key with my TPM to prevent the key from being used anywhere besides my laptop[0]. [0]: https://github.com/psanford/awsv4signer/tree/main/examples/tpmsigner https://github.com/psanford/awsv4signer/tree/main/examples/t...
- gnabgib 5y agoThe article is vague (no link to docs, no formula, no block diagram), but it seems to say the fundamental process still runs via a pre-shared-secret. That is secret you're trying to secure, and one that AWS knows. However, that secret is no longer used for the signing. Instead it's combined with a date and service-description to generate a public-private keypair, which is itself not secret (both AWS and you can derive this if you know the algorithm). Your requests are now signed with this private key (client side) and can be verified by the services with the public key (server side). AWS still knows your secret, but it keeps it in a more secured location verses out on the farms. The farms would need a way to request your public key for the day from the secured location, so they can verify the sender. But, the farms no longer have access to your secret key, so cannot sign other requests masquerading as you.
- psanford 5y agoIf the ECDSA key pair is still derived using an hmac keyed by the private key, I can do that with the key loaded in the tpm. Otherwise it is unlikely to work with my current setup. Once the key is loaded into the TPM as an HMAC key, you can no longer get the plaintext key back out. You have to use the TPM to perform any HMAC operations using the key.
- twistedpair 5y agoFeels odd that AWS still uses secret tokens all these years later. Take GCP for example. They use your personal identity (via OAuth, service accounts use certs), and you can use FIDO tokens to authenticate your local keys (e.g. for gcloud CLI). I was very surprised to see that awscli still doesn't support Yubikeys.
- GauntletWizard 5y agoI flat out don't trust AWS SIG; It falls into the classic encryption antipattern of signing the meaning and not the bytes - Anytime a transformation of the data is required before or after the signature is applied, you open up a hole for attackers to exploit. See V1 and V3, which were flat out insecure and have been abandoned. Sign the bytes, not the meaning.
- tptacek 5y agoYou often can't sign the meaning, because the systems handling the request can reorder the bytes, and there are unbounded number of byte sequences mapping to a given meaning. That's the whole reason we have canonicalization. Canonicalization is treacherous and generalist developers shouldn't freelance their own formats. Canonicalization problems are one of the reasons we're at SIGV4 and not just, like, SIG. But Amazon has at this point thought about this as carefully as any organization in the Internet. Which is why it's good advice to, if you're going for a signed-request API, just shoplift SIGV4 from them.
- OJFord 5y agoCould you explain (even just a link if you have one) what you and GP mean by 'bytes vs. meaning'? I can't really imagine what the latter is, other than perhaps signing byte values within a structure rather than the structure as a whole, and if that's it I don't understand how it's a non trivial difference, or what you mean by 'byte sequences mapping to a given meaning'. Not looking to roll my own, just curious!
- tptacek 5y agoThe parent is noting that what actually gets signed in SIGV4 isn't the raw request, but a pseudo-request derived from the raw; the pseudo is the "meaning" (in their terminology); a cryptography engineer would call it the "canonicalized" request. What they'd like instead is a signature format that simply signed the raw request, the "bytes".
- 5y ago
- nielsole 5y agoI wonder why the subkeys are valid for 1d and not something much shorter like 15 minutes. IIUC if the frontend IAM servers of a service get compromised you can only accept traffic for that service again on the next day, when a new subkey is valid. I also wonder whether those subkeys are copied over to the frontend server (that might be a lot of data to copy) every day or requested and cached (possibly high latency for first call)
- gjs278 5y ago>IIUC never use this again. we already have IIRC and “I think” for fucks sake just stick with those
- killingtime74 5y agoMaybe they have infrastructure to deal with these situations?
- Corrado 5y agoI took it as a way to handle clock skew. Like many others, I've had lots of problems with clients having a clock that is pretty inaccurate. Though I think I agree with you that I don't think I've ever had a client with a skew of 1d.
- nielsole 5y agoYou could choose a grace period independently. Downside is that you have to keep all currently accepted keys around so there is a limit somewhere on storage costs.
- georgekb 5y agoThanks for this great post. I actually just signed up to say that I actually wasted a considerable amount of time trying to create my own SIGv4 just to avoid replay attacks especially if the server is behind some L-7 load balancer such as Cloudflare more than any other reason. It was implemented like this: the server creates a ecdsa/ed22519 key pair, hands over the private key to the user and keeps only the public key, then for every request to the actual APIs, the user uses the private key to create a JWT that contains some attributes such as the userID, sessionID, the request path, etc...). While this method obviously looked much more secure than just a long-term static JWT or an access token, the real problem I found was in the clock skew, if the client doesn't have a well synchronized clock, then the JWT is deemed invalid by the server since such this JWT should be very short-lived by design otherwise what's the point of having them in the first place. If your clients are browsers or running on PCs, as opposed to servers, containers, etc... this will be a huge problem for you. In fact, AWS clients implement their own clock skewing correction https://aws.amazon.com/blogs/developer/clock-skew-correction/ https://aws.amazon.com/blogs/developer/clock-skew-correction... I ended up implementing an additional auth API endpoint where you use your long-lived secret to periodically request a short-lived JWT that is only valid for a few hours, this short-lived secret is what is used for the actual APIs to authenticate the users.
- smashed 5y agoYour first approach reminds me of http message signature: https://datatracker.ietf.org/doc/draft-ietf-httpbis-message-signatures/ https://datatracker.ietf.org/doc/draft-ietf-httpbis-message-... Library support for it is spotty, depending on your framework of choice but they exists. Your second approach is covered by oAuth refreshable tokens I believe. I went through the same process as you in an API project, eventually ended up using Keycloak as the auth server.
- georgekb 5y agoWow, never heard of HTTP Message Signatures until now. However, like I said, Clock skew is a huge problem in this method. I found out about it much later after implementation when I used the client on some old Windows machine whose clock wasn't synchronized and I spent like a day until I figured out it was the clock skew which made this whole method look suspicious to me even it's obviously much more secure. It could be great in a microservice environment where you're pretty sure that all your endpoints have well synced clock, but when your clients are browsers and PCs, it is about time until you experience it. As for the second approach, yes exactly it's very similar to OAuth2. In fact I guess this is the method used by GCP APIs by service accounts. The clients use their long-lived secrets to get an oauth2 access token, this access token is JWT I guess that also contains authorization information such as scopes.