12 ms·

HMAC in Go, Python, Ruby, PHP, and Node.js

27 points by wilsonfiifi 11y ago

fdomig 11y ago
There might be a chance of timing attacks against such "security". A secure system should never depend on obscurity.
- trebor 11y ago
  It's also pretty sad that they didn't use a constant time comparison function. Whether it exists in the target language standard library or not, it's easy enough to write one.
  - muaddirac 11y ago
    Wanted to add that as of 3.3, Python has a nice constant-time compare built in: https://docs.python.org/3/library/hmac.html#hmac.compare_digest https://docs.python.org/3/library/hmac.html#hmac.compare_dig...
    - d0vs 11y ago
      2.7 actually https://docs.python.org/2.7/library/hmac.html#hmac.compare_digest https://docs.python.org/2.7/library/hmac.html#hmac.compare_d...
- jdhawk 11y ago
  It looks like they're including and checking a timestamp with the payload. Is this not sufficient?
  - trebor 11y ago
    No. A timing attack is used to determine the shared key that creates the HMAC. Because == and === (depending on the language) use memcmp(), the time of the comparison varies. EG: 36D5F4EA999342FED17D7488CB260FC92926 36D5F4EA999342FED18D7488CB260FC92926 The code would compare only up to the difference and return, indicating through the time spent analyzing the HMAC how many characters are the same. The attacker can then work the HMAC like they would a combination lock, till they reproduce the key used. That's why a constant time comparison is so important: it leaks far less information.
    - sdevlin 11y ago
      This is a forgery attack, not a key-recovery attack.
      - trebor 11y ago
        I am not a cryptographer. I don't quite understand the distinction you are drawing with so little information. Care to explain what I misunderstood?
        sdevlin 11y ago
        Given a message M and a secret key K, we have: MAC(K, M) -> T To validate a pair (M, T), we verify: T = MAC(K, M) Ideally, the execution time in this verification is independent of T. But many languages use string-comparison algorithms that exit immediately on failure. If Eve can detect this difference with high granularity, she has an oracle telling her how many leading bytes of a guessed tag T' are valid: O(M, T') -> n She can use this to recover the first byte of a valid tag for M: for k in [0, 255]: if O(M, k || 0000...) > 0: return k She can extend this to recover the second byte, the third, and so on. Eventually, Eve will recover T such that MAC(K, M) = T. In other words, Eve is able to forge an authentication tag T for an arbitrary message M. What she won't do is recover K. So while she can forge tags for arbitrary messages, each forgery will require a fresh, online interaction with the verifying party. She cannot work backwards from (M, T) to K.
        sarciszewski 11y ago
        To put it another way: You (slowly and painstakingly) defeat the authentication without having to brute force, e.g. 2^128, possible keys. However, the key is still unknown to you. It just doesn't matter, because you can forge messages without it.
        trebor 11y ago
        Thank you for the in-depth explanation!
    - 11y ago
- sarciszewski 11y ago
  Node.js example: return new Buffer(sig).toString('base64') == signature; PHP example: return hash_hmac("sha512", $string_to_verify, $shared_secret) == $signature; Yeah, like fdomig said: timing attacks. The Python example included a mitigation. PHP includes hash_equals() and a constant time comparison in Javascript isn't difficult to write. Possibly also, the Ruby example: return OpenSSL::HMAC.digest('sha512', shared_secret, string_to_verify) == signature (I don't write Ruby so I don't know if this is overloadable somewhere.)
  - deleted 11y ago
    [deleted]
  - beefhash 11y ago
    > (I don't write Ruby so I don't know if this is overloadable somewhere.) Ruby allows overriding the == operator[1], but OpenSSL::HMAC.digest returns an instance of String[2], rather than returning a special subclass of string or some other kind of special HMAC-representing class overloading ==. [1] http://docs.ruby-lang.org/en/2.2.0/syntax/methods_rdoc.html#label-Method+Names http://docs.ruby-lang.org/en/2.2.0/syntax/methods_rdoc.html#... [2] http://ruby-doc.org/stdlib-2.0.0/libdoc/openssl/rdoc/OpenSSL/HMAC.html#method-c-digest http://ruby-doc.org/stdlib-2.0.0/libdoc/openssl/rdoc/OpenSSL...
    - vidarh 11y ago
      To be pedantic, in Ruby, the fact that an object is of class String is not sufficient evidence that a method has not been overloaded (they can be overloaded via the objects eigenclass), though you're probably right.
      - Freaky 11y ago
        Even if it did (and it doesn't, I checked), you're probably still better off doing it explicitly since it's easier to audit, less likely to differ across implementations, and less error prone (e.g. swap the order of the comparison and you're back to plain old String#==).
  - 11y ago
- tantalic 11y ago
  All crypto libraries should provide and developers should use constant time comparisons for exactly this purpose. A good example from the same page is the Go crypto/hmac package includes mac.Equal to check for equality without introducing timing weaknesses.
jakejake 11y ago
Adding a nonce to the request and storing it is a good idea to add to prevent replay attacks. Especially in the case of OAuth where some implementations hash the query string but don't hash the request body - which could allow all kinds of mischief without even cracking the secret key.
- thenrich99 11y ago
  In this scenario a replay attack is limited to a 30s window during which the timestamp is valid. Depending on the implementation, this may or may not suffice, in which case an nonce would definitely be an improvement albeit with the added complexity.
voidlogic 11y ago
Gah, Go pet peave, they could just use a map literal (good for short low-perf code). Or even better perf/safety wise: type Payload struct { Name string `json:"name"` Category string `json:"category"` Action string `json:"action"` Where string `json:"where"` Timestamp int64 `json:"timestamp,string"` } payload := Payload{ Name: "joe smith", Category: "people", Action: "transport" Where: "pluto", Timestamp: time.Now().Unix(), } jsonBytes, err := json.Marshal(&payload) This code doesn't allocate heap for payload, is much faster and typesafe... P.S. And could be shorter, but I was trying to be clear.
tptacek 11y ago
If some of these dev environments required you to implement HMAC yourself from the SHA2 primitive, I'd see some of the value of a "Rosetta Code"-style page for it. But all of these are just examples of calling into the environment's library interface for HMAC. In fact, except for Golang, all of these examples are essentially just calling into OpenSSL's HMAC. Also: * Do not use string/buffer comparison functions to verify HMAC. In incredibly common error. * Do not use human-intelligible strings as HMAC keys. HMAC keys are like AES keys: fully random; they should come from /dev/urandom. * Do not embed HMAC keys in your source code! Ultimately: I'm not sure I understand the impulse behind pages like this. It's clear that the authors aren't domain experts (that's fine; they're probably better programmers than the crypto domain experts are). They know they aren't getting everything perfect. In fact: they know some of the examples on their own page have flaws, because they were corrected about timing attacks and only fixed Python. Crypto bugs in your own code are one thing, but propagating crypto bugs to other people's code seems much worse.
- ars 11y ago
  > Do not embed HMAC keys in your source code! Where else would you put it? A separate configuration file is still your code (or is that what you meant?) But don't check the key into source control.
  - tlb 11y ago
    In a large Node.js project, I have a file secrets.js, which is full of lines like: exports.passwdHmacKey = 'awaibaeSeingoo4phahjeekee6Ufetev'; Put secrets.js in .gitignore, and copy it where needed manually.
    - tptacek 11y ago
      Things like this are fine. Big sites sometimes have key management servers, provisioned sort of like Zookeeper is. (Just using your comment to riff now:) As long as you're not (a) checking it into VCS and (b) having every developer hold a copy on their laptop, whatever you're doing is fine. What this site does, though, means that any time someone loses their laptop, or really just leaves their laptop unattended somewhere unsafe, everything needs to be reprovisioned. In fact: if you don't have audit controls (who's used which key when), technically, any time that happens, you need to disclose that it happened to your users. A nightmare.
thenrich99 11y ago
Original author here -- thanks to everyone for the feedback. Some time has passed since this was originally written and the remaining languages were never updated to improve the safety of the signature verifications. The original reasoning behind this post was to provide a single reference for signature generation and verification in some common languages -- something I struggled to locate myself. Admittedly, I should've provided warnings about using simple and hardcoded keys in the examples, which were done that way for readability. While there is still a lot of debate about the ability to perform true constant-time comparisons in many of these languages (https://bugs.python.org/issue15061 https://bugs.python.org/issue15061, https://github.com/joyent/node/issues/8560 https://github.com/joyent/node/issues/8560) I agree that for those who would be otherwise using the unsafe string comparisons, the benefits certainly outweigh the slightly added complexity. The updated gists are available in the post to anyone with comments or improvements. Cheers!

27 points by wilsonfiifi 11y ago

fdomig 11y ago

There might be a chance of timing attacks against such "security". A secure system should never depend on obscurity.

trebor 11y ago
It's also pretty sad that they didn't use a constant time comparison function. Whether it exists in the target language standard library or not, it's easy enough to write one.
- muaddirac 11y ago
  Wanted to add that as of 3.3, Python has a nice constant-time compare built in: https://docs.python.org/3/library/hmac.html#hmac.compare_digest https://docs.python.org/3/library/hmac.html#hmac.compare_dig...
  - d0vs 11y ago
    2.7 actually https://docs.python.org/2.7/library/hmac.html#hmac.compare_digest https://docs.python.org/2.7/library/hmac.html#hmac.compare_d...
jdhawk 11y ago
It looks like they're including and checking a timestamp with the payload. Is this not sufficient?
- trebor 11y ago
  No. A timing attack is used to determine the shared key that creates the HMAC. Because == and === (depending on the language) use memcmp(), the time of the comparison varies. EG: 36D5F4EA999342FED17D7488CB260FC92926 36D5F4EA999342FED18D7488CB260FC92926 The code would compare only up to the difference and return, indicating through the time spent analyzing the HMAC how many characters are the same. The attacker can then work the HMAC like they would a combination lock, till they reproduce the key used. That's why a constant time comparison is so important: it leaks far less information.
  - sdevlin 11y ago
    This is a forgery attack, not a key-recovery attack.
    - trebor 11y ago
      I am not a cryptographer. I don't quite understand the distinction you are drawing with so little information. Care to explain what I misunderstood?
      - sdevlin 11y ago
        Given a message M and a secret key K, we have: MAC(K, M) -> T To validate a pair (M, T), we verify: T = MAC(K, M) Ideally, the execution time in this verification is independent of T. But many languages use string-comparison algorithms that exit immediately on failure. If Eve can detect this difference with high granularity, she has an oracle telling her how many leading bytes of a guessed tag T' are valid: O(M, T') -> n She can use this to recover the first byte of a valid tag for M: for k in [0, 255]: if O(M, k || 0000...) > 0: return k She can extend this to recover the second byte, the third, and so on. Eventually, Eve will recover T such that MAC(K, M) = T. In other words, Eve is able to forge an authentication tag T for an arbitrary message M. What she won't do is recover K. So while she can forge tags for arbitrary messages, each forgery will require a fresh, online interaction with the verifying party. She cannot work backwards from (M, T) to K.
        sarciszewski 11y ago
        To put it another way: You (slowly and painstakingly) defeat the authentication without having to brute force, e.g. 2^128, possible keys. However, the key is still unknown to you. It just doesn't matter, because you can forge messages without it.
        trebor 11y ago
        Thank you for the in-depth explanation!
  - 11y ago
sarciszewski 11y ago
Node.js example: return new Buffer(sig).toString('base64') == signature; PHP example: return hash_hmac("sha512", $string_to_verify, $shared_secret) == $signature; Yeah, like fdomig said: timing attacks. The Python example included a mitigation. PHP includes hash_equals() and a constant time comparison in Javascript isn't difficult to write. Possibly also, the Ruby example: return OpenSSL::HMAC.digest('sha512', shared_secret, string_to_verify) == signature (I don't write Ruby so I don't know if this is overloadable somewhere.)
- deleted 11y ago
  [deleted]
- beefhash 11y ago
  > (I don't write Ruby so I don't know if this is overloadable somewhere.) Ruby allows overriding the == operator[1], but OpenSSL::HMAC.digest returns an instance of String[2], rather than returning a special subclass of string or some other kind of special HMAC-representing class overloading ==. [1] http://docs.ruby-lang.org/en/2.2.0/syntax/methods_rdoc.html#label-Method+Names http://docs.ruby-lang.org/en/2.2.0/syntax/methods_rdoc.html#... [2] http://ruby-doc.org/stdlib-2.0.0/libdoc/openssl/rdoc/OpenSSL/HMAC.html#method-c-digest http://ruby-doc.org/stdlib-2.0.0/libdoc/openssl/rdoc/OpenSSL...
  - vidarh 11y ago
    To be pedantic, in Ruby, the fact that an object is of class String is not sufficient evidence that a method has not been overloaded (they can be overloaded via the objects eigenclass), though you're probably right.
    - Freaky 11y ago
      Even if it did (and it doesn't, I checked), you're probably still better off doing it explicitly since it's easier to audit, less likely to differ across implementations, and less error prone (e.g. swap the order of the comparison and you're back to plain old String#==).
- 11y ago
tantalic 11y ago
All crypto libraries should provide and developers should use constant time comparisons for exactly this purpose. A good example from the same page is the Go crypto/hmac package includes mac.Equal to check for equality without introducing timing weaknesses.

jakejake 11y ago

Adding a nonce to the request and storing it is a good idea to add to prevent replay attacks. Especially in the case of OAuth where some implementations hash the query string but don't hash the request body - which could allow all kinds of mischief without even cracking the secret key.

thenrich99 11y ago
In this scenario a replay attack is limited to a 30s window during which the timestamp is valid. Depending on the implementation, this may or may not suffice, in which case an nonce would definitely be an improvement albeit with the added complexity.

voidlogic 11y ago

Gah, Go pet peave, they could just use a map literal (good for short low-perf code). Or even better perf/safety wise: type Payload struct { Name string `json:"name"` Category string `json:"category"` Action string `json:"action"` Where string `json:"where"` Timestamp int64 `json:"timestamp,string"` } payload := Payload{ Name: "joe smith", Category: "people", Action: "transport" Where: "pluto", Timestamp: time.Now().Unix(), } jsonBytes, err := json.Marshal(&payload) This code doesn't allocate heap for payload, is much faster and typesafe... P.S. And could be shorter, but I was trying to be clear.

tptacek 11y ago

If some of these dev environments required you to implement HMAC yourself from the SHA2 primitive, I'd see some of the value of a "Rosetta Code"-style page for it. But all of these are just examples of calling into the environment's library interface for HMAC. In fact, except for Golang, all of these examples are essentially just calling into OpenSSL's HMAC. Also: * Do not use string/buffer comparison functions to verify HMAC. In incredibly common error. * Do not use human-intelligible strings as HMAC keys. HMAC keys are like AES keys: fully random; they should come from /dev/urandom. * Do not embed HMAC keys in your source code! Ultimately: I'm not sure I understand the impulse behind pages like this. It's clear that the authors aren't domain experts (that's fine; they're probably better programmers than the crypto domain experts are). They know they aren't getting everything perfect. In fact: they know some of the examples on their own page have flaws, because they were corrected about timing attacks and only fixed Python. Crypto bugs in your own code are one thing, but propagating crypto bugs to other people's code seems much worse.

ars 11y ago
> Do not embed HMAC keys in your source code! Where else would you put it? A separate configuration file is still your code (or is that what you meant?) But don't check the key into source control.
- tlb 11y ago
  In a large Node.js project, I have a file secrets.js, which is full of lines like: exports.passwdHmacKey = 'awaibaeSeingoo4phahjeekee6Ufetev'; Put secrets.js in .gitignore, and copy it where needed manually.
  - tptacek 11y ago
    Things like this are fine. Big sites sometimes have key management servers, provisioned sort of like Zookeeper is. (Just using your comment to riff now:) As long as you're not (a) checking it into VCS and (b) having every developer hold a copy on their laptop, whatever you're doing is fine. What this site does, though, means that any time someone loses their laptop, or really just leaves their laptop unattended somewhere unsafe, everything needs to be reprovisioned. In fact: if you don't have audit controls (who's used which key when), technically, any time that happens, you need to disclose that it happened to your users. A nightmare.

thenrich99 11y ago

Original author here -- thanks to everyone for the feedback. Some time has passed since this was originally written and the remaining languages were never updated to improve the safety of the signature verifications. The original reasoning behind this post was to provide a single reference for signature generation and verification in some common languages -- something I struggled to locate myself. Admittedly, I should've provided warnings about using simple and hardcoded keys in the examples, which were done that way for readability. While there is still a lot of debate about the ability to perform true constant-time comparisons in many of these languages (https://bugs.python.org/issue15061 https://bugs.python.org/issue15061, https://github.com/joyent/node/issues/8560 https://github.com/joyent/node/issues/8560) I agree that for those who would be otherwise using the unsafe string comparisons, the benefits certainly outweigh the slightly added complexity. The updated gists are available in the post to anyone with comments or improvements. Cheers!