7 ms·

Show HN: Securely Handle Encryption Keys in Go

119 points by libeclipse 9y ago

nemo1618 9y ago
Why is Protect a non-blocking call? That means that after I call Protect on some data, it's not actually protected yet.
- libeclipse 9y ago
  Hey there! Thanks for this advice, I'll definitely consider this in the next release. Like I said in other parts of this thread, the project is in very early stages right now and, as I understand from reading a lot of the replies, I seem to have missed a lot. All of this stuff will be fixed soon.
  - stouset 9y ago
    I would strongly encourage you to just wrap `libsodium`. The authors have thought about this problem a lot, and have done the hard work for you — there is a surprising amount being done behind the scenes to give the kinds of guarantees cryptographic keys warrant. If CGo isn't acceptable, at least use their implementation to guide the design your go-native version.
amenghra 9y ago
Prior art: https://github.com/stouset/go.secrets https://github.com/stouset/go.secrets (with a technical description at https://github.com/stouset/go.secrets/blob/master/secrets.go#L4 https://github.com/stouset/go.secrets/blob/master/secrets.go...)
- encryptThrow32 9y ago
  go.secrets uses the libsodium sodium_memzero to clear the bytes, whereas memguard sets each byte to 0. I feel more comfortable with the first, but can't exactly explain why. memguard seems better organised in the repo like a ready to go package; I think go.secrets would be a better solution if it was organised as well as memguard.
  - libeclipse 9y ago
    I've been meaning to implement libsodium's sodium_memzero instead of zeroing out. It's next on the Agenda.
    - stouset 9y ago
      Hey there! Thanks for building this. I hate to rain on your parade, but your approach, unfortunately, won't work as currently written[1]. You have to manage the memory yourself. https://news.ycombinator.com/item?id=14174500 https://news.ycombinator.com/item?id=14174500
      - libeclipse 9y ago
        Hey there stouset! I've read your comments and I thank you for the tips you've given. I do realise that there are a lot of things that I have missed, and I plan on having them fixed soon. I definitely will be managing memory myself. The project is in very early stages at this point, just in v0.1.0 right now.
        eropple 9y ago
        I'm not trying to pick on you or discourage you, but "just v0.1.0" doesn't excuse that it doesn't work. Whether they should or not, people trust projects like this because they, too, don't have the capability to evaluate whether or not this works. I wouldn't expect potential consumers to understand that Go's garbage collector makes this a nonviable approach any more than this project currently does. Right now, this is a primed hand grenade of a project. You should disclaim its insecurity at the top of the README. Be very, very specific that it is not currently functional and discourage anyone from using this until it is functional.
        eropple 9y ago
        And to be clear, saying "don't use it in production" isn't the same thing as saying it doesn't work at all. Your change is irresponsible, because it doesn't work at all.
e79 9y ago
Looking at this more closely, this takes arbitrary buffers of data and uses syscalls such as mlock to prevent paging memory to disk, as well as cleans up at the end by zero'ing out the buffers for you. Has this been audited in any way? Is there a garuntee that the Go runtime won't, say, keep a duplicate of the buffer you copy() from in memory somewhere that can be paged? Additionally, a technical description of how this works in the README would be nice for those that aren't familiar with how the memory gets locked. Neat project. Curious to see how this develops.
- stouset 9y ago
  The linked approach does not work for exactly the reasons you specify. Go can and will move and copy memory around as it sees fit, which defeats the purpose of this library. You must manage the memory yourself to have any chance of doing this reliably. https://news.ycombinator.com/item?id=14174500 https://news.ycombinator.com/item?id=14174500
  - empath75 9y ago
    Does hashicorp's vault have similar problems?
    - cheeseprocedure 9y ago
      Yes, but it's (reasonably) excluded from their threat model: https://github.com/hashicorp/vault/issues/1446#issuecomment-221625921 https://github.com/hashicorp/vault/issues/1446#issuecomment-...
      - deleted 9y ago
        [deleted]
    - mitchellh 9y ago
      Not in the same way. Vault performs an mlock on its entire memory space (all current and future pages). So even as Go copies and moves around memory, none of the process memory should ever be paged to disk. As cheeseprocedure correctly stated, we exclude memory access from our threat model. Even if you're disallowing the system from paging your memory, there are other ways of accessing a process's memory. Direct access to Vault's address space is not covered by our threat model. This doesn't mean we don't care about the problem completely: we'll make a reasonable effort in cases to protect against things even not covered by our threat model. However, due to the nature of Go or the complexity to create a locked down solution, we can't claim to cover it in our model. Vault has mlocked memory (what this library does) since its first release (0.1).
ziikutv 9y ago
I have no technical expertise to chime in. But I just wanted to say, that I like how nicely formatted the README is. I think the only feedback would be to show sample usages and adding a bit more technical information* [*] That would be understandable to people with basic encryption knowledge.
__michaelg 9y ago
This is fun in theory, but doesn't really give you a lot in practice. A few random points: * On many server systems swap is already disabled today (for availability reasons), so you don't really win anything. (Although what you're doing also won't hurt, so it's fine.) * On many desktop systems on the other hand, swap is not the only reason for memory to end up on disk, mainly due to hibernation modes. This doesn't just include OS-implemented hibernation modes, but also firmware-provided ones (e.g. Intel's RST.) * If you're running in a VM (in the cloud?), the hypervisor pretty much doesn't care what you lock in memory. * If your process crashes you may end up with a crash dump, depending on the system's configuration. (On Linux you can avoid that using prctl's PR_SET_DUMPABLE option.) Even if none of that is a problem for you, you still need to fill and use the values you protected. Where do you get the keys from? Is that path protected as well? This might be fine if you can generate them in place, but even then it's pretty hard. The same is true for key usage: How do you make sure that the key doesn't end up in another portion of your memory? It's almost certain that it ends up either on the stack or somewhere else sooner or later...
zimbatm 9y ago
Hashicorp Vault seems to be locking all the memory: https://github.com/hashicorp/vault/blob/c44f1c9817955d4c7cd5822a19fb492e1c2d0c54/helper/mlock/mlock_unix.go https://github.com/hashicorp/vault/blob/c44f1c9817955d4c7cd5... They also set the ipc_lock capability before starting the process in docker: https://github.com/hashicorp/docker-vault/blob/e8edfef53deb64e1898fea66e5fb7488b7dc3154/0.6.3/docker-entrypoint.sh#L83 https://github.com/hashicorp/docker-vault/blob/e8edfef53deb6...
libeclipse 9y ago
To anyone reading this in the future, all of the issues raised in this thread have been handled in v0.2.0.

119 points by libeclipse 9y ago

nemo1618 9y ago

Why is Protect a non-blocking call? That means that after I call Protect on some data, it's not actually protected yet.

libeclipse 9y ago
Hey there! Thanks for this advice, I'll definitely consider this in the next release. Like I said in other parts of this thread, the project is in very early stages right now and, as I understand from reading a lot of the replies, I seem to have missed a lot. All of this stuff will be fixed soon.
- stouset 9y ago
  I would strongly encourage you to just wrap `libsodium`. The authors have thought about this problem a lot, and have done the hard work for you — there is a surprising amount being done behind the scenes to give the kinds of guarantees cryptographic keys warrant. If CGo isn't acceptable, at least use their implementation to guide the design your go-native version.

amenghra 9y ago

Prior art: https://github.com/stouset/go.secrets https://github.com/stouset/go.secrets (with a technical description at https://github.com/stouset/go.secrets/blob/master/secrets.go#L4 https://github.com/stouset/go.secrets/blob/master/secrets.go...)

encryptThrow32 9y ago
go.secrets uses the libsodium sodium_memzero to clear the bytes, whereas memguard sets each byte to 0. I feel more comfortable with the first, but can't exactly explain why. memguard seems better organised in the repo like a ready to go package; I think go.secrets would be a better solution if it was organised as well as memguard.
- libeclipse 9y ago
  I've been meaning to implement libsodium's sodium_memzero instead of zeroing out. It's next on the Agenda.
  - stouset 9y ago
    Hey there! Thanks for building this. I hate to rain on your parade, but your approach, unfortunately, won't work as currently written[1]. You have to manage the memory yourself. https://news.ycombinator.com/item?id=14174500 https://news.ycombinator.com/item?id=14174500
    - libeclipse 9y ago
      Hey there stouset! I've read your comments and I thank you for the tips you've given. I do realise that there are a lot of things that I have missed, and I plan on having them fixed soon. I definitely will be managing memory myself. The project is in very early stages at this point, just in v0.1.0 right now.
      - eropple 9y ago
        I'm not trying to pick on you or discourage you, but "just v0.1.0" doesn't excuse that it doesn't work. Whether they should or not, people trust projects like this because they, too, don't have the capability to evaluate whether or not this works. I wouldn't expect potential consumers to understand that Go's garbage collector makes this a nonviable approach any more than this project currently does. Right now, this is a primed hand grenade of a project. You should disclaim its insecurity at the top of the README. Be very, very specific that it is not currently functional and discourage anyone from using this until it is functional.
        eropple 9y ago
        And to be clear, saying "don't use it in production" isn't the same thing as saying it doesn't work at all. Your change is irresponsible, because it doesn't work at all.

e79 9y ago

Looking at this more closely, this takes arbitrary buffers of data and uses syscalls such as mlock to prevent paging memory to disk, as well as cleans up at the end by zero'ing out the buffers for you. Has this been audited in any way? Is there a garuntee that the Go runtime won't, say, keep a duplicate of the buffer you copy() from in memory somewhere that can be paged? Additionally, a technical description of how this works in the README would be nice for those that aren't familiar with how the memory gets locked. Neat project. Curious to see how this develops.

stouset 9y ago
The linked approach does not work for exactly the reasons you specify. Go can and will move and copy memory around as it sees fit, which defeats the purpose of this library. You must manage the memory yourself to have any chance of doing this reliably. https://news.ycombinator.com/item?id=14174500 https://news.ycombinator.com/item?id=14174500
- empath75 9y ago
  Does hashicorp's vault have similar problems?
  - cheeseprocedure 9y ago
    Yes, but it's (reasonably) excluded from their threat model: https://github.com/hashicorp/vault/issues/1446#issuecomment-221625921 https://github.com/hashicorp/vault/issues/1446#issuecomment-...
    - deleted 9y ago
      [deleted]
  - mitchellh 9y ago
    Not in the same way. Vault performs an mlock on its entire memory space (all current and future pages). So even as Go copies and moves around memory, none of the process memory should ever be paged to disk. As cheeseprocedure correctly stated, we exclude memory access from our threat model. Even if you're disallowing the system from paging your memory, there are other ways of accessing a process's memory. Direct access to Vault's address space is not covered by our threat model. This doesn't mean we don't care about the problem completely: we'll make a reasonable effort in cases to protect against things even not covered by our threat model. However, due to the nature of Go or the complexity to create a locked down solution, we can't claim to cover it in our model. Vault has mlocked memory (what this library does) since its first release (0.1).

ziikutv 9y ago

I have no technical expertise to chime in. But I just wanted to say, that I like how nicely formatted the README is. I think the only feedback would be to show sample usages and adding a bit more technical information* [*] That would be understandable to people with basic encryption knowledge.

__michaelg 9y ago

This is fun in theory, but doesn't really give you a lot in practice. A few random points: * On many server systems swap is already disabled today (for availability reasons), so you don't really win anything. (Although what you're doing also won't hurt, so it's fine.) * On many desktop systems on the other hand, swap is not the only reason for memory to end up on disk, mainly due to hibernation modes. This doesn't just include OS-implemented hibernation modes, but also firmware-provided ones (e.g. Intel's RST.) * If you're running in a VM (in the cloud?), the hypervisor pretty much doesn't care what you lock in memory. * If your process crashes you may end up with a crash dump, depending on the system's configuration. (On Linux you can avoid that using prctl's PR_SET_DUMPABLE option.) Even if none of that is a problem for you, you still need to fill and use the values you protected. Where do you get the keys from? Is that path protected as well? This might be fine if you can generate them in place, but even then it's pretty hard. The same is true for key usage: How do you make sure that the key doesn't end up in another portion of your memory? It's almost certain that it ends up either on the stack or somewhere else sooner or later...

zimbatm 9y ago

Hashicorp Vault seems to be locking all the memory: https://github.com/hashicorp/vault/blob/c44f1c9817955d4c7cd5822a19fb492e1c2d0c54/helper/mlock/mlock_unix.go https://github.com/hashicorp/vault/blob/c44f1c9817955d4c7cd5... They also set the ipc_lock capability before starting the process in docker: https://github.com/hashicorp/docker-vault/blob/e8edfef53deb64e1898fea66e5fb7488b7dc3154/0.6.3/docker-entrypoint.sh#L83 https://github.com/hashicorp/docker-vault/blob/e8edfef53deb6...

libeclipse 9y ago

To anyone reading this in the future, all of the issues raised in this thread have been handled in v0.2.0.