4 ms·

Hah, I wrote the crypto/rsa comments. We take Hyrum's Law (and backwards compatibility [1]) extremely seriously in Go. Here are a couple more examples: - We ra

Hah, I wrote the crypto/rsa comments. We take Hyrum's Law (and backwards compatibility [1]) extremely seriously in Go. Here are a couple more examples: - We randomly read an extra byte from random streams in various GenerateKey functions (which are not marked like the ones in OP) with MaybeReadByte [2] to avoid having our algorithm locked in - Just yesterday someone reported that a private ECDSA key with a nil public key used to work, and now it doesn't, so we probably have to make it work again [3] - Iterating over a map uses a randomized order to avoid exposing the internals - The output of rand.Rand is considered part of the compatibility promise, so we had to go to great lengths to improve it [4] - We discuss all the time what commitments to make in docs and what behaviors to disclaim, knowing we can never change something documented and probably something that's not explicitly documented as "this may change" [6] [1]: https://go.dev/doc/go1compat https://go.dev/doc/go1compat [2]: https://pkg.go.dev/crypto/internal/randutil#MaybeReadByte https://pkg.go.dev/crypto/internal/randutil#MaybeReadByte [3]: https://go.dev/issue/70468 https://go.dev/issue/70468 [4]: https://go.dev/blog/randv2 https://go.dev/blog/randv2 [5]: https://go.dev/blog/chacha8rand https://go.dev/blog/chacha8rand [6]: https://go-review.googlesource.com/c/go/+/598336/comment/5d6877aa_f45d6fa5/ https://go-review.googlesource.com/c/go/+/598336/comment/5d6...

mkesper 2y ago
The nil key case really makes me wonder how sane it is to support these cases. You will be forced to lug this broken behavior with you forever, like the infamous A20 line (https://en.wikipedia.org/wiki/A20_line https://en.wikipedia.org/wiki/A20_line).
- FiloSottile 2y ago
  > You will be forced to lug this broken behavior with you forever Yep, welcome to my life.
  - atsjie 2y ago
    Wouldn't that broken behaviour be a potential security issue by itself? I do remember Go making backwards incompatible changes in some rare scenarios like that. (and technically the loopvar fix was a big backwards incompatible change; granted that was done with a lot of consideration)
  - whizzter 2y ago
    Wouldn't a nil ECDSA key be a security risk?
    - unscaled 2y ago
      If a private key is available, the public key can be derived from the private key using scalar multiplication. This is how ecdsa.GenerateKey works by itself - it first generates a private key from the provided random byte stream and then derives a public key from that private key. I don't see how this can be a security risk, but allowing a public key that has a curve but a nil value is definitely a messy API.
gnfargbl 2y ago
As a user of your code this is true, and I'm very grateful indeed that you take this approach. I would add as a slight caveat that to benefit from this policy, users absolutely must read the release notes on major go versions before upgrading. We recently didn't, and we were burnt somewhat by the change to disallow negative serial numbers in the x509 parser without enabling the new feature flag. Completely our fault and not yours, but I add the caveat nevertheless.
- FiloSottile 2y ago
  We have gotten a liiiiittle more liberal ever since we introduced the new GODEBUG feature flag mechanism. I've been meaning to write a "how to safely update Go" post for a while, because the GODEBUG mechanism is very powerful but not well-known and we could build a bit of tooling around it. In short, you can upgrade your toolchain without changing the go.mod version, and these things will keep working like they did, and set a metric every time the behavior would have changed, but didn't. (Here's where we could build a bit of tooling to check that metric in prod/tests/CLIs more easily.) Then you can update the go.mod version, which updates the default set of GODEBUGs, and if anything breaks, try reverting GODEBUGs one by one.
  - gnfargbl 2y ago
    That sounds good. Breaking changes in major version updates is a completely normal thing in most software and we usually check for it. Ironically the only reason we weren't previously bothering in go is that the maintainers were historically so hyper-focused on absolute backwards compatibility that there were never any breaking changes!
boloust 2y ago
Ironically, I once wrote a load balancer in Go that relied on the randomized map iteration ordering.
- aleksi 2y ago
  That is a bad idea: https://dev.to/wallyqs/gos-map-iteration-order-is-not-that-random-mag https://dev.to/wallyqs/gos-map-iteration-order-is-not-that-r...
- OskarS 2y ago
  Man, you really can’t escape Hyrum’s Law ever! Now we have people depending on the iteration order being random!
  - ahoka 2y ago
    That's why it's totally stupid to randomize it.
  - dwattttt 2y ago
    Clearly you need to randomly decide whether or not to randomise it.
unscaled 2y ago
> We randomly read an extra byte from random streams in various GenerateKey functions (which are not marked like the ones in OP) with MaybeReadByte [2] to avoid having our algorithm locked in You don't seem to do that in ed25519. Back before ed25519.NewKeyFromSeed() existed, that was the only way to derive a public Ed25519 key from a private key, and I'm pretty sure I've written code that relied on that (it's easy to remember, since I wasn't very happy about it, but this was all I could do). The documentation of ed25519.GenerateKey mentions that the output is deterministic, so kudos for that. It seems you've really done a great job with investigating and maintaining ossified behavior in the Go cryptography APIs and preventing new ones from happening.
mjw_byrne 2y ago
The map iteration order change helps to avoid breaking changes in future, by preventing reliance on any specific ordering, but when the change was made it was breaking for anything that was relying on the previous ordering behaviour. IMO this is a worthwhile tradeoff. I use Go a lot and love the strong backwards compatibility, but I would happily accept a (slightly) higher rate of breaking changes if it meant greater freedom for the Go devs to improve performance, add features etc. Based on the kind of hell users of other ecosystems seem willing to tolerate (cough Python cough), I believe I am not alone in this viewpoint.
- wild_egg 2y ago
  Data point of one, but I've been using Go since 2012 and would drop it instantly if any of the backwards compatibility guarantees were relaxed. Having bugs imposed on you from outside your project is a waste of time to deal with and there are dozens of other languages you can pick from if you enjoy that time sink. Most of them give you greater capabilities as the balance. Go's stability is a core feature and compensates for the lack of other niceties. Adding features isn't a good reason to break things. I can go use something else if I want to make that trade.
  - otterley 2y ago
    Respectfully, I don’t think you would just pack up and leave. The cost of switching to an entirely different language—which might have even worse backwards compatibility issues—is significantly higher than fixing bugs you inadvertently introduced due to prior invalid assumptions. I’d call your bluff.
    - hn34381 2y ago
      Also, there is a time and a place for things. Breaking API changes in a minor version update sucks and is often an unexpected time sink, and often mandatory because it has some security patch, critical bug fix, or something. Breaking API changes in a major version update is expected, can be planned for, and often can be delayed if one chooses.
    - 2y ago
abtinf 2y ago
This is one of the least appreciated aspects of Go. Code I wrote 12 years ago still just works.

mkesper 2y ago

The nil key case really makes me wonder how sane it is to support these cases. You will be forced to lug this broken behavior with you forever, like the infamous A20 line (https://en.wikipedia.org/wiki/A20_line https://en.wikipedia.org/wiki/A20_line).

FiloSottile 2y ago
> You will be forced to lug this broken behavior with you forever Yep, welcome to my life.
- atsjie 2y ago
  Wouldn't that broken behaviour be a potential security issue by itself? I do remember Go making backwards incompatible changes in some rare scenarios like that. (and technically the loopvar fix was a big backwards incompatible change; granted that was done with a lot of consideration)
- whizzter 2y ago
  Wouldn't a nil ECDSA key be a security risk?
  - unscaled 2y ago
    If a private key is available, the public key can be derived from the private key using scalar multiplication. This is how ecdsa.GenerateKey works by itself - it first generates a private key from the provided random byte stream and then derives a public key from that private key. I don't see how this can be a security risk, but allowing a public key that has a curve but a nil value is definitely a messy API.

gnfargbl 2y ago

As a user of your code this is true, and I'm very grateful indeed that you take this approach. I would add as a slight caveat that to benefit from this policy, users absolutely must read the release notes on major go versions before upgrading. We recently didn't, and we were burnt somewhat by the change to disallow negative serial numbers in the x509 parser without enabling the new feature flag. Completely our fault and not yours, but I add the caveat nevertheless.

FiloSottile 2y ago
We have gotten a liiiiittle more liberal ever since we introduced the new GODEBUG feature flag mechanism. I've been meaning to write a "how to safely update Go" post for a while, because the GODEBUG mechanism is very powerful but not well-known and we could build a bit of tooling around it. In short, you can upgrade your toolchain without changing the go.mod version, and these things will keep working like they did, and set a metric every time the behavior would have changed, but didn't. (Here's where we could build a bit of tooling to check that metric in prod/tests/CLIs more easily.) Then you can update the go.mod version, which updates the default set of GODEBUGs, and if anything breaks, try reverting GODEBUGs one by one.
- gnfargbl 2y ago
  That sounds good. Breaking changes in major version updates is a completely normal thing in most software and we usually check for it. Ironically the only reason we weren't previously bothering in go is that the maintainers were historically so hyper-focused on absolute backwards compatibility that there were never any breaking changes!

boloust 2y ago

Ironically, I once wrote a load balancer in Go that relied on the randomized map iteration ordering.

aleksi 2y ago
That is a bad idea: https://dev.to/wallyqs/gos-map-iteration-order-is-not-that-random-mag https://dev.to/wallyqs/gos-map-iteration-order-is-not-that-r...
OskarS 2y ago
Man, you really can’t escape Hyrum’s Law ever! Now we have people depending on the iteration order being random!
- ahoka 2y ago
  That's why it's totally stupid to randomize it.
- dwattttt 2y ago
  Clearly you need to randomly decide whether or not to randomise it.

unscaled 2y ago

> We randomly read an extra byte from random streams in various GenerateKey functions (which are not marked like the ones in OP) with MaybeReadByte [2] to avoid having our algorithm locked in You don't seem to do that in ed25519. Back before ed25519.NewKeyFromSeed() existed, that was the only way to derive a public Ed25519 key from a private key, and I'm pretty sure I've written code that relied on that (it's easy to remember, since I wasn't very happy about it, but this was all I could do). The documentation of ed25519.GenerateKey mentions that the output is deterministic, so kudos for that. It seems you've really done a great job with investigating and maintaining ossified behavior in the Go cryptography APIs and preventing new ones from happening.

mjw_byrne 2y ago

The map iteration order change helps to avoid breaking changes in future, by preventing reliance on any specific ordering, but when the change was made it was breaking for anything that was relying on the previous ordering behaviour. IMO this is a worthwhile tradeoff. I use Go a lot and love the strong backwards compatibility, but I would happily accept a (slightly) higher rate of breaking changes if it meant greater freedom for the Go devs to improve performance, add features etc. Based on the kind of hell users of other ecosystems seem willing to tolerate (cough Python cough), I believe I am not alone in this viewpoint.

wild_egg 2y ago
Data point of one, but I've been using Go since 2012 and would drop it instantly if any of the backwards compatibility guarantees were relaxed. Having bugs imposed on you from outside your project is a waste of time to deal with and there are dozens of other languages you can pick from if you enjoy that time sink. Most of them give you greater capabilities as the balance. Go's stability is a core feature and compensates for the lack of other niceties. Adding features isn't a good reason to break things. I can go use something else if I want to make that trade.
- otterley 2y ago
  Respectfully, I don’t think you would just pack up and leave. The cost of switching to an entirely different language—which might have even worse backwards compatibility issues—is significantly higher than fixing bugs you inadvertently introduced due to prior invalid assumptions. I’d call your bluff.
  - hn34381 2y ago
    Also, there is a time and a place for things. Breaking API changes in a minor version update sucks and is often an unexpected time sink, and often mandatory because it has some security patch, critical bug fix, or something. Breaking API changes in a major version update is expected, can be planned for, and often can be delayed if one chooses.
  - 2y ago

abtinf 2y ago

This is one of the least appreciated aspects of Go. Code I wrote 12 years ago still just works.