6 ms·

The fun thing about the Go compatibility promise is that it doesn't work as well as the go developers probably imagine; just last week one of my older Go projec

by zaarn 6y ago

The fun thing about the Go compatibility promise is that it doesn't work as well as the go developers probably imagine; just last week one of my older Go project was discovered to be broken; wouldn't compile anymore. golang.org/x/crypto was responsible, they used some code that wouldn't work in more modern go compilers anymore. Granted, I had not maintained the project in 3 years but I also had everything vendored just in case go decided to break anything upstream, and the broken parts didn't even touch on anything that the compatibility promise excludes. I think the Go devs should improve in that section a bit.

todotask 6y ago
One of those point under "Expectations" can be explain in this page https://golang.org/doc/go1compat https://golang.org/doc/go1compat
- zaarn 6y ago
  Nope, none of these points apply to the specific change, atleast as far as I am able to tell. The code that broke did not use unsafe when I last updated it and then it simply didn't work anymore in the current go version.
  - deleted 6y ago
    [deleted]
  - todotask 6y ago
    I agreed with jeffsmith82's comment.
    - zaarn 6y ago
      That comment isn't applicable either; I had x/crypto vendored so any fix upstream wouldn't break my app.
      - Aissen 6y ago
        I'm afraid blake2 used assembly and runtime.support_avx2, which doesn't seem to be covered by the compatibility promise: assembly is part of the toolchain, and the symbol it used was an internal one. See: https://go-review.googlesource.com/c/crypto/+/110355/ https://go-review.googlesource.com/c/crypto/+/110355/ https://go-review.googlesource.com/c/crypto/+/110016/ https://go-review.googlesource.com/c/crypto/+/110016/
        zaarn 6y ago
        So, my code broke because a library that was provided by the go developers used some unstable feature I had no idea about and had no real way of knowing short of checking a dependency three levels below my project? And to top that, that library was certainly subject to API breakage, the entire reason I vendored it in the first place. It doesn't seem reasonable to me. When I developed my project, vendoring /x/ projects was recommended exactly due to the breakage, so now it breaks my projects because of recommendations of the golang team.
  - TheDong 6y ago
    I believe one of those probably applies. > Use of package unsafe. Packages that import unsafe may depend on internal properties of the Go implementation. We reserve the right to make changes to the implementation that may break such programs. I suspect the crypto code you vendored used unsafe somewhere within it (whether to use `//go:linkname` or for other reasons), which would except it from the go1 compatibility promise.
jeffsmith82 6y ago
I had the same issue. The issue was a security one https://nvd.nist.gov/vuln/detail/CVE-2019-11840 https://nvd.nist.gov/vuln/detail/CVE-2019-11840 which required a breaking change to fix so their options where leave everyone vulnerable but could compile or break anyone that uses this code and fix it. I think they made the right choice. Also golang.org/x/crypto is not a core library so doesn't actually fall under the compatibility guarantee but they seem to try their best not to break it.
- zaarn 6y ago
  Neither of those things applied though; I had x/crypto vendored, so it didn't change it's code either and the CVE is unrelated to the actual code that broke (I used blake2b) and again; it was vendored, so my code wouldn't break but be vulnerable.
  - Scaevolus 6y ago
    How did vendored code that didn't change break your build?
    - zaarn 6y ago
      I updated the compiler.
    - TheDong 6y ago
      The easiest way to accomplish that is definitely with //go:linkname You can use a comment of the format '//go:linkname [localname] [method]' to link some function as another one. Notably, this includes unexported private methods in the go runtime and stdlib. I've seen code in the wild that uses this to grab go's map hash algorithm, get monotonic time, and other things. The go authors have used this hack themselves in the stdlib often enough because the runtime doesn't expose some knob, and rather than thinking that perhaps other developers may want that knob too, they instead use such hacks, but they also don't support said hacks (because how could you, obviously keeping all private functions stable forever is silly and impossible)
steeve 6y ago
I just compiled a fairly complex (CGo etc...) project a few weeks ago which I hadn't compiled since 2014, and it just worked. So I guess YMMV...
- zaarn 6y ago
  Of course, not every project will fail, but in my case it failed and I think that is the issue.
  - steeve 6y ago
    Do you remember the specifics? I am not aware of syntax constructs that were removed in Go, I guess ever. EDIT: I'm sad folks are downvoting for asking a question...
    - zaarn 6y ago
      Older versions of the /x/crypto library no longer compile on newer go versions, there wasn't any indication this would happen and I vendored the library to avoid the API breakage in that library. From my standpoint, that completely ruins using that library as after 3 years either I vendored it and it breaks because of something the go compiler does or I didn't vendor it but the API changed. Either way, my program will break eventually and there is no way to avoid it.
      - coldtea 6y ago
        Well, that's for the better still. If you use vulnerable crypto (willing to go with old unpatched vendored code) then you don't really need crypto at all..
        hibbelig 6y ago
        @zaarn explained in another comment that the vulnerability did not impact him, he did not exercise that code path.
deleted 6y ago
[deleted]
rat9988 6y ago
I'm pretty sure x packages are not part of go core.
- zaarn 6y ago
  x packages should still compile on newer go compilers in my opinion. Their compatibility isn't guaranteed if you don't vendor it, which was a recommendation when I started the project and as I pointed out in the comment you're responding to, it was vendored, so it didn't break itself.
FiloSottile 6y ago
That was golang.org/x/crypto's fault for using internals not covered by the compatibility promise [0], sorry about that. However, we try to hold x/crypto itself to the same compatibility standard as the standard library, so I think simply updating x/crypto at the same time should have made it a smooth update. [0]: https://news.ycombinator.com/item?id=23164795 https://news.ycombinator.com/item?id=23164795
scottlawson 6y ago
Isn't /x/ experimental by definition? They introduced a breaking changed to an experimental library. This does not fall under the scope of the compatibility promise. The compatibility promise works great.

by zaarn 6y ago

todotask 6y ago

One of those point under "Expectations" can be explain in this page https://golang.org/doc/go1compat https://golang.org/doc/go1compat

zaarn 6y ago
Nope, none of these points apply to the specific change, atleast as far as I am able to tell. The code that broke did not use unsafe when I last updated it and then it simply didn't work anymore in the current go version.
- deleted 6y ago
  [deleted]
- todotask 6y ago
  I agreed with jeffsmith82's comment.
  - zaarn 6y ago
    That comment isn't applicable either; I had x/crypto vendored so any fix upstream wouldn't break my app.
    - Aissen 6y ago
      I'm afraid blake2 used assembly and runtime.support_avx2, which doesn't seem to be covered by the compatibility promise: assembly is part of the toolchain, and the symbol it used was an internal one. See: https://go-review.googlesource.com/c/crypto/+/110355/ https://go-review.googlesource.com/c/crypto/+/110355/ https://go-review.googlesource.com/c/crypto/+/110016/ https://go-review.googlesource.com/c/crypto/+/110016/
      - zaarn 6y ago
        So, my code broke because a library that was provided by the go developers used some unstable feature I had no idea about and had no real way of knowing short of checking a dependency three levels below my project? And to top that, that library was certainly subject to API breakage, the entire reason I vendored it in the first place. It doesn't seem reasonable to me. When I developed my project, vendoring /x/ projects was recommended exactly due to the breakage, so now it breaks my projects because of recommendations of the golang team.
- TheDong 6y ago
  I believe one of those probably applies. > Use of package unsafe. Packages that import unsafe may depend on internal properties of the Go implementation. We reserve the right to make changes to the implementation that may break such programs. I suspect the crypto code you vendored used unsafe somewhere within it (whether to use `//go:linkname` or for other reasons), which would except it from the go1 compatibility promise.

jeffsmith82 6y ago

I had the same issue. The issue was a security one https://nvd.nist.gov/vuln/detail/CVE-2019-11840 https://nvd.nist.gov/vuln/detail/CVE-2019-11840 which required a breaking change to fix so their options where leave everyone vulnerable but could compile or break anyone that uses this code and fix it. I think they made the right choice. Also golang.org/x/crypto is not a core library so doesn't actually fall under the compatibility guarantee but they seem to try their best not to break it.

zaarn 6y ago
Neither of those things applied though; I had x/crypto vendored, so it didn't change it's code either and the CVE is unrelated to the actual code that broke (I used blake2b) and again; it was vendored, so my code wouldn't break but be vulnerable.
- Scaevolus 6y ago
  How did vendored code that didn't change break your build?
  - zaarn 6y ago
    I updated the compiler.
  - TheDong 6y ago
    The easiest way to accomplish that is definitely with //go:linkname You can use a comment of the format '//go:linkname [localname] [method]' to link some function as another one. Notably, this includes unexported private methods in the go runtime and stdlib. I've seen code in the wild that uses this to grab go's map hash algorithm, get monotonic time, and other things. The go authors have used this hack themselves in the stdlib often enough because the runtime doesn't expose some knob, and rather than thinking that perhaps other developers may want that knob too, they instead use such hacks, but they also don't support said hacks (because how could you, obviously keeping all private functions stable forever is silly and impossible)

steeve 6y ago

I just compiled a fairly complex (CGo etc...) project a few weeks ago which I hadn't compiled since 2014, and it just worked. So I guess YMMV...

zaarn 6y ago
Of course, not every project will fail, but in my case it failed and I think that is the issue.
- steeve 6y ago
  Do you remember the specifics? I am not aware of syntax constructs that were removed in Go, I guess ever. EDIT: I'm sad folks are downvoting for asking a question...
  - zaarn 6y ago
    Older versions of the /x/crypto library no longer compile on newer go versions, there wasn't any indication this would happen and I vendored the library to avoid the API breakage in that library. From my standpoint, that completely ruins using that library as after 3 years either I vendored it and it breaks because of something the go compiler does or I didn't vendor it but the API changed. Either way, my program will break eventually and there is no way to avoid it.
    - coldtea 6y ago
      Well, that's for the better still. If you use vulnerable crypto (willing to go with old unpatched vendored code) then you don't really need crypto at all..
      - hibbelig 6y ago
        @zaarn explained in another comment that the vulnerability did not impact him, he did not exercise that code path.

deleted 6y ago

[deleted]

rat9988 6y ago

I'm pretty sure x packages are not part of go core.

zaarn 6y ago
x packages should still compile on newer go compilers in my opinion. Their compatibility isn't guaranteed if you don't vendor it, which was a recommendation when I started the project and as I pointed out in the comment you're responding to, it was vendored, so it didn't break itself.

FiloSottile 6y ago

That was golang.org/x/crypto's fault for using internals not covered by the compatibility promise [0], sorry about that. However, we try to hold x/crypto itself to the same compatibility standard as the standard library, so I think simply updating x/crypto at the same time should have made it a smooth update. [0]: https://news.ycombinator.com/item?id=23164795 https://news.ycombinator.com/item?id=23164795

scottlawson 6y ago

Isn't /x/ experimental by definition? They introduced a breaking changed to an experimental library. This does not fall under the scope of the compatibility promise. The compatibility promise works great.