9 ms·

What's Coming in Go 1.15

151 points by ra7 6y ago

yegle 6y ago
I was surprised to see ASLR was not enabled by default on Windows. Is enabling ASLR by default a common thing on Windows?
- mappu 6y ago
  It is commonly enabled for C++ compilers, but ASLR defends against attacks that Go code is not generally vulnerable to.
  - zaarn 6y ago
    I'm not sure if that is accurate, ASLR defends against quite a range of attacks that are the result of code execution, which AFAIK can still occur in Go, Rowhammer comes to mind here as a universal crack against languages. There is no downside to activating ASLR either, so I'm not sure why it isn't.
    - swiley 6y ago
      Does addr2line still work with ASLR? Because that could be a reason not to.
      - zaarn 6y ago
        addr2line works depending on the code and compiler/libc. Your libunwind may be able to output relativ code addresses, in which case addr2line will work if you use the stacktrace. GDB should handle this as well. If your c compiler doesn't support it you can just disable compiling PIEs which automatically disables ASLR but that should be reserved for debug builds.
greendave 6y ago
> Michael Knysze significantly increased throughput of memory allocation for large blocks by redesigning the memory allocator's "mcentral" data structure to reduce lock contention. The new allocation code is more than twice as fast for blocks of 12KB or larger. Excellent! Nice to see these sorts of infrastructure enhancements.
hactually 6y ago
A good presentation did the rounds a while ago[0] that does a deeper dive but the main things for us are the performance, smaller binaries and the improvements in tooling. The one thing I wish they added though? `go new` and have it initialise a new project structure that stopped tiny package syndrome and bizarre Makefiles in repos! [1] [0] https://docs.google.com/presentation/d/1veyF0y6Ynr6AFzd9gXi4foaURlgbMxM-tmB4StDrdAM/edit#slide=id.g57812b4045_0_66 https://docs.google.com/presentation/d/1veyF0y6Ynr6AFzd9gXi4... [1] https://rakyll.org/style-packages/ https://rakyll.org/style-packages/
- benhoyt 6y ago
  Re "go new": I think part of the problem with that is that people have quite different ideas of what a good project structure is. It also varies a lot between a simple CLI tool (for which main.go and maybe go.mod is enough) and a large multi-package server project. Can you explain what you mean by "bizarre Makefiles"? I've never used Makefiles for my (admittedly fairly small) Go projects, because "go build" just works.
  - hactually 6y ago
    I've seen a number of projects where the root directory is a load of other folders and then there's a Makefile and it's a kafkaesque nightmare. I agree on the toplevel `main.go` vs the `./cmd/myapp/main.go` style needing catering for that only comes out as the project evolves... maybe it should be a go vet addition instead :D
    - stubish 6y ago
      Go modules are removing the need for much of the Makefile magic. You still have use of a Makefile if you need vendoring or are dealing with non-public branches, but things can be much simpler.
- mvdan 6y ago
  I'm the author of that presentation, and I'm surprised to see the very similar content and structure, to say the least. I left a comment on the Reddit thread[0] to see if I'm missing something. [0] https://www.reddit.com/r/golang/comments/ginp22/whats_coming_in_go_115/fqh7ynj/ https://www.reddit.com/r/golang/comments/ginp22/whats_coming...
jeffdavis 6y ago
What are some examples of medium-large Go projects where the code is well-written? By that, I mean code that would make a good example for someone trying to learn the "right" way to do things. I was briefly learning Go a while back. I'm curious how well gorotuines and channels have held up as building blocks, and how extensively they are used in real projects.
- benhoyt 6y ago
  These days when I see a fast, reliable CLI tool I think "this has got to be written in Go" and a lot of the time they are. Some examples: the new GitHub CLI, Docker, Hashicorp tools, ngrok (though v2 is not open source), Caddy, the list is quite long. I can't vouch for the quality of that code, but I suspect some of those projects are good starting points. And I've found the Go standard library source to be really helpful (and generally fairly easy to understand).
  - megameter 6y ago
    Speaking from some experience, Go is probably the best option these days at "do one thing well" command line tools - the mix of library support, native binaries, and the simple, stable language make it a good fit for a weekend project that opens files, munges strings and pushes them over standard protocols. When Go has to support a large application that defines a platform in itself, the downsides of the language rise to the forefront, but that level of expression usually isn't entailed in a Unix-style CLI tool. (Rust also has more potential for performance scaling in this space, but with the "if you have the time for that" kind of caveat.)
    - majewsky 6y ago
      > Rust also has more potential for performance scaling in this space, but with the "if you have the time for that" kind of caveat. When you have a CLI for some network-based service, the network latency is going to be magnitudes larger than the performance differences between Go and Rust anyway. (Or at least in 99% of cases.)
3fe9a03ccd14ca5 6y ago
> A new time/tzdata package was added to allow embedding a static copy of the time zone database in executables. This is a BIG improvement. One of the big benefits of Go is static binaries. Having the TZ data includes will make deploying even easier.
- jfkebwjsbx 6y ago
  What do you mean by a benefit? Most compiled/native languages can make static binaries.
  - TheDong 6y ago
    To paraphrase rob pike incorrectly from an only mildly misleading quote, "The key point here is our programmers are not researchers. They learned java, python, javascript. They're not capable of understanding a brilliant language, so we built go". The comparison isn't other compiled languages that can easily produce static binaries (like ghc/haskell, rust, c++ (with about 2000 lines of meson or cmake or makefile, ...), etc). The comparison is other languages Go programmers might use, like python and javascript. If someone cared about having a good type system, they wouldn't be using Go, so we have to compare Go's capabilities to other languages that are targeted at people who don't want any brilliance or type-theory, but just want a "simple language" where it's accepted as part of the "programmer condition" that you write bad code and get things done. However, in reality, it's worth pointing out that there are very few other languages that have as seamless a static linking experience as Go. Sure, it's quite possible to statically link C(++), rust, etc, but it's more involved than it is for go. Furthermore, for those languages, there's a larger library ecosystem that dynamically depends on C libraries, so i.e. in rust you may have trouble linking any crate that depends on openssl (which quite a few do transitively). Go is the only language I know of that's created an entire culture of eschewing the C ABI as much as possible, that uses linux's syscall interface even though that means there are various things that are just plain broken (like nsswitch). So yeah, while it's true that go isn't the only language that can be statically linked, it's one of the few that focuses on it to such an (arguably unhealthy) extreme.
    - 6y ago
deleted 6y ago
[deleted]
FiloSottile 6y ago
I'm also pretty happy about what landed in the cryptography packages. Session ticket keys, which are a major weak link in Forward Secrecy in TLS 1.0-1.2 [0], are now rotated automatically every day, and dropped after 7 days [1]. Session keys are also dropped once they get too old [2], instead of becoming an increasingly powerful Forward Secrecy liability. The tls.Config.VerifyConnection callback is a much more powerful and usable way to customize peer verification [3]. X.509 root CAs are now extracted on macOS by linking directly against Security.framework with some assembly glue, instead of using cgo [4]. (Warning: awakens Cthulhu.) crypto/x509 now ignores the 20-year-deprecated Common Name field by default [5] and has a consistent rule about invalid hostnames [6]. crypto/elliptic now provides MarshalCompressed and UnmarshalCompressed for compressed point encodings [7]. The clever TLS 1.3 downgrade canary in the server random is now enforced client side [8]. PrivateKey and PublicKey types implement Equal, giving a consistent method to build type-safe interfaces for them [9]. math/big has a new Int.FillBytes method for fixed size, zero allocation byte serialization [9], which we used all over the crypto packages. RevocationList and CreateRevocationList were added to crypto/x509 to generate spec compliant CRLs [11]. crypto/ecdsa has two new functions, SignASN1 and VerifyASN1 that operate on encoded signatures instead of big.Ints that everyone was encoding/decoding anyway [12]. Plus some performance, cosmetic, correctness, and documentation improvements! Not a bad cycle overall, even if it made sense to focus on smaller scope tasks for a while. [0]: https://blog.filippo.io/we-need-to-talk-about-session-tickets/ https://blog.filippo.io/we-need-to-talk-about-session-ticket... [1]: https://go.googlesource.com/go/+/43f2f5024b https://go.googlesource.com/go/+/43f2f5024b [2]: https://go.googlesource.com/go/+/6ea19bb668 https://go.googlesource.com/go/+/6ea19bb668 [3]: https://go.googlesource.com/go/+/62a3f2e27c https://go.googlesource.com/go/+/62a3f2e27c [4]: https://go.googlesource.com/go/+/6f52790a20 https://go.googlesource.com/go/+/6f52790a20 [5]: https://go.googlesource.com/go/+/d65e1b2e41 https://go.googlesource.com/go/+/d65e1b2e41 [6]: https://go.googlesource.com/go/+/9d1e120c42 https://go.googlesource.com/go/+/9d1e120c42 [7]: https://go.googlesource.com/go/+/5c13cab36b https://go.googlesource.com/go/+/5c13cab36b [8]: https://go.googlesource.com/go/+/a6c6e59655 https://go.googlesource.com/go/+/a6c6e59655 [9]: https://go.googlesource.com/go/+/b5f2c0f502 https://go.googlesource.com/go/+/b5f2c0f502 [10]: https://go.googlesource.com/go/+/c9d5f60eaa https://go.googlesource.com/go/+/c9d5f60eaa [11]: https://go.googlesource.com/go/+/5d47f870a6 https://go.googlesource.com/go/+/5d47f870a6 [12]: https://go.googlesource.com/go/+/8c09e8af36 https://go.googlesource.com/go/+/8c09e8af36
mholt 6y ago
This is a well-written article for lwn subscribers, by HN's very own @benhoyt. If you read lwn, I definitely recommend subscribing; it consistently has high-quality technical content.
EE84M3i 6y ago
>On Windows, Go 1.15 now generates executables that use address-space layout randomization (ASLR) by default. ASLR uses position-independent code to randomize the addresses of various data areas on startup, making it harder for attackers to predict target addresses and create memory-corruption exploits. Impressive this is only landing for Windows now after PPP made a spectacle of the lack of ASLR back in 2013. http://pwning.net/2013/04/21/exploiting-a-go-binary/ http://pwning.net/2013/04/21/exploiting-a-go-binary/
- zxcmx 6y ago
  Ehh I agree with you but I also haven't seen an "actual" exploit for a go memory corruption vulnerability outside of POCs. Like I know it's possible with effort and maybe it was just lack of interest or adoption (like lets be fair its not a browser writing or pdf rendering engine type of language...) so while that situation was not great it never struck me as a burning problem with practical impact in the ecosystem either. Interested in any pointers if possible.
zaarn 6y ago
The fun thing about the Go compatibility promise is that it doesn't work as well as the go developers probably imagine; just last week one of my older Go project was discovered to be broken; wouldn't compile anymore. golang.org/x/crypto was responsible, they used some code that wouldn't work in more modern go compilers anymore. Granted, I had not maintained the project in 3 years but I also had everything vendored just in case go decided to break anything upstream, and the broken parts didn't even touch on anything that the compatibility promise excludes. I think the Go devs should improve in that section a bit.
- todotask 6y ago
  One of those point under "Expectations" can be explain in this page https://golang.org/doc/go1compat https://golang.org/doc/go1compat
  - zaarn 6y ago
    Nope, none of these points apply to the specific change, atleast as far as I am able to tell. The code that broke did not use unsafe when I last updated it and then it simply didn't work anymore in the current go version.
    - deleted 6y ago
      [deleted]
    - todotask 6y ago
      I agreed with jeffsmith82's comment.
      - zaarn 6y ago
        That comment isn't applicable either; I had x/crypto vendored so any fix upstream wouldn't break my app.
        Aissen 6y ago
sime2009 6y ago
Speaking of the future, does anyone here have an idea as to the the time frame when generics could land in a release?
- phwak 6y ago
  An year or two.

151 points by ra7 6y ago

I was surprised to see ASLR was not enabled by default on Windows. Is enabling ASLR by default a common thing on Windows?

mappu 6y ago
It is commonly enabled for C++ compilers, but ASLR defends against attacks that Go code is not generally vulnerable to.
- zaarn 6y ago
  I'm not sure if that is accurate, ASLR defends against quite a range of attacks that are the result of code execution, which AFAIK can still occur in Go, Rowhammer comes to mind here as a universal crack against languages. There is no downside to activating ASLR either, so I'm not sure why it isn't.
  - swiley 6y ago
    Does addr2line still work with ASLR? Because that could be a reason not to.
    - zaarn 6y ago
      addr2line works depending on the code and compiler/libc. Your libunwind may be able to output relativ code addresses, in which case addr2line will work if you use the stacktrace. GDB should handle this as well. If your c compiler doesn't support it you can just disable compiling PIEs which automatically disables ASLR but that should be reserved for debug builds.

greendave 6y ago

> Michael Knysze significantly increased throughput of memory allocation for large blocks by redesigning the memory allocator's "mcentral" data structure to reduce lock contention. The new allocation code is more than twice as fast for blocks of 12KB or larger. Excellent! Nice to see these sorts of infrastructure enhancements.

hactually 6y ago

A good presentation did the rounds a while ago[0] that does a deeper dive but the main things for us are the performance, smaller binaries and the improvements in tooling. The one thing I wish they added though? `go new` and have it initialise a new project structure that stopped tiny package syndrome and bizarre Makefiles in repos! [1] [0] https://docs.google.com/presentation/d/1veyF0y6Ynr6AFzd9gXi4foaURlgbMxM-tmB4StDrdAM/edit#slide=id.g57812b4045_0_66 https://docs.google.com/presentation/d/1veyF0y6Ynr6AFzd9gXi4... [1] https://rakyll.org/style-packages/ https://rakyll.org/style-packages/

benhoyt 6y ago
Re "go new": I think part of the problem with that is that people have quite different ideas of what a good project structure is. It also varies a lot between a simple CLI tool (for which main.go and maybe go.mod is enough) and a large multi-package server project. Can you explain what you mean by "bizarre Makefiles"? I've never used Makefiles for my (admittedly fairly small) Go projects, because "go build" just works.
- hactually 6y ago
  I've seen a number of projects where the root directory is a load of other folders and then there's a Makefile and it's a kafkaesque nightmare. I agree on the toplevel `main.go` vs the `./cmd/myapp/main.go` style needing catering for that only comes out as the project evolves... maybe it should be a go vet addition instead :D
  - stubish 6y ago
    Go modules are removing the need for much of the Makefile magic. You still have use of a Makefile if you need vendoring or are dealing with non-public branches, but things can be much simpler.
mvdan 6y ago
I'm the author of that presentation, and I'm surprised to see the very similar content and structure, to say the least. I left a comment on the Reddit thread[0] to see if I'm missing something. [0] https://www.reddit.com/r/golang/comments/ginp22/whats_coming_in_go_115/fqh7ynj/ https://www.reddit.com/r/golang/comments/ginp22/whats_coming...

jeffdavis 6y ago

What are some examples of medium-large Go projects where the code is well-written? By that, I mean code that would make a good example for someone trying to learn the "right" way to do things. I was briefly learning Go a while back. I'm curious how well gorotuines and channels have held up as building blocks, and how extensively they are used in real projects.

benhoyt 6y ago
These days when I see a fast, reliable CLI tool I think "this has got to be written in Go" and a lot of the time they are. Some examples: the new GitHub CLI, Docker, Hashicorp tools, ngrok (though v2 is not open source), Caddy, the list is quite long. I can't vouch for the quality of that code, but I suspect some of those projects are good starting points. And I've found the Go standard library source to be really helpful (and generally fairly easy to understand).
- megameter 6y ago
  Speaking from some experience, Go is probably the best option these days at "do one thing well" command line tools - the mix of library support, native binaries, and the simple, stable language make it a good fit for a weekend project that opens files, munges strings and pushes them over standard protocols. When Go has to support a large application that defines a platform in itself, the downsides of the language rise to the forefront, but that level of expression usually isn't entailed in a Unix-style CLI tool. (Rust also has more potential for performance scaling in this space, but with the "if you have the time for that" kind of caveat.)
  - majewsky 6y ago
    > Rust also has more potential for performance scaling in this space, but with the "if you have the time for that" kind of caveat. When you have a CLI for some network-based service, the network latency is going to be magnitudes larger than the performance differences between Go and Rust anyway. (Or at least in 99% of cases.)

3fe9a03ccd14ca5 6y ago

> A new time/tzdata package was added to allow embedding a static copy of the time zone database in executables. This is a BIG improvement. One of the big benefits of Go is static binaries. Having the TZ data includes will make deploying even easier.

jfkebwjsbx 6y ago
What do you mean by a benefit? Most compiled/native languages can make static binaries.
- TheDong 6y ago
  To paraphrase rob pike incorrectly from an only mildly misleading quote, "The key point here is our programmers are not researchers. They learned java, python, javascript. They're not capable of understanding a brilliant language, so we built go". The comparison isn't other compiled languages that can easily produce static binaries (like ghc/haskell, rust, c++ (with about 2000 lines of meson or cmake or makefile, ...), etc). The comparison is other languages Go programmers might use, like python and javascript. If someone cared about having a good type system, they wouldn't be using Go, so we have to compare Go's capabilities to other languages that are targeted at people who don't want any brilliance or type-theory, but just want a "simple language" where it's accepted as part of the "programmer condition" that you write bad code and get things done. However, in reality, it's worth pointing out that there are very few other languages that have as seamless a static linking experience as Go. Sure, it's quite possible to statically link C(++), rust, etc, but it's more involved than it is for go. Furthermore, for those languages, there's a larger library ecosystem that dynamically depends on C libraries, so i.e. in rust you may have trouble linking any crate that depends on openssl (which quite a few do transitively). Go is the only language I know of that's created an entire culture of eschewing the C ABI as much as possible, that uses linux's syscall interface even though that means there are various things that are just plain broken (like nsswitch). So yeah, while it's true that go isn't the only language that can be statically linked, it's one of the few that focuses on it to such an (arguably unhealthy) extreme.
  - 6y ago

deleted 6y ago

[deleted]

FiloSottile 6y ago

I'm also pretty happy about what landed in the cryptography packages. Session ticket keys, which are a major weak link in Forward Secrecy in TLS 1.0-1.2 [0], are now rotated automatically every day, and dropped after 7 days [1]. Session keys are also dropped once they get too old [2], instead of becoming an increasingly powerful Forward Secrecy liability. The tls.Config.VerifyConnection callback is a much more powerful and usable way to customize peer verification [3]. X.509 root CAs are now extracted on macOS by linking directly against Security.framework with some assembly glue, instead of using cgo [4]. (Warning: awakens Cthulhu.) crypto/x509 now ignores the 20-year-deprecated Common Name field by default [5] and has a consistent rule about invalid hostnames [6]. crypto/elliptic now provides MarshalCompressed and UnmarshalCompressed for compressed point encodings [7]. The clever TLS 1.3 downgrade canary in the server random is now enforced client side [8]. PrivateKey and PublicKey types implement Equal, giving a consistent method to build type-safe interfaces for them [9]. math/big has a new Int.FillBytes method for fixed size, zero allocation byte serialization [9], which we used all over the crypto packages. RevocationList and CreateRevocationList were added to crypto/x509 to generate spec compliant CRLs [11]. crypto/ecdsa has two new functions, SignASN1 and VerifyASN1 that operate on encoded signatures instead of big.Ints that everyone was encoding/decoding anyway [12]. Plus some performance, cosmetic, correctness, and documentation improvements! Not a bad cycle overall, even if it made sense to focus on smaller scope tasks for a while. [0]: https://blog.filippo.io/we-need-to-talk-about-session-tickets/ https://blog.filippo.io/we-need-to-talk-about-session-ticket... [1]: https://go.googlesource.com/go/+/43f2f5024b https://go.googlesource.com/go/+/43f2f5024b [2]: https://go.googlesource.com/go/+/6ea19bb668 https://go.googlesource.com/go/+/6ea19bb668 [3]: https://go.googlesource.com/go/+/62a3f2e27c https://go.googlesource.com/go/+/62a3f2e27c [4]: https://go.googlesource.com/go/+/6f52790a20 https://go.googlesource.com/go/+/6f52790a20 [5]: https://go.googlesource.com/go/+/d65e1b2e41 https://go.googlesource.com/go/+/d65e1b2e41 [6]: https://go.googlesource.com/go/+/9d1e120c42 https://go.googlesource.com/go/+/9d1e120c42 [7]: https://go.googlesource.com/go/+/5c13cab36b https://go.googlesource.com/go/+/5c13cab36b [8]: https://go.googlesource.com/go/+/a6c6e59655 https://go.googlesource.com/go/+/a6c6e59655 [9]: https://go.googlesource.com/go/+/b5f2c0f502 https://go.googlesource.com/go/+/b5f2c0f502 [10]: https://go.googlesource.com/go/+/c9d5f60eaa https://go.googlesource.com/go/+/c9d5f60eaa [11]: https://go.googlesource.com/go/+/5d47f870a6 https://go.googlesource.com/go/+/5d47f870a6 [12]: https://go.googlesource.com/go/+/8c09e8af36 https://go.googlesource.com/go/+/8c09e8af36

mholt 6y ago

This is a well-written article for lwn subscribers, by HN's very own @benhoyt. If you read lwn, I definitely recommend subscribing; it consistently has high-quality technical content.

EE84M3i 6y ago

>On Windows, Go 1.15 now generates executables that use address-space layout randomization (ASLR) by default. ASLR uses position-independent code to randomize the addresses of various data areas on startup, making it harder for attackers to predict target addresses and create memory-corruption exploits. Impressive this is only landing for Windows now after PPP made a spectacle of the lack of ASLR back in 2013. http://pwning.net/2013/04/21/exploiting-a-go-binary/ http://pwning.net/2013/04/21/exploiting-a-go-binary/

zxcmx 6y ago
Ehh I agree with you but I also haven't seen an "actual" exploit for a go memory corruption vulnerability outside of POCs. Like I know it's possible with effort and maybe it was just lack of interest or adoption (like lets be fair its not a browser writing or pdf rendering engine type of language...) so while that situation was not great it never struck me as a burning problem with practical impact in the ecosystem either. Interested in any pointers if possible.

zaarn 6y ago

The fun thing about the Go compatibility promise is that it doesn't work as well as the go developers probably imagine; just last week one of my older Go project was discovered to be broken; wouldn't compile anymore. golang.org/x/crypto was responsible, they used some code that wouldn't work in more modern go compilers anymore. Granted, I had not maintained the project in 3 years but I also had everything vendored just in case go decided to break anything upstream, and the broken parts didn't even touch on anything that the compatibility promise excludes. I think the Go devs should improve in that section a bit.

todotask 6y ago
One of those point under "Expectations" can be explain in this page https://golang.org/doc/go1compat https://golang.org/doc/go1compat
- zaarn 6y ago
  Nope, none of these points apply to the specific change, atleast as far as I am able to tell. The code that broke did not use unsafe when I last updated it and then it simply didn't work anymore in the current go version.
  - deleted 6y ago
    [deleted]
  - todotask 6y ago
    I agreed with jeffsmith82's comment.
    - zaarn 6y ago
      That comment isn't applicable either; I had x/crypto vendored so any fix upstream wouldn't break my app.
      - Aissen 6y ago

sime2009 6y ago

Speaking of the future, does anyone here have an idea as to the the time frame when generics could land in a release?

phwak 6y ago
An year or two.