11 ms·
Go crypto: bridging the performance gap
- 4ad 12y agoI will happily look over these once they are part of upstream Go, or I will happily review them once they are proposed upstream, but I will take a pass on their "special fork of Go".
- aikah 12y agoI'm curious whether the Go team will merge Go crypto into the core or not. It will show how open they are to third party contributions.
- 4ad 12y agoYour comment is significantly out of place. The Go team is extremely open to quality external contributors. 463 people have contributed to Go so far, (obviously) most of them outside the core Go team. The Windows port was exclusively done and maintained by contributors. I have done the arm64 Go compiler, which is upstream now, the Solaris port, and I am now doing the sparc64 compiler. Many 3rd party contributors do many things every day. The Go project is an extremely open project. More than half of the people who have direct commit access are external contributors.
- deleted 12y ago[deleted]
- coldtea 12y ago>Your comment is significantly out of place. The Go team is extremely open to quality external contributors. I think his comment is still valid. Adapting something major like this is not the same as accepting bugfixes from hundrends of people, or ports to a different architecture. Even more different would be accepting some code for the standard library whose API wasn't designed by the core team. From what I've seen the core team is quite opinionated and micro-managing things.
- nickcw 12y agoThe go team was quite happy to accept my (individual; non core developer) contributions of md5 and sha1 written in ARM assembler given the appropriate review, so I don't see why Cloudflare's contribution should be any different.
- DannyBee 12y agowell, for starters, to accept it, the licensing would need to be changed slightly. Parts have +// Copyright 2015 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Copyright 2015 Intel Corporation +// Copyright 2015 CloudFlare, Inc. + +// This file contains constant-time, 64-bit assembly implementation of +// P256. The optimizations performed here are described in detail in: +// S.Gueron and V.Krasnov, "Fast prime field elliptic-curve cryptography with +// 256-bit primes" +" The additional copyright notices would not be okay. They would cause everyone who uses this library to have to reproduce not just the standard go copyright notices, but those, too.
- twotwotwo 12y agoThey seem OK to me. A quick grep of my Go tree turns up lots of third-party copyright notices. It's OK as long as their code was released under licenses compatible with Go's. These additional copyright notices could be removed by CloudFlare and Intel going in the AUTHORS file (which defines "The Go Authors"), presumably after they and Google do any required paperwork. Red Hat, Dropbox, and Fastly are in AUTHORS. But that needn't be a condition of integrating their code as long as it's licensed properly. The paper citation doesn't appear to be copyright-related, and others like it are sprinkled around the codebase, e.g., package sort's source cites some papers on efficient sorting.
- DannyBee 12y ago"They seem OK to me. A quick grep of my Go tree turns up lots of third-party copyright notices. It's OK as long as their code was released under licenses compatible with Go's." Okay, let me rephrase: "they aren't okay". It's actually my job to make these decisions and tell teams what is and what isn't okay :) The other issues you mention are in the process of being fixed. "These additional copyright notices could be removed by CloudFlare and Intel going in the AUTHORS file (which defines "The Go Authors")" Yes, they could, but that requires agreement from more than just cloudfare. This is code Intel donated to openssl, not to Go, so it's simply not as trivial as cloudfare saying "sure, here's some code". Intel has to agree to have their copyright notice changed, etc. "The paper citation doesn't appear to be copyright-related, and others like it are sprinkled around the codebase, e.g., package sort's source cites some papers on efficient sorting. " I have no care in the world about this part.
- CyberShadow 12y agoI don't understand, why reimplement this if it is already implemented with adequate performance/quality in OpenSSL? I thought Go didn't have issues calling C code?
- 4ad 12y agoCalling C code from Go is something you do as a last resort. It breaks some desirable properties of Go programming, like fast compilation, cross compiling (at least without effort), and static binaries (without serious effort). Code called through cgo will never be idiomatic Go code, unless perhaps if hidden under an abstraction layer which takes time and effort to write and adds overhead. On top of that, the cgo overhead is significant, not only in time, but in threads, so you'd want it only for big chunks of data and program accounting for it. Another advantage of Go crypto is having been written in a memory safe language, and with runtime bounds checking. The assembly fast paths break some of this, but not most of it.
- andrewchambers 12y agoGo has moving memory so it generally means calling C requires a bunch of copying. This is slow and annoying.
- 1_player 12y ago> Go has moving memory so it generally means calling C SHOULD/WILL require a bunch of copying. I've been working on a few Go frontend to C libraries and researching what's the exact memory model when calling C: Yes, Go has moving memory and you _shouldn't_ pass a Go pointer to C, and it's actively discouraged [1], but since many libraries just ignore that advice [2] for performance reason Go isn't currently AFAIK not moving any pointer passed through cgo. That will probably be changed in Go 1.5+, I reckon an official post from the Go developer about the current and future state of C/Go memory interaction would help clarify this. 1: https://github.com/golang/go/issues/8310 https://github.com/golang/go/issues/8310 2: comment #31 of https://github.com/golang/go/issues/8310 https://github.com/golang/go/issues/8310
- robmccoll 12y ago
- travjones 12y agoIt's great to see that companies can turn a profit AND contribute to open source projects that benefit the web as a whole. Thanks, Cloudflare.
- michaelt 12y agoCloudflare's Universal SSL is pretty great - I used to host my static website on S3, but that means no SSL or ipv6. Putting everything through cloudflare sorts both those out. The original announcement [1] mentioned they were planning support for adding in the HSTS header - as jgrahamc is here responding to comments, I'd be interested to hear how far they've got with that :) [1] https://blog.cloudflare.com/introducing-universal-ssl/ https://blog.cloudflare.com/introducing-universal-ssl/
- jgrahamc 12y agoWe added it during "Week of SSL": https://blog.cloudflare.com/enforce-web-policy-with-hypertext-strict-transport-security-hsts/ https://blog.cloudflare.com/enforce-web-policy-with-hypertex...
- thomasahle 12y ago"Let's just rewrite all the crypto ourselves! In assembly!" Sorry to be a buzzkill, but that sounds like a recipe for disaster.
- imaginenore 12y agoIf it's tested extensively, it should be fine. Somebody has to write the crypto.
- vitriol83 12y agowithout formal methods it's impossible to fully test crypto implementations, such as ECDH, because the number of possible inputs are enormous. bugs in small proportion of inputs can lead to fault attacks. and furthermore side channel attacks are very common.
- imaginenore 12y agoThat's true of every crypto implementation.
- vitriol83 12y agoSure, that's why I think there's an argument to use ones which are more 'battle tested'.
- amaranth 12y agoThat's not an option here though because of the poor performance of Go's C FFI infrastructure. It's like Java in this regard, unless you're handing off large batches of work to the C level all at once it's more efficient to just do the work in Go. Except in this case pure Go isn't fast enough either thus assembly.
- Twirrim 12y agoGo's existing crypto library is far from battle tested either (as well as being comparatively slow)
- ufo 12y ago> Given the many vulnerabilities related to the use of AES-CBC with HMAC What are they talking about here? Are there any important ones if you MAC after encryption? The only vulnerabilities I know of are when you MAC before you encrypt.
- tptacek 12y agoThey're talking about the TLS CBC constructions, not AES-CBC and HMAC in the abstract.
- atlbeer 12y agoOff-topic but, they probably shouldn't be using a Southeast train image in this blog post as they are notoriously bad for delays and slow service in the UK[1] :) [1]http://www.which.co.uk/home-and-garden/leisure/reviews-ns/best-and-worst-uk-train-companies/best-and-worst-trains-for-delays-/ http://www.which.co.uk/home-and-garden/leisure/reviews-ns/be...
- jgrahamc 12y agoI added that. As someone who lives in London I'm well aware of how bad their service is. It was a deliberate ploy to make you spend more time on the blog post by getting your to shake you head about the image and then read the post.
- ashearer 12y agoNice work! The article benchmarks a > 20X speedup for AES-128-GCM, for performance described in the text as "on par" with OpenSSL. It would be helpful for reference to have an OpenSSL column added to the benchmark table.
- Intermernet 12y agoI didn't know the name Vlad Krasnow (I'm not that familiar with the crypto world), but from looking at these [1], [2], it seems that he would know what he's talking about from both a crypto and performance viewpoint. > attempting to make them part of the official Go build for the good of the community Would be interesting to hear what Adam Langley thinks about this, but I couldn't find anything recent on the golang-dev list. [1]: http://dblp.uni-trier.de/pers/hd/k/Krasnov:Vlad http://dblp.uni-trier.de/pers/hd/k/Krasnov:Vlad [2]: http://stackoverflow.com/users/1516766/vlad-krasnov http://stackoverflow.com/users/1516766/vlad-krasnov
- stock_toaster 12y agoI found this: https://github.com/cloudflare/go/issues/5 https://github.com/cloudflare/go/issues/5
- tav 12y agoNice work! Do you guys also have a similar implementation of ChaCha20-Poly1305 for Go? If so, any chance you could share that too?
- jgrahamc 12y agoThat work has been done for OpenSSL, I guess we could add for Go: https://blog.cloudflare.com/do-the-chacha-better-mobile-performance-with-cryptography/ https://blog.cloudflare.com/do-the-chacha-better-mobile-perf...
- tav 12y agoThat would be pretty awesome! Also, do you happen to know if your current patch is safe to apply on top of OpenSSL 1.0.2a?
- biftek 12y agoI'm not sure what's involved with getting assembly working in Go (my experience has only been with servers/clients and CLI's), but could this not have been implemented as a stand alone package? What's the benefit of a fork in this case?
- jzelinskie 12y agoI'm also wondering the answer to this question. To my understanding, go-crypto is actually maintained separately from the Go stdlib by the Go developers[0]. So why isn't this just a fork of crypto? Why fork the entire language? Why can't this be upstreamed? [0]: https://github.com/golang/crypto https://github.com/golang/crypto
- sdevlin 12y agoThis is a library of supplemental crypto algorithms. You won't see things like AES, SHA-2, or the NIST curves in here. Those things are part of the standard library in Go.
- sdevlin 12y agoI can't speak for Cloudflare, but I would guess that they want these changes to be merged upstream. If that happens, the benefit is that consumers of the stdlib crypto API will get increased security and performance for free in a future update.
- elithrar 12y ago> I can't speak for Cloudflare, but I would guess that they want these changes to be merged upstream. If that happens, the benefit is that consumers of the stdlib crypto API will get increased security and performance for free in a future update. Yes, they do: https://go-review.googlesource.com/#/c/8968/ https://go-review.googlesource.com/#/c/8968/
- amelius 12y agoI'm also wondering, why fork the compiler? What if another company forks Go to implement, say, a faster spam filter. How would one ever combine these two forks?
- jvermillard 12y agoStrange they dont mention AES-CCM at all. I'm using it a lot in IoT applications (dtls). It's not frequent in the "web" world?
- donpark 12y agoWish Cloudflare spread some of that Go asm love to Ed25519.
- higherpurpose 12y agoIndeed. Couldn't they have supported it for a small part of users like they're doing with ChaCha20-Poly1305?
- donpark 12y agoChaCha20-Poly1305's 3-4x performance gain over AES-GCM made it worthwhile for them where Ed25519's benefits are not as clear cut to non-proponents. Too bad.