7 ms·
Two years of production Go at CloudFlare
- jorgem 13y agoPretty cool stuff. How big is the team working on these Go projects?
- jbarham 13y agoNice writeup. Especially interesting to see that CloudFlare is using Go for its DNS service. I run a DNS hosting service (https://www.SlickDNS.com https://www.SlickDNS.com) and currently use tinydns from the djbdns suite in my name servers, but I'm writing a drop-in replacement in Go. The goal is to make it much higher performance by using multiple goroutines (tinydns is a single process).
- tptacek 13y agoHave you done any analysis to see if tinydns was actually a bottleneck? The work of an authoritative DNS server is extraordinarily simple, and tinydns capitalizes on that.
- jbarham 13y agoUnder normal load tinydns' performance is more than adequate, but under DDoS attack the fact that it's a single process is a definite limitation when it's running on a multi-core machine. By default tinydns opens its database file for every request which can lead to I/O errors under very heavy load. I've patched it (http://tinydns.org/one-second.patch http://tinydns.org/one-second.patch) to only open the database every second, but with my Go rewrite I can easily keep the database in memory and reload it when the disk file changes.
- seunosewa 13y agoCould you run one instance of TinyDNS on each core?
- halayli 13y agoYou can also write it in C and use lthread. It will be faster than what Go has to offer. https://github.com/halayli/lthread https://github.com/halayli/lthread
- tptacek 13y agoIt will probably not be significantly faster than Golang for this workload.
- halayli 13y agoIt can be. You have total control of memory layout. You can gain a lot of performance by considering locality of reference, slab allocators, and parsing DNS packets in general will most probably be much faster.
- tptacek 13y agoGolang also allows control over memory layout.
- halayli 13y agoof course it does, but it does it in a way that aims to provide an overall good performance for all apps. But it doesn't mean that's the best performance you can get.
- tptacek 13y agoWhat "way" would that be? Golang's approach to data layout is very similar to C's.
- halayli 13y agoC doesn't control memory. brk() does. There are various malloc libraries out there to improve the way malloc manages memory. But ideally you don't want to allocate memory at all. You want to have memory preallocated and preferably accessing a page that's still in cache. You can control memory pages in C because it doesn't manage that for you.
- staunch 13y agoCloudFlare is awesome. They're living out one of my fantasies of productizing really well done front-end web serving infrastructure, which I love to see. They appear to have gathered solid team. And they're doing it with Go! I'd love to see those libraries (sans proprietary modules) show up on GitHub!
- cloudflare 13y agohttp://blog.cloudflare.com/open-source-two-way-street http://blog.cloudflare.com/open-source-two-way-street
- chimeracoder 13y agoMan, there are so many things I love about that blog post (the Go being the least of them). Wish you guys had a presence here in NYC!
- eastdakota 13y agoI'm in NYC this week if you want to grab coffee. Always interested in talking to smart engineers interested in GoLang. Email: matthewatcloudflaredotcom cc: jennatcloudflaredotcom and we'll see if we can set something up.
- billmalarky 13y ago>CloudFlare is awesome. Seconded, I'll throw my website's name into the hat of "websites saved by cloudflare." I was really skeptical of them at first but I'm really glad I decided to give them a try.
- alec 13y ago"The guarantees needed to avoid leaving the server in a bad state when handling panics would be impossible without the defer mechanism Go provides." I'm only passingly familiar with defer, but I understand it to be equivalent to RAII in C++, Python's with statement, Common Lisp's unwind-protect, and others - does is actually provide something more, and if so, what?
- tptacek 13y agoIt's not at all equivalent to RAII, or really any of those other examples; it's a way of clearing up the control flow of a function that needs cleanup work before it returns, but it is a fussy and error-prone way of expressing scoped resource access. "Defer" is nice to have, and because it does less than scoped acquisition, it's easier to repurpose for other jobs; that's kind of thematic of Golang --- simple, orthogonal advances over C/Java/C++; a distinct lack of "theoretical" ambition.
- raybejjani 13y agoDeferred statements will run even when the program panics so you can ensure you clean up. It's more like a finally statement in some languages.
- pcwalton 13y agoGo's "defer" is not equivalent to RAII. It is function-scoped rather than block-scoped and has semantics based on mutating hidden per-function mutable state at runtime. For example: func Foo() { for i := 0; i < 5; i++ { if Something() { defer Whatever() } } // ... the compiler can't tell how many // Whatever()s run here ... } Compared to RAII as implemented in for example D with its "scope" statement, "defer" has much more complex semantics, inhibits refactoring since moving things to function bodies or inlining function bodies silently changes semantics, and cannot be optimized as easily, because of the dynamic aspects. IMHO, it has essentially no advantages over RAII and many disadvantages.
- enneff 13y ago> defer inhibits refactoring since moving things to function bodies or inlining function bodies silently changes semantics, and cannot be optimized as easily, because of the dynamic aspects. As someone who has written and reviewed hundreds of thousands of lines of Go code, I haven't observed this to be the case in practice. RAII doesn't fit into Go, philosophically, as it lets you trigger hidden functionality on the creation or destruction of data structures, whereas a deferred function can only be run if there's a defer statement there in the code (where you can see it). In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures. This brings a huge benefit in terms of readability and transparency. Anyway, I'm not sure why we're comparing defer and RAII, because they're generally used for different purposes.
- tptacek 13y agoGolang is perfect for DNS: its concurrency model matches that of DNS extremely well, its standard library is heavily informed both by Unix and by Internet protocols, and its fast and simple. That's unsurprising, since Golang's charter was to modernize C/C++, and DNS is essentially an expression of 1980's C code style.
- staunch 13y agoGo is perfect for almost every kind of server! Though handling DNS requests over UDP is pretty easy and fast in almost any language. TCP connections are much more pleasant to handle by spawning a goroutine per connection. I do assume using goroutines is slower than multiplexing though, but still plenty fast.
- pmjordan 13y agoI do assume using goroutines is slower than multiplexing though, but still plenty fast. Surely, the goroutine model is just an abstraction built on multiplexing? As in, it uses the exact same syscalls that a multiplexing server written in C would use, except instead of storing state explicitly in a connection structure, it's stored implicitly in the goroutine's stack. The main (only?) real difference is code structure and thus the dispatch mechanism to where in the connection handling code processing should resume: In the goroutine, you're literally storing the instruction pointer and jumping straight to it. In the hand-written case, you'll probably go through some buffering code and an explicit state machine. Basically, Go and its goroutines save you reinventing the connection state capture and resuming code for every single protocol you implement - it comes for free as part of the language. (you can make C do similar things with makecontext/swapcontext plus some extra plumbing code of course) Performance wise it should be a pretty close call: resuming a goroutine will presumably cause a branch prediction miss (due to the change in function return pointer), whereas the multiplexed code will need to wind its way through the state machine logic - almost certainly also a branch prediction miss. CPU cache behaviour should likewise be similar (you're storing the same state either way).
- pcwalton 13y ago
- fizx 13y agoI want an independent write-up of Red October! Sounds really cool.
- pjscott 13y agoI'm not an expert, so take this speculation with a grain of salt, but I'd guess this is how it encrypts something in such a way as to require two people: 1. Generate a key, and use it to encrypt and sign your payload. Nothing fancy here; just plain old symmetric encryption and authentication. 2. Use Shamir's Secret Sharing to split the key into pieces. You need any two pieces to reconstruct the key. This is where the magic happens: http://en.wikipedia.org/wiki/Shamir's_Secret_Sharing http://en.wikipedia.org/wiki/Shamir's_Secret_Sharing 3. Encrypt-and-sign each piece with a secret key derived from its owner's password/passphrase using a secure KDF like scrypt. 4. Throw away the keys, and put those encrypted pieces on disk. Now you need passwords from any two people to decrypt the secret payload. Cool, right?
- bradleyjg 13y agoThanks for the link to SSS. I'd never seen it before, and it's a really, really cool algorithm. One of those things that as soon as you see it you think "of course!"
- codebeaker 13y agoWe're building something similar for https://harrow.io/ https://harrow.io/ which acts as the secret store, it's a Redis API compatible database with a strong auditing layer, and baked in crypto, it requires a single key to start and unlock the server, but we didn't know how to split the key to keep the secrets out of a single dev's hands. Now that we do, we'll build something based on the SSS algorithm and YubiKeys (which we use for all other tokens)
- KMag 13y agoFor millions of secrets and tens of people holding those secrets, there's a much more space-efficient solution. There's an added benefit that hiding each secret is plain vanilla RSA encryption. Victor Shoup discovered a relatively straight forward way to take an RSA private key and apply Shamir secret sharing in a way that commutes with RSA public key operations. Just like Shamir secret sharing, each of the N shares of the secret is a point on a random polynomial on a finite ring (in this case, the set of quadratic residues modulo the key's public exponent). Each of the N secret holders, using their secret share, can generate a public share. Any T of the N public shares can be used to invert the RSA public key operation. Shoup's threshold RSA signature scheme (http://www.shoup.net/papers/thsig.pdf http://www.shoup.net/papers/thsig.pdf) when used for RSA encryption caries all of the caveats that using the raw RSA algorithm entails (use OAEP, etc.) I'm really surprised Shoup-Cramer encryption is more famous than Shoup's threshold RSA algorithm.
- pstuart 13y agoThe "seamless binary upgrade" part is intriguing (unless it's just a reference to the fact that binaries are self-contained (which is indeed wonderful)).
- free652 13y agoI have a feeling they just change symlinks between self-contained binaries. Easy to upgrade or revert binaries.
- dknecht 13y agoWe use a supervising process [1] that has bound the sockets and the pass in the file descriptors to Go process. When the new process is ready to take connections it sense a signal to the original process to shut down. [1] https://github.com/cloudflare/circus/commits/cloudflare2 https://github.com/cloudflare/circus/commits/cloudflare2
- craigyk 13y agoI do part time IT and I have repeatedly wanted a more programatically friendly DNS and DHCP solution for our small network. dnsmasq doesn't have a great high-availability story, and BIND+ISCDHCP can be quite complicated to configure for high-availability (plus BIND doesn't seem to have convenient options such as expand-hosts from dnsmasq). Does this mean Go is ready to use to make a 'simple' forwarding/caching DNS server with the ability to have DHCP update local hostnames? How about DNSSEC validation?
- tptacek 13y agoThere's no official support for DNSSEC in Golang, but then, there's very very little support for DNSSEC in the real world. Which is just as well.
- axaxs 13y agoTrue, but then again, the de facto dns library (https://github.com/miekg/dns https://github.com/miekg/dns) is simply amazing for all things DNS, including DNSSEC.
- amalag 13y agoYou can try powerdns
- alberth 13y agoI wonder how Lua fits into the picture at CloudFlare with this Go infrastructure, since CloudFlare is also a heavy user of Lua [1] as well and key financial sponsor of LuaJIT [2]. [1] http://blog.cloudflare.com/pushing-nginx-to-its-limit-with-lua http://blog.cloudflare.com/pushing-nginx-to-its-limit-with-l... [2] http://luajit.org/sponsors.html#sponsorship_perf http://luajit.org/sponsors.html#sponsorship_perf
- dknecht 13y agoEvery request that passes through CloudFlare goes through Lua. It has how we look up and apply a customer rules to a recess. We just recently realized a non-blocking logger option for Nginx [2] using Lua and are in the process of developing a new KT library [3] in Lua. Here are the most recent lua libraries we have open sourced... [1] https://github.com/agentzh/lua-resty-lock https://github.com/agentzh/lua-resty-lock [2] http://github.com/cloudflare/lua-resty-logger-socket http://github.com/cloudflare/lua-resty-logger-socket [3] http://github.com/cloudflare/lua-resty-kyototycoon http://github.com/cloudflare/lua-resty-kyototycoon
- minikomi 13y agoI love that you brought on agentzh to the team.
- eastdakota 13y agoI would say that C, Lua, and Go are our three primary systems languages. We also use PHP for some of our frontend UI. Dabble in Python in places it makes sense, but usually not in the critical path.
- possibilistic 13y agoHow do you manage large-scale Lua code bases effectively? Is there any thought or literature on that topic? I'd love to learn if you can share.
- camus2 13y agoVery good article,though i dont use go(yet) ,that's the kind of writeup i'd like to see more on HN home page.
- sard420 13y agoCloudFlare is great stuff just wish they offered SSL on their free tier, should anyone really be running without?