13 ms·
Porting dl.google.com from C++ to Go
- codereflection 13y agoI don't remember where I saw this, but somewhere, someone from Google said that all of their code changes every 5 to 6 months (or some reasonably short amount of time). That clearly sounded... optimistic at best. It's nice to see that even companies like Google have 5 year old old that is legacy and causing problems.
- CoryG89 13y agotoo long... ?
- e98cuenc 13y agoThese slides are practically unreadable in an iPhone. They are split in half and it's impossible to get a full page on the screen (I can only see the right half of the previous slide and the left half of the next slide). Anybody has an alternative to read these slides? The content itself seems quite interesting
- packetslave 13y agoSee also https://github.com/golang/groupcache https://github.com/golang/groupcache for the peer-to-peer memcached replacement mentioned in the slides.
- otterley 13y agoI can't wait till it's stable and has a man page!
- kisielk 13y agoIt's a library meant to be used in your Go application, it's not going to have a man page. It already has API documentation: http://godoc.org/github.com/golang/groupcache http://godoc.org/github.com/golang/groupcache
- wbl 13y agoSo what do you think section 3 of the manual is for?
- pjmlp 13y agoUNIX and libc syscalls?
- e12e 13y ago"The table below shows the section numbers of the manual followed by the types of pages they contain. 1 Executable programs or shell commands 2 System calls (functions provided by the kernel) 3 Library calls (functions within program libraries) 4 Special files (usually found in /dev) 5 File formats and conventions eg /etc/passwd 6 Games 7 Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7) 8 System administration commands (usually only for root) 9 Kernel routines [Non standard]" So section 3 is a little wider than that, eg: $ apropos apache::xmlrpc Apache::XMLRPC::Lite (3pm) - mod_perl-based XML-RPC server with minimum configuration
- IzzyMurad 13y agoToo many Google employees in Hacker News trying to advocate Go...
- tptacek 13y agoYes. Nobody here wants to learn how Brad Fitzpatrick reasons about, designs, and implements server software. It's all spam.
- drhayes9 13y agoSecond this. One of my favorite things is getting a slice, however small, of how truly great programmers get their work done, why they chose what they did, and what the results were. We actually need more of this on HN, not less.
- ballard 13y agoThis doesn't reveal any secrets into systems engineering, but it slices: package main import "fmt" func main() { cake := make([]byte, 2600) copy(cake, []byte("Happy Birthday, Lundberg !")) lundberg := cake[:8] milton := cake[:0] fmt.Println("Lundberg got :", string(lundberg)) fmt.Println("Milton got :", string(milton)) } // :-) (Ran out of UTF8 smileys and movie fact checkers). // And yes could've used string but bytes are awesomer.
- pjmlp 13y agoSlices aren't Go specific.
- ballard 13y agoBrowsing the go goog client source is 30% of it. Go and others could use screencasts of how they fixed production problems, regardless of what lawyers, job security worriers or "hide the dust" folks would say. Mindset is best communicated by solving something real. http://code.google.com/p/google-api-go-client/source/checkout http://code.google.com/p/google-api-go-client/source/checkou...
- JulianMorrison 13y agoWhat this actually means: groupcache is awesome. You just act as if the cache is full, and if it isn't, it will be. Where did the data come from? That's pluggable. And no concern of the part that just serves it up. Very subtle, very nice.
- revelation 13y agoWhat this actually means (2): Go standard library has reasonable performance. This won't work so beautifully when whoever implemented all the magic bits under your business logic didn't do so to deliver reasonable performance for your use case. Or if the underlying code is in fact wrong. This turns into the kind of code you can see from Line 242: https://code.google.com/p/google-api-go-client/source/browse/googleapi/googleapi.go#242 https://code.google.com/p/google-api-go-client/source/browse...
- enneff 13y agoI'm puzzled by this comment. We optimise the standard library for the majority cases, and putting the download server into production helped us improve Go's HTTP stack for everyone. Similarly, when other users report issues with our standard library, we fix them. The file you linked to contains some code to accommodate a bug that existed in Go 1.0 and was fixed in 1.1. Not sure what the relevance is here though.
- revelation 13y agoThe new software relies heavily on the Go net/http implementation of a HTTP server. That is arguably a very very big chunk of the complexity in a HTTP download server. So it leaves a bad taste when the presentation mocks the original C++ implementation while the Go solution only works because there is a fully functional high performance HTTP server already in the standard library. There is not a single slide that tells us it may not be a good idea to rely on the standard library for such high-level functionality. Thats why I linked to the bug. A normal user has very little options when it turns out theres a bug in the standard library, other than to report it and wait for you to fix it. The problem is exacerbated since standard library and language versions are tightly coupled.
- joebo 13y agoI don't understand the need for the payload server from the slides. That makes me wonder - why not just use a HTTP server to serve the static files (e.g. nginx)? I'm sure I'm missing the obvious, but I'm probably not the only person wondering it.
- evmar 13y agoIt appears from slides 14, 21, and 24, that copying all of the static files that needed be served onto a single machine could take over 12 hours. Another way to say that is that the totality of content that is served from that single host is too large to put on a single machine. The "groupcache" abstraction lets the http-serving machines dynamically serve the hot subsets of the larger file set (represented by "storage svc" on slide 55).
- drdaeman 13y agoBut nginx can work as cache, acting as reverse proxy on misses. Minor amount of Lua scripting or C (or, better, whatever fancies one's taste if that could be called from C) hacking can teach it any reasonably complex semantics (fancy ACLs, not-before dates etc.). Many modules (dynamic archive generation, GeoIP, various cache backends) are already there, well tested in production environments. Surely, one would probably need a separate external tool that would pre-fill (nginx wouldn't know that some file's pending before it's requested) and clean up caches (provided that rules are complex than trivial LRU removal on some threshold), as one probably wouldn't like webserver doing this unsuitable job, but that should be another story.
- Aloisius 13y agoIt appears to be a caching proxy that has some very specialized features like dynamic zip files and what appears to be origin server notifications which you don't normally see in off the shelf caching proxies like say, Squid. Then again, I'm not sure why the caching wasn't better handled in the CDN that lives in front of it (cache hierarchies work well) leaving this server to simply serve only the very first request.
- bsaul 13y ago
- hosay123 13y agoEither I'm having deja vu, or despite the date on the presentation, this is at least a year or two old
- skyebook 13y agoI don't know if its that old, but there was definitely a talk on this presented some time back. From what I remember, there was a lot of HN discussion around "well that's great but its just a download server" and some other people saying "yeah but its a complicated download server".
- fizx 13y agoGroupcache is the interesting new part of the presentation, and it was just open-sourced a few days ago.
- codereflection 13y agoIt could be least a year old, title from a slide: "Summary of 5-year old code in 2012", but then again perhaps that's when they started the re-write in Go.
- conroy 13y agoThe Go rewrite of dl.google.com happened last year [1]. Today, Brad Fitz gave a presentation about the rewrite and open-sourced a component of the rewrite, groupcache[2]. [1]: https://groups.google.com/forum/#!searchin/golang-nuts/dl.google.com/golang-nuts/BNUNbKSypE0/E4qSfpx9qI8J https://groups.google.com/forum/#!searchin/golang-nuts/dl.go... [2]: https://github.com/golang/groupcache https://github.com/golang/groupcache
- fizx 13y agoHow does groupcache handle consensus? Edit: Scanned the source, looks a like a best-effort distributed lock, rather than any sort of consensus protocol. This works for a cache setting, where e.g. having a split-brain scenario and duplicating the work is no big deal.
- tux21b 13y agoI was thinking about that too, but since it is not possible to change existing items and the computation of those items must be re-entrant (groupcache just tries to avoid duplicate computations but does not guarantee it), there seems to be no reason for any distributed consensus. In fact, the groupcache design is astonishing simple.
- skriticos2 13y agoSo what I take from this is that the previous implementation sustained a huge amount of code rot and new code got layered over it with a staple instead of proper re factoring. So he put the whole mess in a bin and re-done it cleanly with Go. Now it's much nicer. Some of Go's attributes helped along the way. Did I miss something?
- rkrzr 13y agoYes. In my opinion the biggest win comes from goroutines and Go's runtime, which essentially allow you to write asynchronous non-blocking multi-threaded(!)code that looks like synchronous code and is as easy to read and reason about. This leads to a huge reduction in complexity compared to the callback spaghetti you often get in other async frameworks. Basically putting concurrency first in the language and not relying on libraries seems to really pay off.
- YZF 13y agoInteresting story. Is this a "port" or a "rewrite from scratch"? It's kind of hard to tell.
- conroy 13y agoIt's both. The code was completely rewritten, but the interface to the service stayed the same.
- bradfitz 13y agoParts of both. Some of the logic is ported from C++ to Go almost line-for-line. Some of the architectural parts are completely redone. But it has the same binary name and flags and RPC interface
- c0rtex 13y agoAside: Does anyone know how these slides are generated?
- bradfitz 13y agohttp://godoc.org/code.google.com/p/go.talks/present http://godoc.org/code.google.com/p/go.talks/present http://godoc.org/code.google.com/p/go.talks/pkg/present http://godoc.org/code.google.com/p/go.talks/pkg/present
- azth 13y agoPretty disingenuous on slide 58 to attempt to make the Go code look shorter than it actually is. Note how he left out all the verbose error checking code.
- bradfitz 13y agoSorry, wasn't my intention. But it's only 4 more lines. In my defense, I only showed three pages of C++, and not all of it, which would've been longer than the whole presentation. So I cut less from the Go snippets than the C++ snippets.
- ubershmekel 13y agoYour intention was to show the good Go code vs the bad C++ code. The different syntax highlighting colors and the missing error handling code really leave me feeling amiss. My main gripe and reason for not jumping on the Go bandwagon is the error handling strategy, it was exactly what I wanted to see in the Go version. But thanks for the info anyhow.
- PommeDeTerre 13y agoAdditionally, I think that this comparison would only be valid after the Go version experiences an equivalent amount of developer turnover, features added in a rush by developers inexperienced with the software, external infrastructure changes, and the other pressures that the C++ implementation was apparently subjected to over the years. The Go code may look good in 2012 and 2013, while it's still fresh. But I'd be very curious to see how it looks in 2017 or 2018, assuming it's still even being used then.
- drivebyacct2 13y agoIf you actually just wrote some Go, you'd realize it's not even remotely an issue.
- godbolev 13y agoDoes anyone have a link to the video?
- dakimov 13y agoGoogle's approach to development at least in this part looks very unprofessional. The have had a bunch of crap code nobody cared about, and then some guy has rewritten it arbitrarily choosing development tools just because of his own preference. What next, another guy will show up who likes JavaScript and considers the previous work unsatisfactory, and will rewrite everything in JavaScript? Who works like that?
- _random_ 13y agoIt seems that Go is a good replacement for Python as well?
- rbanffy 13y agoAs someone who wrote a lot of Python code ans who is learning Go while implementing a somewhat important application, i can tell you Go is a good replacement for Python if it fits the problem better (mine was concurrence). Go is pleasant, but there are Go problems an Python problems (and C problems, Lisp problems, and so on)
- STRML 13y agoMaybe I'm showing my allegiance to my platform of choice, but the subtle dig on nodejs wasn't warranted on slide 25 (http://talks.golang.org/2013/oscon-dl.slide#25 http://talks.golang.org/2013/oscon-dl.slide#25). As everyone's pal `substack` will tell you, use streams! Instead of explicit buffering, handling backpressure, etc., it's as simple as: readable.pipe(writable); Additionally the link to `http-proxy` on slide 30 is misleading; 60% of that file is comments, and about 50% of what's left is websocket support, with the rest being header parsing & redirect parsing. The actual proxying bit is very simple and straightforward, and if you don't need every feature `http-proxy` offers you can do it yourself with streams in < 10 lines.
- bradfitz 13y agoIt wasn't a dig on nodejs. It was a dig at event-based programming, on which I've wasted years of my life in many languages. Node.js isn't unique in that regard. As I mentioned in my talk, that code looks like fine Node.js code. But it's still event-based, and the flow isn't readable. In the actual presentation I went through the code to show how control flow jumps around. I picked a Javascript project (and the top hit I got from a search) because people know Javascript. Websocket support doesn't matter. In Go, you can also just io.Copy(websocket, src). I agree Stream makes Node.js code better.
- test-it 13y agoHave you by any chance seen the async keyword in C# 5.0? It allows one to write event-based code without callbacks obscuring the control flow. From what I've heard Python is in the process of copying this feature. Iced Coffescrip does something similar also.
- bradfitz 13y agoThis answer sums up my feelings best: http://stackoverflow.com/questions/7479276/what-is-the-main-difference-between-net-async-and-google-go-light-weight-thread http://stackoverflow.com/questions/7479276/what-is-the-main-... It's good for C#, but still a language wart that could be built-in. I like that Go only has one set of APIs for everything, not the sync way and the async way. It's sad that C#, which started out as a fixed-up Java, is now growing its own warts. Of course, Go's not perfect either.
- deleted 13y ago[deleted]
- __Joker 13y agoI still don't understand why google does not give option to download via torrent ? Downloading android studio from dl.google.com last week over a slow connection was a horrible experience. I had to retry three times before I managed to get a successful download.
- artagnon 13y agoWhile I won't dispute that Go has some cute primitives, I thought the examples were terrible. On slide 25, it talks about why a simple operation is painful (http://talks.golang.org/2013/oscon-dl.slide#25 http://talks.golang.org/2013/oscon-dl.slide#25), and then goes on to evangelize io.Copy() on slide 31. Okay, so the standard library saves me from open-coding it: func Copy(dst Writer, src Reader) (written int64, err error) { // If the reader has a WriteTo method, use it to do the copy. // Avoids an allocation and a copy. if wt, ok := src.(WriterTo); ok { return wt.WriteTo(dst) } // Similarly, if the writer has a ReadFrom method, use it to do the copy. if rt, ok := dst.(ReaderFrom); ok { return rt.ReadFrom(src) } buf := make([]byte, 32*1024) for { nr, er := src.Read(buf) if nr > 0 { nw, ew := dst.Write(buf[0:nr]) if nw > 0 { written += int64(nw) } if ew != nil { err = ew break } if nr != nw { err = ErrShortWrite break } } if er == EOF { break } if er != nil { err = er break } } return written, err } Uh, big deal? The chunk of what's important isn't explained at all: - runtime/ takes care of memory management quite efficiently with a decent tracing gc in runtime/mgc0.c. I haven't benchmarked it against other stop-the-world collectors, but it should be no match for truly concurrent gc. - runtime/proc.c schedules various blocking and non-blocking (called netpoll, which resolves to epoll on systems where it is available) calls. It seems to account for number of cores and use native threads, but I'm not sure how it interacts with the Linux scheduler. - runtime/malloc.goc is the core memory allocator/deallocator. Seems to be a relatively straighforward arena allocator using a bitmap. I didn't have time to go through groupcache, but the presentation certainly didn't tell me much about it.