8 ms·
Open Sourcing Our Go Libraries (2014)
- joss82 12y agoI find the comments most interesting. An example: For us, one of the biggest latency wins comes from the fact that go can truly execute sql statements in parallel (whereas python's GIL serialized these parallelizable operations). In general, single-threaded go is at least 5x faster than pure python (without c-module).
- zzzcpan 12y agoThey are not executed in parallel though, only asynchronously.
- noselasd 12y agoHow/why are they not executed in parallel ?
- themartorana 12y agoThey MAY be executed in parallel. Concurrency != parallelism[0] BY DEFAULT. That said, if you're running Go against n number of CPUs, then yes, the concurrency may in fact happen in parallel. [0] http://blog.golang.org/concurrency-is-not-parallelism http://blog.golang.org/concurrency-is-not-parallelism
- roller 12y agoIt's a slightly odd comment, SQL queries performed by a C driver are one of the times that you actually get to release the GIL and let the thread do its work without the python interpreter. Depending on the SQL engine, you're ok waiting for the sql backend in parallel, but you may suffer single threaded python object creation processing the result set.
- sandGorgon 12y agothat is interesting - does Gevent not give this in python (or twisted) ?
- vbit 12y agoIt does. I execute many db and filesystem queries in parallel with single threaded Python using gevent. I'm not quite sure whether dropbox didn't or couldn't get gevent or a similar system to work - it's fairly straightforward.
- rudolf0 12y agoI'm a huge fan of Python and gevent, and not the biggest fan of Go, but there's something to be said about built-in concurrency constructs like goroutines compared to something quite hacky like gevent (gevent.monkey.patch_all() everywhere). gevent is my favorite async networking library for Python, and I use it in almost all of my projects, but one can't deny it's basically one giant hack over CPython.
- vbit 12y agoI agree that gevent is a hack. however goroutines correspond to greenlets rather than gevent. You could write clean asynchronous io libraries in python using just greenlets and then it's not a hack. I wonder if dropbox evaluated this option or just rejected python outright.
- rudolf0 12y agoTrue, but the API for pure Greenlets is a bit ugly. In Go, the API for goroutines and channels is superb, concise, and clear. For me, the negative traits of Go outweigh the positive, but I really wish other languages would adapt some of its concurrency constructs as native language features.
- skizm 12y agoCan someone sell me on Go over Java? I know there is a lot of hate for Java on HN but it is easily the most mature of the "fast" languages for the web. Are their glaring features missing in Java or huge wins in Go that I am missing?
- sz4kerto 12y agoVery different use cases. If you can simply choose between the two, then your situation is quite unique. It means that you don't need to rely on the wealth of Java libs available already; hiring is not really an issue; probably you're starting a new project and you either don't have any kind of time pressure or you don't know neither of the languages. :) There isn't anything that you can write in Go but you cannot in Java. (Probably it's true the other way around as well.)
- yalue 12y agoThere are several reasons we've been using go for web backends where I work: - Very easy deployment (we can just copy the same binaries between our 64 bit linux boxes and run them) - Go's libraries for developing minimal web interfaces are excellent - Being strongly encouraged to do all the explicit error checking made our Go code much more reliable and easy to debug than our previous python versions. Of course, all of this depends on how big the project is. We just have several components of a project that require a simple webpage frontend with a small REST backend to interact with Redis. Our entire server for both the frontend and backend is ~1200 lines of Go and easy to understand. If your project is much bigger, you may end up needing features that aren't present in Go (as was the case with Dropbox), which may slow you down some.
- jdcarter 12y agoIn my case I frequently need to link to C libraries, so Java is a non-starter. (I realize many others have the opposite problem.) Go allows me to easily build wrappers for the libraries I need, then get to business. From there it's the concurrency primitives, goroutines and channels, that make Go a true winner. It's easy to build network services that can move work through various stages quickly and in parallel. SEDA-style workloads are a pleasure to implement in Go.
- 12y ago
- gnufied 12y agoThis is one the cases where I wouldn't mind reverting the original title to what OP linked and let readers draw their own conclusions.
- thrownaway2424 12y agoI'm detecting a bit of a trend here. * Start with python * Find out it's a little horrible. * Hire Guido * Python still a little horrible. * Switch to Go
- hunt 12y agoI don't think they found out Python is horrible - the first line is praising the language, it just didn't suit their needs anymore.
- coldtea 12y ago>the first line is praising the language, it just didn't suit their needs anymore. Well, praise for a language is telling people how it suits your needs, so...
- hunt 12y agoIt's certainly possible to praise a language without it suiting your needs.
- TillE 12y agoI use C++ because it meets several key needs that other languages don't. That really isn't praise. I like C++, but it's not wonderful and perfect. It's just the right tool for the job.
- simonh 12y agoApparently it suits their needs quite well: >For clarification, Dropbox will continue to develop majority of its features in Python.
- calebm 12y agoDropbox is not dropping Python: "Dropbox will continue to develop majority of its features in Python. We have only migrated performance critical components to Go."
- 12y ago
- noelwelsh 12y agoSo I guess that Python JIT compiler (https://tech.dropbox.com/2014/04/introducing-pyston-an-upcoming-jit-based-python-implementation/ https://tech.dropbox.com/2014/04/introducing-pyston-an-upcom...) isn't working out so well then?
- bhouston 12y agoSeems to still be actively developed: https://github.com/dropbox/pyston/commits/master https://github.com/dropbox/pyston/commits/master
- rcarmo 12y agoSnark aside, I don't see what one has to do with the other. It's perfectly sensible for them to keep pushing pyston to have an alternate solution.
- noelwelsh 12y agoI wasn't snarking, merely asking a question. It's valid to ask whether the Python compiler is working out since the stated goal is "to produce a high-performance Python implementation that can push Python into domains dominated by traditional systems languages like C++." Since Dropbox has rewritten a chunk of systems in Go it suggests that Pyston isn't working out.
- rcarmo 12y agoIf you check out the announcement timings, it was far too soon for Pyston to have an effect. You can't churn out PyPy-grade JITs like they were burgers...
- jamwt 12y agoWe still have tons of Python code, and likely always will. Therefore, Pyston is still really interesting to us. Only specific, common, core services (storage etc.), are being written in golang; the "long tail" of application code on the backend is (and will remain) in Python. - Dropbox infrastructure engineer
- carbocation 12y agoI'm a little surprised they went with memcache instead of groupcache (unless they are also using memcache on their Python processes). Would love to know more about that choice.
- jdcarter 12y agoProbably because of other services--not necessarily written in Go--which already use memcache. Don't fix things that aren't broken, right?
- deleted 12y ago[deleted]
- thomasfromcdnjs 12y agoAwesome work but I imagine at some point it really is going to make sense to split those projects into their own repos.
- mikebo 12y agoInteresting to note they released these back in July of last year.
- wnevets 12y agowhen/where is Go faster than Python?
- nkozyra 12y agoIt's a statically-typed and compiled language, by most accounts you can expect it to be faster in a lot of places. This has more to do with the interpreter, of course, you can compete with PyPy.
- scott_s 12y agoSince they mention "scalability", I think they're more concerned with the fact that Go has concurrency baked into the language. Python does not.
- voidlogic 12y ago1. Go, being a statically compiled language/having a closer to the metal memory model, generally has higher single threaded performance. 2. Go allows true concurrency in many situations Python cannot (due to GIL etc); Go also supports lightweight threads that are multiplexed over actual OS threads. 3. Go makes it easier to control and reason about heap allocations. 4. Go is even easier than Python to integrate with C/ASM code.
- dragonwriter 12y ago> 2. Go allows true concurrency in many situations Python cannot (due to GIL etc); It would be more accurate to say that Go allows parallelism in situations where Python (in the standard implementation, at least) does not. Calling parallelism (what the GIL prevents) "true concurrency" isn't particularly helpful.
- voidlogic 12y agoTrue, I was imprecise. Thanks.
- alexbardas 12y agoGo has a much better runtime system than Python (can spawn lots of green threads in parallel using goroutines), but it's also broken in so many ways (e.g.: type system).
- Jabbles 12y ago"switch from dropbox formatting to std formatting" https://github.com/dropbox/godropbox/commit/5ed34e410e1c9fe8ea274eaf809d0ee8ad2bdc03 https://github.com/dropbox/godropbox/commit/5ed34e410e1c9fe8...
- mikecb 12y agoDid they just run fmt on it?
- jayrox 12y agoyes
- mikecb 12y agoI wonder what dropboxes reasoning is for not doing that in the first place.
- rcarmo 12y agoProbably because Go indentation and spacing can be a little annoying at first when you're used to Python (or simply because they followed some kind of generic coding standard). It took me a while to "give up" and find fmt-massaged code natural.
- lowmagnet 12y agoThe first thing I usually do in a new language is find out how people use it. With go and go fmt, I don't have to worry about most of it. (golint and go vet cover my other mistakes most of the time)
- lowmagnet 12y agoThe first thing I usually do in a new language is find out how people use it. With go and go fmt, I don't have to worry about most of it. (golint and go vet cover my other mistakes most of the time)
- lowmagnet 12y ago
- mjackson 12y agoUsing Go (or Scala or Java or whatever) doesn't magically scale a large system. Scaling problems in large systems are rarely solved at the micro level, i.e. you don't "scale" by simply gaining the ability to run more operations in a single thread. This is always the problem with "language X is more scalable than Y" debates. From my experience scale has little to do with language, everything to do with how you use it. This post brings to mind another written by Alex Payne a few years back about scaling in the large vs. scaling in the small: https://al3x.net/2010/07/27/node.html https://al3x.net/2010/07/27/node.html The danger with making hand-wavy claims with very few technical details is that it perpetuates the notion that there are certain magic bullets out there that will magically make your system "scale" if you use them. In the past few years we've seen quite a few large companies in SV and SF make the switch to some shiny new language (Scala, anyone?) only to find several million dollars and many months later that they're still having conversations about how to make stuff scale. Only now they're talking about making it happen in a language that is fairly new to their organization and with which they have only a few real experts.
- dewitt 12y agoEverything you said is true, of course, but I think it applies more generally to inexperienced startups with first-time technical founders, or larger corporations with a "B" technical staff. In this case, you're talking about Dropbox, which already has achieved the scale, in both the large and small senses, that require a deeper level of problem solving than simply switching languages would provide. Even if I personally find Go an odd choice for this project (why not C++, which already has the library support they need, and much more?), their present-day successes demonstrate that they didn't expect a scalability panacea from Go, just a better runtime for a subset of their critical path code.
- scott_s 12y agoYour comment reads to me as if you think the author implied that they think Go "magically" provides scalability. I inferred nothing of the sort. Rather, I assumed they wanted to use a language which provides concurrency as a primitive, to make it easier for them to write concurrent code. This switch (Python to Go) for this motivation (better concurrency support) seems reasonable to me, particularly since Python does not have a good concurrency story. (I love Python. But I would not choose it if I wanted high concurrency.)
- SEJeff 12y agoSerious question, not having looked at their caching lib just yet, are they going to be able to beat groupcache, written by golang upstream?
- mediocregopher 12y agoGroupcache addresses a very narrow use-case of caching: immutable data. Key's in groupcache cannot be changed or deleted, which is what allows for some of the cool things it does like distributing keys to multiple nodes automatically and prevention of stampeding. It's useful for things like caching lots of small static files (which I believe is what google uses it for), but it's not useful as a db cache where things are constantly changing. Just glancing at dbox's caching package it looks like a much more general cache, with deletes and sets and all of that. So the two aren't really comparable.
- SEJeff 12y agoThanks for the comparison, that makes a ton of sense. As someone else also pointed out, since a lot of Dropbox's infra is python, it does make sense for them to have a drop in memcached replacement. That means groupcache is effectively out.
- didip 12y agoI hope Dropbox replace the python client agent to Go. That will hopefully cut down the memory consumption.
- cdnsteve 12y agoWhat version of Python does Dropbox use? 2 or 3?
- cdnsteve 12y agoWhat version of Python does Dropbox use? 2 or 3?