13 ms·

From Python to Go and Back Again

366 points by azth 11y ago

CatDevURandom 11y ago
I use and like both python and go. The presentation mirrors my own experiences, though I haven't come back again yet.
- kawera 11y ago
  I use and like both too; webapps in python (django/flask) and microservices in go. My systems also use some large programs written in go but more like black boxes (nsq, heka, influxdb).
jbeja 11y ago
Glad that some people see the light.
hvs 11y ago
I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war stories in this industry because everything has pros and cons, and our job as software engineers is to be able to make decisions based on limited information. Case studies are great way to glean real-world experience from others without having to implement every new technology yourself in order to make high-level assumptions about that technology. This article shouldn't say to you: "See, Go is BAD, Python is GOOD!" It should say, "That's an interesting case study. If I'm working on a project that involves lots of sockets and concurrency, I'll want to take what they said into account when I'm making technology decisions."
- sethammons 11y ago
  I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that email now :) [edit: grammar]
  - windlep 11y ago
    I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a goroutine each... you can easily exceed the memory constraint. As such, we had to put goroutine pools in place, careful select statements around them from connections to ensure we didn't overwhelm external resources, etc. It was a huge pain. It has been drastically easier to control resource usage with these constraints under python/twisted. YMMV, of course, this is just our experience. Part of the reason for putting it out there is that there are already many people who have talked/blogged about going from Python -> Go. I thought maybe the world could handle just one story about going the other direction.
    - mrits 11y ago
      I miss the time when you bragged about increasing performance without resulting in having to switch frameworks or languages.
      - windlep 11y ago
        So do I! Eventually you 'top-out' in a language/framework though... and then its all tears.
    - Jabbles 11y ago
      Typically if you wish to limit the number of goroutines you would spawn N workers and have them read from a single channel. If 20k of your incoming connections want to do something they send on the channel, without spawning a goroutine themselves. Did you try something like that?
      - windlep 11y ago
        Yep, this is what I meant by 'goroutine pools'. The select statements were on the sending side to ensure if the feed channel was full we wouldn't retain too much additional state. It works, but at that point its starting to look like an async event-loop with a thread-pool....
- truncate 11y ago
  Just wanted to share my own very small case study. I had a homework assignment to build a polite crawler. I initially built it in Python, and it was awfully slow. I rewrote same thing in Go, it turned out to very very fast (10x at least IIRC). I liked the fact how quickly I was able to write something so quickly (with not so shabby design) in Go with so much less experience in it. Go is definitely awesome for writing concurrent code quickly. Its not big industry story, but as a busy student I still feel great about using Go. Reason being, we had to use same crawler for doing other stuff, for which a fast crawler was really handy and saved me hours. The problems I observed with Go was that its regex seemed to be slower than Python, and memory usage was way higher. I explicitly added some GC requests.
  - crdoconnor 11y ago
    Did you try pypy before rewriting?
    - truncate 11y ago
      Didn't solve my problem. PyPy still retains multi-threading and GIL stuff if I remember correctly.
      - lqdc13 11y ago
        Did you try requests_futures lib? It's all about async and network speed with crawling. Not so much cpu.
        truncate 11y ago
        I thought it would be IO bound (that's why I started with Python at first place), but since I was extracting links as well and working a bit on graph it turned out to be more CPU intensive. But well, maybe I could have written better code, better libraries, maybe multiprocessing (would have been painful though with multiprocessing). I do admit, I didn't look much into how I could improve it within Python. I just went with Go because it was quicker that way for me.
- cs702 11y ago
  Amen. This post reminds me of another post I recently saw on HN, in which the author (someone with an Erlang background) lays out all sorts of reasons why he chose Ruby for a highly concurrent application that launches lots of (heavyweight) threads. Upon seeing the link on HN, my first thought was, Ruby!!?? But then I read the post and the reasons were all very sensible and practical-minded, so in that case Ruby was arguably a much better choice than Erlang, Go, Scala, Rust, etc. for a highly concurrent application. Edit: here's the post I mentioned about Ruby being used for a highly concurrent application: https://news.ycombinator.com/item?id=10394450 https://news.ycombinator.com/item?id=10394450
tinco 11y ago
I've been porting a project from Ruby to Go in search of making it a bit lighter. The project is about a 1000 lines now so I'm not allowed to criticize Go yet, but so far it's very nice. I picked the language up in just a few hours as I went and the whole project took just about a week and a half. So as someone who is clearly in no position to be criticizing other projects yet, isn't Heka exactly the sort of project you shouldn't do in Go? I say that because I have the feeling you should use Go only for very concrete cases, given its lack of proper abstractions. I.e. writing a tool that receives log lines over HTTP, extracts metrics and forwards those to StatsD? Perfect use case for Go. But writing a tool that lets you plug in arbitrary frontends to forwards to arbitrary backends? Perhaps they got it to work nice, but that sounds more like a case for a more general language.
- windlep 11y ago
  For heka, one of the things we wanted besides performance, was a small, easy-to-distribute binary you could drop on a system and 'just run'. Go is fantastic for distribution thanks to the small staticly linked binaries you can get out of it. To accommodate the pluggable filters and such later, a Lua sandbox system was added... and oddly some of the other Go pieces ended up being faster in Lua which is why Lua decoders/inputs are now an option for heka as well. Note: I gave this talk.
  - mdup 11y ago
    Not advocating anything, but could C++ have been a good choice? You'd get top performance, small memory footprint, opt-in static linking, at the price of a less "fun" language and verbosity. Regarding memory leaks, it's easy to write a leak-free C++ program these days with smart pointers. However, it doesn't come "batteries included" for net-related stuff, and ASIO can give a couple of headaches.
    - windlep 11y ago
      It could've been. Though now that so much of heka is involved in moving data between the Lua layer, turning it into plain C is a better option. trink, one of the heka authors has a project called hindsight that does just that.
zeckalpha 11y ago
Yes, it is time for Rust.
- scott_karana 11y ago
  Yeah, Rust sounds like a great Python replacement. /s I do like Rust, but must we clutter every Go-related thread with comments about it? The zealousness is annoying.
  - cwyers 11y ago
    Well, the last slide said "time for Rust" and the presentation was from Mozilla.
    - windlep 11y ago
      I meant that partly as a joke, I do happen to like Rust though. And in the context of this technical problem, its memory control would make it an attractive possibility (as would C/C++/etc). Lots of context missing as these are merely slides, and I said quite a bit more than what is presented here.
  - DasIch 11y ago
    If you're problem is that you're memory bound, using a language that allows precise control over memory allocation and layout is certainly a good idea. The constant conflict might be annoying but using Rust or even C++ would certainly be a very reasonable choice.
  - pekk 11y ago
    Well, Python-related threads have been cluttered with comments about Go for a few years now...
- marvel_boy 11y ago
  Sure, Rust is the way to go.
latch 11y ago
Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go. At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a complex API to work with. Go's a good language for some things. But it does nothing special or significant to close the massive productivity gap between dynamic and static language. Yes, it's terse compared to many other static language and it has stuff like implicit interfaces, but those are superficial (but nice) things when it comes to what and how you do things in dynamic land. Go might actually be a step back due to its poor type system and poor reflection capabilities. I think Go's great for a seemingly new breed of "infrastructure" systems which is becoming more important due to how systems are starting to be designed (services hosted in the cloud). It's great for building CLI which don't require your customers to install anything else. And it's good for a services / apps that needs to share memory between threads (which, in my experience, is where dynamic languages really start to fail). But for a traditional web app / service? It's horrible. At least as horrible as most static languages. It sucks for talking to databases (more than almost any other language I've used). It sucks for dealing with user-input. Like most static languages, the stuff you need to do to handle a request, which has essentially 1ms of life, is cumbersome, error prone, slow, inflexible and difficult to test.
- windlep 11y ago
  I mentioned during the talk, that the reason for giving it a spin in Python in the first place was that we had to fill in gaps in a Go library. While there's lots of Go libraries, many of them are immature, and have chunks of functionality missing. So while I was blocked on a coworker filling in one of the gaps we both needed, I was able to rewrite it all in Python. I'll probably give this talk again at a venue where it'll be recorded, which should add a lot of missing context to these slides.
- eliben 11y ago
  > Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go Precisely, and isn't this presentation the perfect example of this phenomenon? * Initially we had an implementation in language X * We then rewrote it in language Y - the lessons learned by making the system anew (this time knowing the exact problematic spots, what really to optimize for, etc) - we got a better system. Long live language Y * We then rewrote it in language X - the lessons learned [...] - we got a better system. Long live language X Good programmers can productively write good and fast code in C, Python, Java, Go, or whatever. The skill of the developer and the understanding of the problem matters much more than the programming language
zem 11y ago
i've seen a lot of these posts ending along the lines of "it's time for rust". two languages that are always conspicuous by their absence are D and ocaml. D in particular seems like it would be the logical upgrade path from python or ruby. it has a comfortably familiar C lineage, supports a variety of programming paradigms, and has good concurrency support. i wonder why people don't at least give it a look. (personal experience - i tried to use it twice, several years ago, and gave up because the tooling was bad, but from what i've heard that's very much improved today)
- FraaJad 11y ago
  D has replaced Python for my regular programming. D is the sweet spot for Python programmers to upgrade to without going backwards to Go (Programming language design wise) nor weighed down by all the new (and very good) stuff in Rust. D has everything from a nice IDE(Xamarin Studio), debugger, package management (Dub), statically compiled binaries, pretty decent std lib (not as good as python or Go, but very good nonetheless). I still write Python if it's a "script" that has to run on a $work server, where it is safe to assume that Python would be available and sufficient for most tasks.
  - ylh 11y ago
    People saying "backwards to Go" instantly reminds me of the following quote: Are you quite sure that all those bells and whistles, all those wonderful facilities of your so called powerful programming languages, belong to the solution set rather than the problem set? — Edsger W. Dijkstra
    - FraaJad 11y ago
      Interestingly, ALGOL failed because it was too complicated to implement.. And Djikstra played a huge role in the formulation of that language. Why aren't you using ALGOL?
      - ylh 11y ago
        ALGOL-60 saw fine use in its day; Dijkstra criticized ALGOL-68, which indeed failed because it was too complicated to implement.
falcolas 11y ago
I'll be interested in coming back to Python when it isn't a headache to deploy into production. I'm tired of an install requiring a GCC compiler on the target node. I'm also tired of having to work around the language and ecosystem to avoid dependency hell.
- rattray 11y ago
  docker?
  - falcolas 11y ago
    Doesn't solve the issue of needing a C compiler for third party extensions, and definitely qualifies as a work-around for the existing toolset. Yes, it helps. But you can use Docker with Go programs as well (and drop a lot more of the base image in the process).
    - bmurphy1976 11y ago
      Actually, nice thing about docker is you can build a compilation container (pre-built with all your C/C++ apps ready to go and shared amongst your coworkers), compile your extensions using that, and then only install them into your target container (sans compilation tools). It's a little more grunt work that way, but you get better control and reproducibility without the explosion in image sizes.
    - zweiterlinde 11y ago
      The way we do this is to have a base image that has already yum installed or pip installed all modules (non trivial, anyway) that our package needs. Then the docker image that needs to be rebuilt (that depends on the first one) is just a minimal pip install away.
- thristian 11y ago
  The way I deploy Python apps at $EMPLOYER: - CI system detects a commit and checks out the latest code - CI system makes a virtualenv and sets up the project and its dependencies into it with "pip install --editable path/to/checkout" - CI system runs tests, computes coverage, etc. - CI system makes a output directory and populates it with "pip wheel --wheel-dir path/to/output path/to/checkout" - Deployment system downloads wheels to a temporary location - Deployment system makes a virtualenv in the right location - Deployment system populates virtualenv with "pip install --no-deps path/to/temp/location/*.whl" The target node only needs a compatible build of python and the virtualenv package installed; it doesn't need a compiler and only needs a network connection if you want to transfer wheel files that way.
mratzloff 11y ago
There are people who have been working in Go for years, successfully, but who don't post comments with the same frequency and dogged determination as the middlebrow dismissers. Both Python and Go are fine. They both have their strengths and weaknesses. I personally wouldn't write a web app in Go (at least, anything beyond the most basic admin interface). I also personally wouldn't write a very large and complex Python system given the huge unit testing burden necessary to ensure safe refactoring down the road. The biggest reason I like Go is because it makes it really hard for engineers to create huge, complex abstractions. Engineers (and especially less experienced engineers) just love them some abstractions. In my experience, most abstractions aren't justifiable. The net effect is that it usually makes their software harder to learn, harder to debug, inflexible (ironically), and late for whatever deadline they were supposed to hit. You can't write Java enterprise software in Go, and I really appreciate that.
- rattray 11y ago
  Do you have any increased optimism about the refactorability of python given the recent addition of type annotations?
  - mratzloff 11y ago
    Definitely, but given the relative recency my comment still stands. Ultimately, all libraries in use in a given application must support this as well for it to be truly comprehensive. As a side note, sometimes you do need that performance gain, without wanting to resort to C or C++. I hope to see Python make some gains there with the addition of the type information.
  - xj9 11y ago
    I would really like to see a type check package that you can run alongside your unit tests to verify type correctness. MyPy is nice, but I don't want to do type checks at runtime.
- anacrolix 11y ago
  > You can't write Java enterprise software in Go, and I really appreciate that. You can, and I've seen it, unfortunately.
stock_toaster 11y ago
The claim that pypy uses less memory than Go seems...rather extraordinary. I worked with a fairly complex http api app that ran as a rather svelte wsgi framework under gunicorn, and we saw at least 10x or more increase in memory usage than cpython when we switched to pypy, once the jit was fully warmed up (memory usage seemed to hit steady state after about an hour). pypy has also historically (in my experience) been a bit more "lazy" (deferring GC of individual objects longer) than cpython when an object falls out of scope. My general rule of thumb for pypy has historically been that you trade memory (requires more of it) for speed (faster) when compared against cpython. Maybe the reimplementation itself was just far more efficient? Hopefully the talk itself clarified that point. I would be very interested in hearing more about that particular aspect.
- windlep 11y ago
  This was before Go 1.4, which dropped memory use per goroutine from 8kb -> 2kb. They're fairly close now (minus my leaking goroutines). ;)
- DasIch 11y ago
  PyPy's JIT requires quite a bit of memory. CPython doesn't have a JIT so it doesn't share that problem. However PyPy also has optimizations specifically to keep memory usage down. Especially if you have a lot of objects, it's certainly possible to consume less memory than CPython. PyPy using less memory than Go does seem weird but depending on how much the GCs differ and how they are configured it could simply be that Go's GC doesn't give up memory as freely to the OS.
- dbaupp 11y ago
  I don't know the details, but the memory difference was almost certainly due to the different approaches to concurrency. Python coroutines just need to save their exact stack frame while a Go goroutine will spawn a "massive" stack that likely has much more space than needed. The first case might only need tens or hundreds of bytes for a dozen local variables, while the latter case is a fixed overhead of several KB (8 previously, but 2 with Go 1.4 according to the slides). The JIT memory is constant at runtime (proportional(-ish) to the amount of code, which is fixed) while it is desirable to have the number of coroutines be as large as possible.
JulianMorrison 11y ago
Huge interfaces are not how you do it. Little ones, each expressing a conceptual unit of the functionality. Loggable. MessageConsumer. Stuff like that, even if it only wraps one method. Make each major subsystem into an object, connect to it through the mini-interfaces it provides, and test its consumers by stubbing those mini-interfaces, not the whole object.
ntrepid8 11y ago
This echoes some of my own experiences. Python (even CPython) waits on the network just as fast as Go (or anything else). I've had good luck writing applications in Python, then profiling them and implementing critical sections that are CPU bound in C as modules. Any sections that are memory hogs can be converted to stream processors. Lately I've started implementing the modules with Rust and the results are promising. It seems like a nice balance of developer productivity and application performance.
eugenekolo2 11y ago
Does anybody have some comments on the closing statements on Google's 10KB secret?
- nulltype 11y ago
  I suppose there's the relevant blog post https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html https://www.imperialviolet.org/2010/06/25/overclocking-ssl.h...
  - spoiler 11y ago
    Is it possible that it's just a typo, though?
    - nulltype 11y ago
      That would be pretty hilarious.
    - eugenekolo2 11y ago
      Not sure what you mean, but the site linked above says they do it <10KB and then a few paragraphs down says how they patched it to go down to a bit more than 5KB. So, I don't think it's a typo.
windlep 11y ago
I have to note this, because I think it deserves quite a bit more attention. SSL is extremely expensive, on RAM (perhaps the implementations have optimized for throughput over RAM). I have yet to benchmark any SSL implementation in any language, with any binding, that can use less than 20kb per SSL connection. I mentioned in my talk here that SSL is very expensive, here's my benchmark suite that others may add to: https://github.com/bbangert/ssl-ram-testing/ https://github.com/bbangert/ssl-ram-testing/ I have implementations in several languages, so far both Go and Python 3.4 can get as low as the 20kb cited. If you can get your per-connection state below 20kb, then merely adding SSL means doubling or worse your RAM requirements, which is huge. I appreciate that everyone loves obsessing on the language wars, but the SSL RAM overhead affects us regardless of language. I covered that in one of the slides near the end, it'd be great to see some movement on reducing the RAM footprint here.
- nulltype 11y ago
  Do you have results for the different tests? I did not see any in the repo.
- icot 11y ago
  Do you have any insight about the mention in the slides of Google having a 10kb solution?
- ssfak 11y ago
  Doesn't a TLS terminator proxy solve this? E.g. I usually put my application services behind HTTPS-enabled nginx and it works wonderfully.
  - windlep 11y ago
    Nope, so, the goal here is to reduce how many machines (each with their own RAM limits) are used. The task is holding open bidirectional SSL wrapped long-lived websocket connections. They're held open for hours at a time, since we need to send notifications when we get them. Every connection has a base cost of the TCP kernel send/recv buffer, which in our case we dropped a bit to 4kb each. So that's still 8kb per connection right there. If we terminate the SSL on a separate machine from where we handle the connection, then it means we'll be using 8kb more memory per connection. Probably even greater because nginx has its own send/recv buffers for data. I'm sure our use-case is a unique one, most people care about raw through-put so the majority of SSL optimization has focused on lowering CPU use under high load rather than memory use under massive amounts of connections.
0x434D53 11y ago
Go is one of my working languages. Like with every other language (Python, OCAML, C#, some Java, Swift) i have a love-hate relationship with go. What i agree on: What i agree on with the presentation: Concurrency using goroutines and channels is f... hard besides very primitive scenarios. Even fork-join isn't that easy. There also the lack of expressiveness hurts: It's nearly impossible to build higher level abstractions above the channels/goroutines. You always have to do the bookkeeping of your goroutines. I also agree on the error-handling problems: It's often hard to locate errors. It requires a big amount of discipline by the programmers to achieve some kind of ability to locate errors. No, i don't want the Java/C#-'i throw exceptions everywhere'-style back, but Go is the other extreme. Some more lightweight-panic wouldn't be bad. What i can't agree on: That you cannot mock without interfaces in Go is typically not a problem: There is no real encapsulation (_, ., no constructors) so in many cases you just an instantiate your structs as you need them. The classic for mocking - time - is problematic as in every language. IO is typically behind the various io.Reader/Writer... interfaces: No problems there. The criticism about memory consumption i don't get: Every system i saw ported to go from Java, Ruby or Python had a much lower memory footprint than before. And typically go allows to optimize allocation quite well when needed.
- fauigerzigerk 11y ago
  I agree that error handling in Go is a headache. But since we are forced to handle every single error where it occurs, we can at least make the best of it and add context information before returning it. I'm using a small utility library to wrap the original error and add function name, line, file name and optionally a descriptive message that explains what failed.
  - 0x434D53 11y ago
    You are not forced to handle errors. Or do you check for the errors returned by fmt.Print? I guess not. This was my point: In many areas go forces the developer to the right thing (no unnecessary imports, gofmt...) and does not rely on developers discipline. But when i comes to error handling it does. What i would wish for is some extended error handling supported by the compiler. I don't want a stack trace, but the compiler easily could produce for example a line number where the error was returned.
gtrubetskoy 11y ago
I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.
- windlep 11y ago
  Why would it? Using twisted (or py3's asyncio), the event loop is spinning, there is no thread for Python to be switching to. Except that we're using a thread-pool of course (to make external blocking network calls), and those threads are.... making network requests, which is I/O, and the GIL releases during that. On a network bound daemon, which is not CPU-bound, the GIL is really not an issue, so it never came up.
  - gtrubetskoy 11y ago
    "On a network bound daemon, which is not CPU-bound" - I think that's a fairly unique constraint which makes your case special. For me, the GIL is unfortunately where Python is unable to compete with Go.
- alextgordon 11y ago
  node.js has a GIL and nobody talks about that either. The GIL is not relevant because Python is so slow that running it in two threads is hardly an improvement.
  - gtrubetskoy 11y ago
    But don't you need to run a bunch of node.js instances in production on a multi-cpu system behind nginx or haproxy? Would it be necessary had it not been for the GIL?
    - alextgordon 11y ago
      Most node.js code spends barely any time on the v8 thread. The node code merely invokes routines in libuv, which has a thread pool for anything computationally intensive.
  - 11y ago
giaosudau 11y ago
btw, python with pypy gonna be a bright future.
bootload 11y ago
"Go experts still can't write leak-free code" Big statement, anyone confirm this?
- dbaupp 11y ago
  The next two slides have screenshots of issue tracker searches for "goroutine leak".
  - bootload 11y ago
    yeah I saw those, what about examples from independent people reading the article on HN?
- codygman 11y ago
  Now I know to ask about goroutine leaks next time my colleague who loves Go talks about how bad space leaks are in Haskell :P
Animats 11y ago
What's unusual is that this project uses PyPy in production. PyPy has been a long time coming. Until recently, Python was defined by CPython, which is a naive interpreter. As this article points out, performance is roughly an order of magnitude better with PyPy. Now PyPy has to be taken as seriously as CPython, if not more seriously. In a few years we may look at CPython as the "toy" implementation.
sandGorgon 11y ago
what is strange is that everyone seems to be using Python 2.7 in some form. There is all this new work being done in Python 3 asyncio but there is nothing that is being used... and this is from a guy who pretty is a core developer in several python projects. I look at Ruby or Go... or even Java and every new language feature has a much more rapider adoption curve. is this a pretty solid statement that the entire Python 3 + asyncio path is a dead end ?
- windlep 11y ago
  We used Python 2.7 for pypy compatibility. I really dig the new async/await primitives added to Python 3.5 and think the asyncio path is the best path forward. I hope the work to get all the latest asyncio stuff working on PyPy picks up some steam.
  - sandGorgon 11y ago
    hey - thanks for replying. if Pypy + asyncio was available, would you have built everything using that stack ? There have been all these benchmarks that asyncio is so much slower than threads [1] How would you compare that with Go ? [1] http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
    - kintamanimatt 11y ago
      For certain things that they probably shouldn't be used for, not across the board.
  - bpicolo 11y ago
    The main issue with asyncio is it is incompatible with some of the best packages in the ecosystem atm, e.g sqlalchemy
- broodbucket 11y ago
  The difference is that Python 3 is far more backwards incompatible than an update to Ruby, Go or Java. If there's any component of your Python project that was written for Python 2 and it hasn't been ported, your project as a whole is probably staying in Python 2. I certainly wouldn't call the Python 3 path "dead", as starting a new project you'd be silly not to use Python 3. It's just a very slow process. Also, porting from 2 to 3 isn't even that hard (there's a script that does 95% of it for you).
zobzu 11y ago
"drank koolaid, used tool without knowing what its good at, lost 2 years and probably a million or two in engineer resources" Gratz. Next time, use C. PyPy won't solve your problem. You're writing a low latency, high performance, large throughput, super-optimized message router. Even the libs you like in Cpython are.. freaking C. Do you know why? Do you understand what happens underneath the language? Do you know why PyPy is faster? Don't get me wrong - and in fact, maybe you get me right: Go rocks. Python rocks. It's just not the tool for that very job. Today, it's still freaking C.
bipin_nag 11y ago
My take on the points mentioned: 1. Goroutine memory use - Post happens to be about 1.2 and 1.4 and I started with 1.5. 2. Debugging - yes handling errors are a bit tedious. I have not written much boilerplate to handle errors. I just copy error handlers a lot. Maybe that’s why error strings are a lot common. 3. Goroutine leaks - This is scary, I have used goroutines with channels but properly so far. Yes you can write code that leaks. This is something you will have to check yourself. 4.Testing - not done much. Overall - I feel author learned of some negative aspects about Go and turned away. Some of them like Goroutine memory footrpint will improve with time (e.g. 1.5 earned a lot of praise for GC improvement). A lot places author mentions possible improvements for e.g godebug or latest Go with SSL but did not try it as much. So it may not be as relevant to new Go adopters.
buro9 11y ago
This is interesting, there are places I would still choose Python over Go. But usually these are front-ends due to the richness of the l10n, i18n and template options that Python has. On the backend I exclusively use Go and have not seen route leak issues or some of the other things. The only thing that I do feel is that some Go code is harder to mock and fully test effectively.
hyperbovine 11y ago
This page displays exactly nothing with JavaScript disabled. I realize that no js makes me something of a Luddite, but there are solid reasons for turning it off, particularly on mobile. Is it really too much to ask of google that the text of a presentation in some way live inside HTML? Novel concept, I know.
- gaius 11y ago
  Javascript is like Flash, it's great for the author of a page to show off, or force an ad on you, but how does it actually benefit the end user? Not at all, never has.
  - Cyph0n 11y ago
    Really now. JS can potentially improve UI and UX, providing users with a smoother experience.
    - mixmastamyk 11y ago
      It can yes, but like anything else it often provides the opposite, even setting aside the download time.
    - gaius 11y ago
      I notice you are careful to say "potentially" because it never actually happens.
agentgt 11y ago
I have my own Go Heka story. I attempted to switch from Apache Flume to Heka mainly because of Flume taking vast amount of memory. I was hoping Heka would work but I think there must be some problems with the Golang AMQP drivers as memory usage would just continue to grow. This might have been my fault as I had to alter the Heka AMQP drivers to do things with the AMQP message headers. The problem was pretty simple: pull event messages from AMQP and then shove them into elastic search and file system. Heka and Flume were both sort of overkill so I decided to write it in Rust. I got extremely far but alas there were some issues with the Elastic Search Rust library that I'm still resolving. Surprisingly the AMQP library worked pretty well. I will vouch for the OP's point on error handling as Rust has a similar issue to Golang but not as bad because of the awesome type system (still I hate to admit but I really miss exceptions at times). Anyway to relate again to the OP I went back to what I know best.. boring ass Java and wrote the app in a hour or so. It took about the same memory as Heka (surprising since its Java) and appeared to be slightly faster than Heka (elastic search indexing became the bottleneck for both so take that with a grain of salt). Long story short.. I think the drivers and libraries really are the deal breakers and not so much the languages themselves (with some minor exceptions like the GIL).
unethical_ban 11y ago
If I can get a nuanced presentation out of a slide deck with no audio, you've built your slide deck incorrectly. If I can't get a nuanced presentation out of a slide deck, then maybe an article or corresponding speech is required!
mixmastamyk 11y ago
Well, I've read that it takes ten years to build great software. So golang has a while to go. Python, while not perfect has the infrastructure already built.
Myrmornis 11y ago
> Go is generally 50-100x faster than CPython (slide 20) I find that a weird sentence in an otherwise carefully written article. The author is talking about writing software which does a lot of socket IO. So I would expect the performance discussion to make some reference to this; I assume what he's talking about in the quote is the behavior of pure CPU-bound code but he doesn't discuss to what extent this is really relevant to his project.
hit8run 11y ago
What color theme and IDE is the author using :D?

366 points by azth 11y ago

CatDevURandom 11y ago

I use and like both python and go. The presentation mirrors my own experiences, though I haven't come back again yet.

kawera 11y ago
I use and like both too; webapps in python (django/flask) and microservices in go. My systems also use some large programs written in go but more like black boxes (nsq, heka, influxdb).

jbeja 11y ago

Glad that some people see the light.

hvs 11y ago

I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war stories in this industry because everything has pros and cons, and our job as software engineers is to be able to make decisions based on limited information. Case studies are great way to glean real-world experience from others without having to implement every new technology yourself in order to make high-level assumptions about that technology. This article shouldn't say to you: "See, Go is BAD, Python is GOOD!" It should say, "That's an interesting case study. If I'm working on a project that involves lots of sockets and concurrency, I'll want to take what they said into account when I'm making technology decisions."

sethammons 11y ago
I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that email now :) [edit: grammar]
- windlep 11y ago
  I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a goroutine each... you can easily exceed the memory constraint. As such, we had to put goroutine pools in place, careful select statements around them from connections to ensure we didn't overwhelm external resources, etc. It was a huge pain. It has been drastically easier to control resource usage with these constraints under python/twisted. YMMV, of course, this is just our experience. Part of the reason for putting it out there is that there are already many people who have talked/blogged about going from Python -> Go. I thought maybe the world could handle just one story about going the other direction.
  - mrits 11y ago
    I miss the time when you bragged about increasing performance without resulting in having to switch frameworks or languages.
    - windlep 11y ago
      So do I! Eventually you 'top-out' in a language/framework though... and then its all tears.
  - Jabbles 11y ago
    Typically if you wish to limit the number of goroutines you would spawn N workers and have them read from a single channel. If 20k of your incoming connections want to do something they send on the channel, without spawning a goroutine themselves. Did you try something like that?
    - windlep 11y ago
      Yep, this is what I meant by 'goroutine pools'. The select statements were on the sending side to ensure if the feed channel was full we wouldn't retain too much additional state. It works, but at that point its starting to look like an async event-loop with a thread-pool....
truncate 11y ago
Just wanted to share my own very small case study. I had a homework assignment to build a polite crawler. I initially built it in Python, and it was awfully slow. I rewrote same thing in Go, it turned out to very very fast (10x at least IIRC). I liked the fact how quickly I was able to write something so quickly (with not so shabby design) in Go with so much less experience in it. Go is definitely awesome for writing concurrent code quickly. Its not big industry story, but as a busy student I still feel great about using Go. Reason being, we had to use same crawler for doing other stuff, for which a fast crawler was really handy and saved me hours. The problems I observed with Go was that its regex seemed to be slower than Python, and memory usage was way higher. I explicitly added some GC requests.
- crdoconnor 11y ago
  Did you try pypy before rewriting?
  - truncate 11y ago
    Didn't solve my problem. PyPy still retains multi-threading and GIL stuff if I remember correctly.
    - lqdc13 11y ago
      Did you try requests_futures lib? It's all about async and network speed with crawling. Not so much cpu.
      - truncate 11y ago
        I thought it would be IO bound (that's why I started with Python at first place), but since I was extracting links as well and working a bit on graph it turned out to be more CPU intensive. But well, maybe I could have written better code, better libraries, maybe multiprocessing (would have been painful though with multiprocessing). I do admit, I didn't look much into how I could improve it within Python. I just went with Go because it was quicker that way for me.
cs702 11y ago
Amen. This post reminds me of another post I recently saw on HN, in which the author (someone with an Erlang background) lays out all sorts of reasons why he chose Ruby for a highly concurrent application that launches lots of (heavyweight) threads. Upon seeing the link on HN, my first thought was, Ruby!!?? But then I read the post and the reasons were all very sensible and practical-minded, so in that case Ruby was arguably a much better choice than Erlang, Go, Scala, Rust, etc. for a highly concurrent application. Edit: here's the post I mentioned about Ruby being used for a highly concurrent application: https://news.ycombinator.com/item?id=10394450 https://news.ycombinator.com/item?id=10394450

tinco 11y ago

I've been porting a project from Ruby to Go in search of making it a bit lighter. The project is about a 1000 lines now so I'm not allowed to criticize Go yet, but so far it's very nice. I picked the language up in just a few hours as I went and the whole project took just about a week and a half. So as someone who is clearly in no position to be criticizing other projects yet, isn't Heka exactly the sort of project you shouldn't do in Go? I say that because I have the feeling you should use Go only for very concrete cases, given its lack of proper abstractions. I.e. writing a tool that receives log lines over HTTP, extracts metrics and forwards those to StatsD? Perfect use case for Go. But writing a tool that lets you plug in arbitrary frontends to forwards to arbitrary backends? Perhaps they got it to work nice, but that sounds more like a case for a more general language.

windlep 11y ago
For heka, one of the things we wanted besides performance, was a small, easy-to-distribute binary you could drop on a system and 'just run'. Go is fantastic for distribution thanks to the small staticly linked binaries you can get out of it. To accommodate the pluggable filters and such later, a Lua sandbox system was added... and oddly some of the other Go pieces ended up being faster in Lua which is why Lua decoders/inputs are now an option for heka as well. Note: I gave this talk.
- mdup 11y ago
  Not advocating anything, but could C++ have been a good choice? You'd get top performance, small memory footprint, opt-in static linking, at the price of a less "fun" language and verbosity. Regarding memory leaks, it's easy to write a leak-free C++ program these days with smart pointers. However, it doesn't come "batteries included" for net-related stuff, and ASIO can give a couple of headaches.
  - windlep 11y ago
    It could've been. Though now that so much of heka is involved in moving data between the Lua layer, turning it into plain C is a better option. trink, one of the heka authors has a project called hindsight that does just that.

zeckalpha 11y ago

Yes, it is time for Rust.

scott_karana 11y ago
Yeah, Rust sounds like a great Python replacement. /s I do like Rust, but must we clutter every Go-related thread with comments about it? The zealousness is annoying.
- cwyers 11y ago
  Well, the last slide said "time for Rust" and the presentation was from Mozilla.
  - windlep 11y ago
    I meant that partly as a joke, I do happen to like Rust though. And in the context of this technical problem, its memory control would make it an attractive possibility (as would C/C++/etc). Lots of context missing as these are merely slides, and I said quite a bit more than what is presented here.
- DasIch 11y ago
  If you're problem is that you're memory bound, using a language that allows precise control over memory allocation and layout is certainly a good idea. The constant conflict might be annoying but using Rust or even C++ would certainly be a very reasonable choice.
- pekk 11y ago
  Well, Python-related threads have been cluttered with comments about Go for a few years now...
marvel_boy 11y ago
Sure, Rust is the way to go.

latch 11y ago

Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go. At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a complex API to work with. Go's a good language for some things. But it does nothing special or significant to close the massive productivity gap between dynamic and static language. Yes, it's terse compared to many other static language and it has stuff like implicit interfaces, but those are superficial (but nice) things when it comes to what and how you do things in dynamic land. Go might actually be a step back due to its poor type system and poor reflection capabilities. I think Go's great for a seemingly new breed of "infrastructure" systems which is becoming more important due to how systems are starting to be designed (services hosted in the cloud). It's great for building CLI which don't require your customers to install anything else. And it's good for a services / apps that needs to share memory between threads (which, in my experience, is where dynamic languages really start to fail). But for a traditional web app / service? It's horrible. At least as horrible as most static languages. It sucks for talking to databases (more than almost any other language I've used). It sucks for dealing with user-input. Like most static languages, the stuff you need to do to handle a request, which has essentially 1ms of life, is cumbersome, error prone, slow, inflexible and difficult to test.

windlep 11y ago
I mentioned during the talk, that the reason for giving it a spin in Python in the first place was that we had to fill in gaps in a Go library. While there's lots of Go libraries, many of them are immature, and have chunks of functionality missing. So while I was blocked on a coworker filling in one of the gaps we both needed, I was able to rewrite it all in Python. I'll probably give this talk again at a venue where it'll be recorded, which should add a lot of missing context to these slides.
eliben 11y ago
> Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go Precisely, and isn't this presentation the perfect example of this phenomenon? * Initially we had an implementation in language X * We then rewrote it in language Y - the lessons learned by making the system anew (this time knowing the exact problematic spots, what really to optimize for, etc) - we got a better system. Long live language Y * We then rewrote it in language X - the lessons learned [...] - we got a better system. Long live language X Good programmers can productively write good and fast code in C, Python, Java, Go, or whatever. The skill of the developer and the understanding of the problem matters much more than the programming language

zem 11y ago

i've seen a lot of these posts ending along the lines of "it's time for rust". two languages that are always conspicuous by their absence are D and ocaml. D in particular seems like it would be the logical upgrade path from python or ruby. it has a comfortably familiar C lineage, supports a variety of programming paradigms, and has good concurrency support. i wonder why people don't at least give it a look. (personal experience - i tried to use it twice, several years ago, and gave up because the tooling was bad, but from what i've heard that's very much improved today)

FraaJad 11y ago
D has replaced Python for my regular programming. D is the sweet spot for Python programmers to upgrade to without going backwards to Go (Programming language design wise) nor weighed down by all the new (and very good) stuff in Rust. D has everything from a nice IDE(Xamarin Studio), debugger, package management (Dub), statically compiled binaries, pretty decent std lib (not as good as python or Go, but very good nonetheless). I still write Python if it's a "script" that has to run on a $work server, where it is safe to assume that Python would be available and sufficient for most tasks.
- ylh 11y ago
  People saying "backwards to Go" instantly reminds me of the following quote: Are you quite sure that all those bells and whistles, all those wonderful facilities of your so called powerful programming languages, belong to the solution set rather than the problem set? — Edsger W. Dijkstra
  - FraaJad 11y ago
    Interestingly, ALGOL failed because it was too complicated to implement.. And Djikstra played a huge role in the formulation of that language. Why aren't you using ALGOL?
    - ylh 11y ago
      ALGOL-60 saw fine use in its day; Dijkstra criticized ALGOL-68, which indeed failed because it was too complicated to implement.

falcolas 11y ago

I'll be interested in coming back to Python when it isn't a headache to deploy into production. I'm tired of an install requiring a GCC compiler on the target node. I'm also tired of having to work around the language and ecosystem to avoid dependency hell.

rattray 11y ago
docker?
- falcolas 11y ago
  Doesn't solve the issue of needing a C compiler for third party extensions, and definitely qualifies as a work-around for the existing toolset. Yes, it helps. But you can use Docker with Go programs as well (and drop a lot more of the base image in the process).
  - bmurphy1976 11y ago
    Actually, nice thing about docker is you can build a compilation container (pre-built with all your C/C++ apps ready to go and shared amongst your coworkers), compile your extensions using that, and then only install them into your target container (sans compilation tools). It's a little more grunt work that way, but you get better control and reproducibility without the explosion in image sizes.
  - zweiterlinde 11y ago
    The way we do this is to have a base image that has already yum installed or pip installed all modules (non trivial, anyway) that our package needs. Then the docker image that needs to be rebuilt (that depends on the first one) is just a minimal pip install away.
thristian 11y ago
The way I deploy Python apps at $EMPLOYER: - CI system detects a commit and checks out the latest code - CI system makes a virtualenv and sets up the project and its dependencies into it with "pip install --editable path/to/checkout" - CI system runs tests, computes coverage, etc. - CI system makes a output directory and populates it with "pip wheel --wheel-dir path/to/output path/to/checkout" - Deployment system downloads wheels to a temporary location - Deployment system makes a virtualenv in the right location - Deployment system populates virtualenv with "pip install --no-deps path/to/temp/location/*.whl" The target node only needs a compatible build of python and the virtualenv package installed; it doesn't need a compiler and only needs a network connection if you want to transfer wheel files that way.

mratzloff 11y ago

There are people who have been working in Go for years, successfully, but who don't post comments with the same frequency and dogged determination as the middlebrow dismissers. Both Python and Go are fine. They both have their strengths and weaknesses. I personally wouldn't write a web app in Go (at least, anything beyond the most basic admin interface). I also personally wouldn't write a very large and complex Python system given the huge unit testing burden necessary to ensure safe refactoring down the road. The biggest reason I like Go is because it makes it really hard for engineers to create huge, complex abstractions. Engineers (and especially less experienced engineers) just love them some abstractions. In my experience, most abstractions aren't justifiable. The net effect is that it usually makes their software harder to learn, harder to debug, inflexible (ironically), and late for whatever deadline they were supposed to hit. You can't write Java enterprise software in Go, and I really appreciate that.

rattray 11y ago
Do you have any increased optimism about the refactorability of python given the recent addition of type annotations?
- mratzloff 11y ago
  Definitely, but given the relative recency my comment still stands. Ultimately, all libraries in use in a given application must support this as well for it to be truly comprehensive. As a side note, sometimes you do need that performance gain, without wanting to resort to C or C++. I hope to see Python make some gains there with the addition of the type information.
- xj9 11y ago
  I would really like to see a type check package that you can run alongside your unit tests to verify type correctness. MyPy is nice, but I don't want to do type checks at runtime.
anacrolix 11y ago
> You can't write Java enterprise software in Go, and I really appreciate that. You can, and I've seen it, unfortunately.

stock_toaster 11y ago

The claim that pypy uses less memory than Go seems...rather extraordinary. I worked with a fairly complex http api app that ran as a rather svelte wsgi framework under gunicorn, and we saw at least 10x or more increase in memory usage than cpython when we switched to pypy, once the jit was fully warmed up (memory usage seemed to hit steady state after about an hour). pypy has also historically (in my experience) been a bit more "lazy" (deferring GC of individual objects longer) than cpython when an object falls out of scope. My general rule of thumb for pypy has historically been that you trade memory (requires more of it) for speed (faster) when compared against cpython. Maybe the reimplementation itself was just far more efficient? Hopefully the talk itself clarified that point. I would be very interested in hearing more about that particular aspect.

windlep 11y ago
This was before Go 1.4, which dropped memory use per goroutine from 8kb -> 2kb. They're fairly close now (minus my leaking goroutines). ;)
DasIch 11y ago
PyPy's JIT requires quite a bit of memory. CPython doesn't have a JIT so it doesn't share that problem. However PyPy also has optimizations specifically to keep memory usage down. Especially if you have a lot of objects, it's certainly possible to consume less memory than CPython. PyPy using less memory than Go does seem weird but depending on how much the GCs differ and how they are configured it could simply be that Go's GC doesn't give up memory as freely to the OS.
dbaupp 11y ago
I don't know the details, but the memory difference was almost certainly due to the different approaches to concurrency. Python coroutines just need to save their exact stack frame while a Go goroutine will spawn a "massive" stack that likely has much more space than needed. The first case might only need tens or hundreds of bytes for a dozen local variables, while the latter case is a fixed overhead of several KB (8 previously, but 2 with Go 1.4 according to the slides). The JIT memory is constant at runtime (proportional(-ish) to the amount of code, which is fixed) while it is desirable to have the number of coroutines be as large as possible.

JulianMorrison 11y ago

Huge interfaces are not how you do it. Little ones, each expressing a conceptual unit of the functionality. Loggable. MessageConsumer. Stuff like that, even if it only wraps one method. Make each major subsystem into an object, connect to it through the mini-interfaces it provides, and test its consumers by stubbing those mini-interfaces, not the whole object.

ntrepid8 11y ago

This echoes some of my own experiences. Python (even CPython) waits on the network just as fast as Go (or anything else). I've had good luck writing applications in Python, then profiling them and implementing critical sections that are CPU bound in C as modules. Any sections that are memory hogs can be converted to stream processors. Lately I've started implementing the modules with Rust and the results are promising. It seems like a nice balance of developer productivity and application performance.

eugenekolo2 11y ago

Does anybody have some comments on the closing statements on Google's 10KB secret?

nulltype 11y ago
I suppose there's the relevant blog post https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html https://www.imperialviolet.org/2010/06/25/overclocking-ssl.h...
- spoiler 11y ago
  Is it possible that it's just a typo, though?
  - nulltype 11y ago
    That would be pretty hilarious.
  - eugenekolo2 11y ago
    Not sure what you mean, but the site linked above says they do it <10KB and then a few paragraphs down says how they patched it to go down to a bit more than 5KB. So, I don't think it's a typo.

windlep 11y ago

I have to note this, because I think it deserves quite a bit more attention. SSL is extremely expensive, on RAM (perhaps the implementations have optimized for throughput over RAM). I have yet to benchmark any SSL implementation in any language, with any binding, that can use less than 20kb per SSL connection. I mentioned in my talk here that SSL is very expensive, here's my benchmark suite that others may add to: https://github.com/bbangert/ssl-ram-testing/ https://github.com/bbangert/ssl-ram-testing/ I have implementations in several languages, so far both Go and Python 3.4 can get as low as the 20kb cited. If you can get your per-connection state below 20kb, then merely adding SSL means doubling or worse your RAM requirements, which is huge. I appreciate that everyone loves obsessing on the language wars, but the SSL RAM overhead affects us regardless of language. I covered that in one of the slides near the end, it'd be great to see some movement on reducing the RAM footprint here.

nulltype 11y ago
Do you have results for the different tests? I did not see any in the repo.
icot 11y ago
Do you have any insight about the mention in the slides of Google having a 10kb solution?
ssfak 11y ago
Doesn't a TLS terminator proxy solve this? E.g. I usually put my application services behind HTTPS-enabled nginx and it works wonderfully.
- windlep 11y ago
  Nope, so, the goal here is to reduce how many machines (each with their own RAM limits) are used. The task is holding open bidirectional SSL wrapped long-lived websocket connections. They're held open for hours at a time, since we need to send notifications when we get them. Every connection has a base cost of the TCP kernel send/recv buffer, which in our case we dropped a bit to 4kb each. So that's still 8kb per connection right there. If we terminate the SSL on a separate machine from where we handle the connection, then it means we'll be using 8kb more memory per connection. Probably even greater because nginx has its own send/recv buffers for data. I'm sure our use-case is a unique one, most people care about raw through-put so the majority of SSL optimization has focused on lowering CPU use under high load rather than memory use under massive amounts of connections.

0x434D53 11y ago

Go is one of my working languages. Like with every other language (Python, OCAML, C#, some Java, Swift) i have a love-hate relationship with go. What i agree on: What i agree on with the presentation: Concurrency using goroutines and channels is f... hard besides very primitive scenarios. Even fork-join isn't that easy. There also the lack of expressiveness hurts: It's nearly impossible to build higher level abstractions above the channels/goroutines. You always have to do the bookkeeping of your goroutines. I also agree on the error-handling problems: It's often hard to locate errors. It requires a big amount of discipline by the programmers to achieve some kind of ability to locate errors. No, i don't want the Java/C#-'i throw exceptions everywhere'-style back, but Go is the other extreme. Some more lightweight-panic wouldn't be bad. What i can't agree on: That you cannot mock without interfaces in Go is typically not a problem: There is no real encapsulation (_, ., no constructors) so in many cases you just an instantiate your structs as you need them. The classic for mocking - time - is problematic as in every language. IO is typically behind the various io.Reader/Writer... interfaces: No problems there. The criticism about memory consumption i don't get: Every system i saw ported to go from Java, Ruby or Python had a much lower memory footprint than before. And typically go allows to optimize allocation quite well when needed.

fauigerzigerk 11y ago
I agree that error handling in Go is a headache. But since we are forced to handle every single error where it occurs, we can at least make the best of it and add context information before returning it. I'm using a small utility library to wrap the original error and add function name, line, file name and optionally a descriptive message that explains what failed.
- 0x434D53 11y ago
  You are not forced to handle errors. Or do you check for the errors returned by fmt.Print? I guess not. This was my point: In many areas go forces the developer to the right thing (no unnecessary imports, gofmt...) and does not rely on developers discipline. But when i comes to error handling it does. What i would wish for is some extended error handling supported by the compiler. I don't want a stack trace, but the compiler easily could produce for example a line number where the error was returned.

gtrubetskoy 11y ago

I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.

windlep 11y ago
Why would it? Using twisted (or py3's asyncio), the event loop is spinning, there is no thread for Python to be switching to. Except that we're using a thread-pool of course (to make external blocking network calls), and those threads are.... making network requests, which is I/O, and the GIL releases during that. On a network bound daemon, which is not CPU-bound, the GIL is really not an issue, so it never came up.
- gtrubetskoy 11y ago
  "On a network bound daemon, which is not CPU-bound" - I think that's a fairly unique constraint which makes your case special. For me, the GIL is unfortunately where Python is unable to compete with Go.
alextgordon 11y ago
node.js has a GIL and nobody talks about that either. The GIL is not relevant because Python is so slow that running it in two threads is hardly an improvement.
- gtrubetskoy 11y ago
  But don't you need to run a bunch of node.js instances in production on a multi-cpu system behind nginx or haproxy? Would it be necessary had it not been for the GIL?
  - alextgordon 11y ago
    Most node.js code spends barely any time on the v8 thread. The node code merely invokes routines in libuv, which has a thread pool for anything computationally intensive.
- 11y ago

giaosudau 11y ago

btw, python with pypy gonna be a bright future.

bootload 11y ago

"Go experts still can't write leak-free code" Big statement, anyone confirm this?

dbaupp 11y ago
The next two slides have screenshots of issue tracker searches for "goroutine leak".
- bootload 11y ago
  yeah I saw those, what about examples from independent people reading the article on HN?
codygman 11y ago
Now I know to ask about goroutine leaks next time my colleague who loves Go talks about how bad space leaks are in Haskell :P

Animats 11y ago

What's unusual is that this project uses PyPy in production. PyPy has been a long time coming. Until recently, Python was defined by CPython, which is a naive interpreter. As this article points out, performance is roughly an order of magnitude better with PyPy. Now PyPy has to be taken as seriously as CPython, if not more seriously. In a few years we may look at CPython as the "toy" implementation.

sandGorgon 11y ago

what is strange is that everyone seems to be using Python 2.7 in some form. There is all this new work being done in Python 3 asyncio but there is nothing that is being used... and this is from a guy who pretty is a core developer in several python projects. I look at Ruby or Go... or even Java and every new language feature has a much more rapider adoption curve. is this a pretty solid statement that the entire Python 3 + asyncio path is a dead end ?

windlep 11y ago
We used Python 2.7 for pypy compatibility. I really dig the new async/await primitives added to Python 3.5 and think the asyncio path is the best path forward. I hope the work to get all the latest asyncio stuff working on PyPy picks up some steam.
- sandGorgon 11y ago
  hey - thanks for replying. if Pypy + asyncio was available, would you have built everything using that stack ? There have been all these benchmarks that asyncio is so much slower than threads [1] How would you compare that with Go ? [1] http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
  - kintamanimatt 11y ago
    For certain things that they probably shouldn't be used for, not across the board.
- bpicolo 11y ago
  The main issue with asyncio is it is incompatible with some of the best packages in the ecosystem atm, e.g sqlalchemy
broodbucket 11y ago
The difference is that Python 3 is far more backwards incompatible than an update to Ruby, Go or Java. If there's any component of your Python project that was written for Python 2 and it hasn't been ported, your project as a whole is probably staying in Python 2. I certainly wouldn't call the Python 3 path "dead", as starting a new project you'd be silly not to use Python 3. It's just a very slow process. Also, porting from 2 to 3 isn't even that hard (there's a script that does 95% of it for you).

zobzu 11y ago

"drank koolaid, used tool without knowing what its good at, lost 2 years and probably a million or two in engineer resources" Gratz. Next time, use C. PyPy won't solve your problem. You're writing a low latency, high performance, large throughput, super-optimized message router. Even the libs you like in Cpython are.. freaking C. Do you know why? Do you understand what happens underneath the language? Do you know why PyPy is faster? Don't get me wrong - and in fact, maybe you get me right: Go rocks. Python rocks. It's just not the tool for that very job. Today, it's still freaking C.

bipin_nag 11y ago

My take on the points mentioned: 1. Goroutine memory use - Post happens to be about 1.2 and 1.4 and I started with 1.5. 2. Debugging - yes handling errors are a bit tedious. I have not written much boilerplate to handle errors. I just copy error handlers a lot. Maybe that’s why error strings are a lot common. 3. Goroutine leaks - This is scary, I have used goroutines with channels but properly so far. Yes you can write code that leaks. This is something you will have to check yourself. 4.Testing - not done much. Overall - I feel author learned of some negative aspects about Go and turned away. Some of them like Goroutine memory footrpint will improve with time (e.g. 1.5 earned a lot of praise for GC improvement). A lot places author mentions possible improvements for e.g godebug or latest Go with SSL but did not try it as much. So it may not be as relevant to new Go adopters.

buro9 11y ago

This is interesting, there are places I would still choose Python over Go. But usually these are front-ends due to the richness of the l10n, i18n and template options that Python has. On the backend I exclusively use Go and have not seen route leak issues or some of the other things. The only thing that I do feel is that some Go code is harder to mock and fully test effectively.

hyperbovine 11y ago

This page displays exactly nothing with JavaScript disabled. I realize that no js makes me something of a Luddite, but there are solid reasons for turning it off, particularly on mobile. Is it really too much to ask of google that the text of a presentation in some way live inside HTML? Novel concept, I know.

gaius 11y ago
Javascript is like Flash, it's great for the author of a page to show off, or force an ad on you, but how does it actually benefit the end user? Not at all, never has.
- Cyph0n 11y ago
  Really now. JS can potentially improve UI and UX, providing users with a smoother experience.
  - mixmastamyk 11y ago
    It can yes, but like anything else it often provides the opposite, even setting aside the download time.
  - gaius 11y ago
    I notice you are careful to say "potentially" because it never actually happens.

agentgt 11y ago

I have my own Go Heka story. I attempted to switch from Apache Flume to Heka mainly because of Flume taking vast amount of memory. I was hoping Heka would work but I think there must be some problems with the Golang AMQP drivers as memory usage would just continue to grow. This might have been my fault as I had to alter the Heka AMQP drivers to do things with the AMQP message headers. The problem was pretty simple: pull event messages from AMQP and then shove them into elastic search and file system. Heka and Flume were both sort of overkill so I decided to write it in Rust. I got extremely far but alas there were some issues with the Elastic Search Rust library that I'm still resolving. Surprisingly the AMQP library worked pretty well. I will vouch for the OP's point on error handling as Rust has a similar issue to Golang but not as bad because of the awesome type system (still I hate to admit but I really miss exceptions at times). Anyway to relate again to the OP I went back to what I know best.. boring ass Java and wrote the app in a hour or so. It took about the same memory as Heka (surprising since its Java) and appeared to be slightly faster than Heka (elastic search indexing became the bottleneck for both so take that with a grain of salt). Long story short.. I think the drivers and libraries really are the deal breakers and not so much the languages themselves (with some minor exceptions like the GIL).

unethical_ban 11y ago

If I can get a nuanced presentation out of a slide deck with no audio, you've built your slide deck incorrectly. If I can't get a nuanced presentation out of a slide deck, then maybe an article or corresponding speech is required!

mixmastamyk 11y ago

Well, I've read that it takes ten years to build great software. So golang has a while to go. Python, while not perfect has the infrastructure already built.

Myrmornis 11y ago

> Go is generally 50-100x faster than CPython (slide 20) I find that a weird sentence in an otherwise carefully written article. The author is talking about writing software which does a lot of socket IO. So I would expect the performance discussion to make some reference to this; I assume what he's talking about in the quote is the behavior of pure CPU-bound code but he doesn't discuss to what extent this is really relevant to his project.

hit8run 11y ago

What color theme and IDE is the author using :D?