7 ms·
I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war s
by hvs 11y ago
I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war stories in this industry because everything has pros and cons, and our job as software engineers is to be able to make decisions based on limited information. Case studies are great way to glean real-world experience from others without having to implement every new technology yourself in order to make high-level assumptions about that technology.
This article shouldn't say to you: "See, Go is BAD, Python is GOOD!" It should say, "That's an interesting case study. If I'm working on a project that involves lots of sockets and concurrency, I'll want to take what they said into account when I'm making technology decisions."
- sethammons 11y agoI should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that email now :) [edit: grammar]
- windlep 11y agoI should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a goroutine each... you can easily exceed the memory constraint. As such, we had to put goroutine pools in place, careful select statements around them from connections to ensure we didn't overwhelm external resources, etc. It was a huge pain. It has been drastically easier to control resource usage with these constraints under python/twisted. YMMV, of course, this is just our experience. Part of the reason for putting it out there is that there are already many people who have talked/blogged about going from Python -> Go. I thought maybe the world could handle just one story about going the other direction.
- mrits 11y agoI miss the time when you bragged about increasing performance without resulting in having to switch frameworks or languages.
- windlep 11y agoSo do I! Eventually you 'top-out' in a language/framework though... and then its all tears.
- Jabbles 11y agoTypically if you wish to limit the number of goroutines you would spawn N workers and have them read from a single channel. If 20k of your incoming connections want to do something they send on the channel, without spawning a goroutine themselves. Did you try something like that?
- windlep 11y agoYep, this is what I meant by 'goroutine pools'. The select statements were on the sending side to ensure if the feed channel was full we wouldn't retain too much additional state. It works, but at that point its starting to look like an async event-loop with a thread-pool....
- zimbatm 11y agoHow do these 20k connection feed the channel without being themselves managed by goroutines ? One thing I wish was possible in go is being able to use the `select` keyword with both channels and IO.
- trentnelson 11y agoWhy 10-20% of RAM? How much RAM does each machine have? What else are they doing? Are they virtualized?
- benji-york 11y agoHe said "within 10-20% of the machines RAM", i.e., utilizing 80-90% of the machine's RAM, without exceeding it.
- truncate 11y agoJust wanted to share my own very small case study. I had a homework assignment to build a polite crawler. I initially built it in Python, and it was awfully slow. I rewrote same thing in Go, it turned out to very very fast (10x at least IIRC). I liked the fact how quickly I was able to write something so quickly (with not so shabby design) in Go with so much less experience in it. Go is definitely awesome for writing concurrent code quickly. Its not big industry story, but as a busy student I still feel great about using Go. Reason being, we had to use same crawler for doing other stuff, for which a fast crawler was really handy and saved me hours. The problems I observed with Go was that its regex seemed to be slower than Python, and memory usage was way higher. I explicitly added some GC requests.
- crdoconnor 11y agoDid you try pypy before rewriting?
- truncate 11y agoDidn't solve my problem. PyPy still retains multi-threading and GIL stuff if I remember correctly.
- lqdc13 11y agoDid you try requests_futures lib? It's all about async and network speed with crawling. Not so much cpu.
- truncate 11y agoI thought it would be IO bound (that's why I started with Python at first place), but since I was extracting links as well and working a bit on graph it turned out to be more CPU intensive. But well, maybe I could have written better code, better libraries, maybe multiprocessing (would have been painful though with multiprocessing). I do admit, I didn't look much into how I could improve it within Python. I just went with Go because it was quicker that way for me.
- cs702 11y agoAmen. This post reminds me of another post I recently saw on HN, in which the author (someone with an Erlang background) lays out all sorts of reasons why he chose Ruby for a highly concurrent application that launches lots of (heavyweight) threads. Upon seeing the link on HN, my first thought was, Ruby!!?? But then I read the post and the reasons were all very sensible and practical-minded, so in that case Ruby was arguably a much better choice than Erlang, Go, Scala, Rust, etc. for a highly concurrent application. Edit: here's the post I mentioned about Ruby being used for a highly concurrent application: https://news.ycombinator.com/item?id=10394450 https://news.ycombinator.com/item?id=10394450