7 ms·

So You Wanna Go Fast?

95 points by tylertreat 11y ago

ej_campbell 11y ago
>Ultimately, you hit the ceiling of what you can do with it, and you have to migrate off (if you’re a business that is trying to grow, anyway). I read this post and the one that was linked to and I still don't understand what the problem was that they faced with Google App Engine. A few specifics would have been helpful.
- lancefisher 11y ago
  So he hinted at it with, "App Engine was stateless". You can't hold a model in memory on the server with App Engine. You can if you own the server. Also, App Engine has request length limits (like many platforms). This means you can't hold a persistent connection for real time communication, and you can't run long tasks. Even using App Engine Backends or Tasks are time-limited to about 10 minutes, and your long-running task will often be killed unexpectedly. Last I checked, App Engine was using an old version of Go, and I don't know if they'll ever move off Python 2.7. As a platform App Engine is great, but it's not the perfect fit for everything. Disclosure: I used to work for Workiva.
  - ec109685 11y ago
    Thanks for the clarification. He advocated stateless anyway, but the other limits you describe could make things hard and preclude any type of actor model.
mhurron 11y ago
If you're not first, you're last.
snikch 11y ago
My favourite bit of this article was where they pad the struct to move each field onto separate cpu cache lines and see a 15% speed increase.
- anon4 11y ago
  At that point maybe you should think about switching to using a struct of arrays rather than an array of structs.
amscanne 11y ago
Replacing a channel with a ring buffer that spins (and uses runtime.Gosched every 1000 failed attempts) is really bad general advice. The other advice is similar. It may be a good idea for very specific use cases and might have served the author well, but the channel will scale far better in the general case and suffer from have less degenerative cases. Looking at artificial benchmarks is very misleading. If you take the ring buffer linked in the article and throw 1000 producers and one consumer at it, it will completely and utterly fall over. Channels on the other hand, will behave just fine. My take: use the language primitives unless you really know what you're doing. Then you should also know that it's not a general optimization.
- kasey_junk 11y ago
  The problem is 2 fold. The marketing on Go is that it has great concurrency built in, but it doesn't support 2 extremely common concurrency use cases out of the box: 1) lock free queues 2) concurrent maps Worse than that, the language makes it very difficult to write libraries to support those use cases. So sure, in your use case, and maybe the Golang language maintainers use cases, dispensing with channels is bad advice. But it turns out that nearly every single team that has tried to scale golang servers has come to the same position. Give up channels as your concurrency primitives and revert to the sync package if you need to really scale golang programs
  - amscanne 11y ago
    Assuming you've got a buffered channel with non-blocking send and receive calls, I seriously doubt that the three lines protected by a mutex are the bottleneck for scaling your service. It's basically exchanging one set of atomic ops for another one. I'm not saying that you couldn't optimize if you have a very specific need, but it's disingenuous to pretend that the lack of a "lock-free" queue is somehow a fatal flaw or preventing you from scaling go programs.
  - reality_czech 11y ago
    Normally if you have too much overhead with a channel, you just send more at a time. For example if you are using a channel to distribute out work, send 100 items at a time rather than 1. Then you do fewer channel read and write operations. Concurrent maps have poor performance because of cache-line bouncing and locking overhead. Everyone I've ever talke to who has benchmarked them has said there isn't much point.
marssaxman 11y ago
This article became hilarious when I reached "App Engine was stateless, so the natural tendency once you're off it was to make everything stateful. And we did." Oh my.
kasey_junk 11y ago
> When performance really matters, you have to start thinking at the hardware level. Formula One driver Jackie Stewart is famous for once saying, “You don’t have to be an engineer to be be a racing driver, but you do have to have mechanical sympathy.” If you are going to reference that quote, talk about ring buffers, and false sharing. I think it behooves you to credit Martin Thompson or LMAX in some way. That is a near verbatim quote from the Disruptor papers.
- jonathanoliver 11y ago
  That was the same thought I had when I read through the ring buffer section. One glaring problem in the Golang space is the memory model doesn't offer Happens-Before guarantees. I wrote a project called Go-Disruptor (github.com/smartystreets/go-disruptor) and found that I can't guarantee ordering of writes and reads between threads which is absolutely critical for a ring buffer implementation.
  - jzelinskie 11y ago
    Would you mind expanding on this? I've never implemented a ring buffer and am genuinely curious.
  - anon4 11y ago
    I haven't written Go, but from what I've heard it lets you mix in platform-specific assembly. Can't you use that to put in memory barriers and/or issue the specific load/store instructions you need?
    - theparanoid 11y ago
      You can, the Go runtime and standard library use assembly for a number of operations. A drawback is difficulty reading and reasoning about assembly. C style intrinsics are easier.
  - deleted 11y ago
    [deleted]
  - 11y ago
carsongross 11y ago
OK, this is completely off topic, but this sentence start from the article jumped out at me: It’s honestly a good question to ask Why is it that so many people now start sentences with some variation of "honestly"? I see it in comments here on HN all the time, and in lots of tech blog posts. Is there a new paper out on how this is effective rhetoric? Is it a regional tick I haven't figured out yet? Honestly, I find it depressing that so many people feel the need to preface what they are saying with a declaration that they are not, in fact, lying to us.
- abiox 11y ago
  are many people really doing that? honestly, perhaps it's some cognitive bias or frequency illusion.
  - carsongross 11y ago
    To be honest, I can't tell. Maybe it's just me, honestly.
- danieltillett 11y ago
  I have always thought this was code for “I was previously lying to you, but now I am telling you the truth”. I think not.
- hamburglar 11y ago
  I have seen this specific complaint on HN several times lately. It's a figure of speech. It means "frankly." Your refusal to pick up on English language idioms -- illogical as they may be -- is, frankly, not interesting. Can we stop?
  - TelmoMenezes 11y ago
    There's a fine line between idiom and annoying cliché. "Can we stop doing X" is another one.
- anon4 11y ago
  It's a verbal tick, a meaningless bit at the start of the sentence because the person feels uncomfortable starting the sentence right off and wants a bit of lead-in, or as a short pause for rhythm. Same thing with "basically" - I've seen people start basically every single sentence with "basically" when they're even a bit stressed.
vvanders 11y ago
It seems like if you're going to care about performance this much you might want to use a language that doesn't discriminate between heap and stack. Sure you can do all these gymnastics, but why not use tools that were built to do this from the start like Rust and C++?
0x1997 11y ago
Seriously, if you care about lock-free algorithms, zero-cost abstractions, control of memory management and avoiding false sharing, why use Go in the first place? C/C++ and Rust are much more practical choices, even Java has mature concurrent data structures. IMHO, Go is a sensible choice to achieve 80/100 raw performance. Beyond that, Go is just not the right tool.

95 points by tylertreat 11y ago

>Ultimately, you hit the ceiling of what you can do with it, and you have to migrate off (if you’re a business that is trying to grow, anyway). I read this post and the one that was linked to and I still don't understand what the problem was that they faced with Google App Engine. A few specifics would have been helpful.

lancefisher 11y ago
So he hinted at it with, "App Engine was stateless". You can't hold a model in memory on the server with App Engine. You can if you own the server. Also, App Engine has request length limits (like many platforms). This means you can't hold a persistent connection for real time communication, and you can't run long tasks. Even using App Engine Backends or Tasks are time-limited to about 10 minutes, and your long-running task will often be killed unexpectedly. Last I checked, App Engine was using an old version of Go, and I don't know if they'll ever move off Python 2.7. As a platform App Engine is great, but it's not the perfect fit for everything. Disclosure: I used to work for Workiva.
- ec109685 11y ago
  Thanks for the clarification. He advocated stateless anyway, but the other limits you describe could make things hard and preclude any type of actor model.

mhurron 11y ago

If you're not first, you're last.

snikch 11y ago

My favourite bit of this article was where they pad the struct to move each field onto separate cpu cache lines and see a 15% speed increase.

anon4 11y ago
At that point maybe you should think about switching to using a struct of arrays rather than an array of structs.

amscanne 11y ago

Replacing a channel with a ring buffer that spins (and uses runtime.Gosched every 1000 failed attempts) is really bad general advice. The other advice is similar. It may be a good idea for very specific use cases and might have served the author well, but the channel will scale far better in the general case and suffer from have less degenerative cases. Looking at artificial benchmarks is very misleading. If you take the ring buffer linked in the article and throw 1000 producers and one consumer at it, it will completely and utterly fall over. Channels on the other hand, will behave just fine. My take: use the language primitives unless you really know what you're doing. Then you should also know that it's not a general optimization.

kasey_junk 11y ago
The problem is 2 fold. The marketing on Go is that it has great concurrency built in, but it doesn't support 2 extremely common concurrency use cases out of the box: 1) lock free queues 2) concurrent maps Worse than that, the language makes it very difficult to write libraries to support those use cases. So sure, in your use case, and maybe the Golang language maintainers use cases, dispensing with channels is bad advice. But it turns out that nearly every single team that has tried to scale golang servers has come to the same position. Give up channels as your concurrency primitives and revert to the sync package if you need to really scale golang programs
- amscanne 11y ago
  Assuming you've got a buffered channel with non-blocking send and receive calls, I seriously doubt that the three lines protected by a mutex are the bottleneck for scaling your service. It's basically exchanging one set of atomic ops for another one. I'm not saying that you couldn't optimize if you have a very specific need, but it's disingenuous to pretend that the lack of a "lock-free" queue is somehow a fatal flaw or preventing you from scaling go programs.
- reality_czech 11y ago
  Normally if you have too much overhead with a channel, you just send more at a time. For example if you are using a channel to distribute out work, send 100 items at a time rather than 1. Then you do fewer channel read and write operations. Concurrent maps have poor performance because of cache-line bouncing and locking overhead. Everyone I've ever talke to who has benchmarked them has said there isn't much point.

marssaxman 11y ago

This article became hilarious when I reached "App Engine was stateless, so the natural tendency once you're off it was to make everything stateful. And we did." Oh my.

kasey_junk 11y ago

> When performance really matters, you have to start thinking at the hardware level. Formula One driver Jackie Stewart is famous for once saying, “You don’t have to be an engineer to be be a racing driver, but you do have to have mechanical sympathy.” If you are going to reference that quote, talk about ring buffers, and false sharing. I think it behooves you to credit Martin Thompson or LMAX in some way. That is a near verbatim quote from the Disruptor papers.

jonathanoliver 11y ago
That was the same thought I had when I read through the ring buffer section. One glaring problem in the Golang space is the memory model doesn't offer Happens-Before guarantees. I wrote a project called Go-Disruptor (github.com/smartystreets/go-disruptor) and found that I can't guarantee ordering of writes and reads between threads which is absolutely critical for a ring buffer implementation.
- jzelinskie 11y ago
  Would you mind expanding on this? I've never implemented a ring buffer and am genuinely curious.
- anon4 11y ago
  I haven't written Go, but from what I've heard it lets you mix in platform-specific assembly. Can't you use that to put in memory barriers and/or issue the specific load/store instructions you need?
  - theparanoid 11y ago
    You can, the Go runtime and standard library use assembly for a number of operations. A drawback is difficulty reading and reasoning about assembly. C style intrinsics are easier.
- deleted 11y ago
  [deleted]
- 11y ago

carsongross 11y ago

OK, this is completely off topic, but this sentence start from the article jumped out at me: It’s honestly a good question to ask Why is it that so many people now start sentences with some variation of "honestly"? I see it in comments here on HN all the time, and in lots of tech blog posts. Is there a new paper out on how this is effective rhetoric? Is it a regional tick I haven't figured out yet? Honestly, I find it depressing that so many people feel the need to preface what they are saying with a declaration that they are not, in fact, lying to us.

abiox 11y ago
are many people really doing that? honestly, perhaps it's some cognitive bias or frequency illusion.
- carsongross 11y ago
  To be honest, I can't tell. Maybe it's just me, honestly.
danieltillett 11y ago
I have always thought this was code for “I was previously lying to you, but now I am telling you the truth”. I think not.
hamburglar 11y ago
I have seen this specific complaint on HN several times lately. It's a figure of speech. It means "frankly." Your refusal to pick up on English language idioms -- illogical as they may be -- is, frankly, not interesting. Can we stop?
- TelmoMenezes 11y ago
  There's a fine line between idiom and annoying cliché. "Can we stop doing X" is another one.
anon4 11y ago
It's a verbal tick, a meaningless bit at the start of the sentence because the person feels uncomfortable starting the sentence right off and wants a bit of lead-in, or as a short pause for rhythm. Same thing with "basically" - I've seen people start basically every single sentence with "basically" when they're even a bit stressed.

vvanders 11y ago

It seems like if you're going to care about performance this much you might want to use a language that doesn't discriminate between heap and stack. Sure you can do all these gymnastics, but why not use tools that were built to do this from the start like Rust and C++?

0x1997 11y ago

Seriously, if you care about lock-free algorithms, zero-cost abstractions, control of memory management and avoiding false sharing, why use Go in the first place? C/C++ and Rust are much more practical choices, even Java has mature concurrent data structures. IMHO, Go is a sensible choice to achieve 80/100 raw performance. Beyond that, Go is just not the right tool.