3 ms·

Forgive my ignorance, but I have a very general question. The article says Go runs on millions of cores at Uber. Obviously, the company is interested in maximiz

Forgive my ignorance, but I have a very general question. The article says Go runs on millions of cores at Uber. Obviously, the company is interested in maximizing the performance, hence the Pprof++ project. My experience with Go is very limited, but in my tests it was always slower than C. Sometimes just a bit, sometimes 2-5 times. So my question is: looking back, don't you guys regret choosing this language? Please don't misunderstand me, I don't intend to start any flamewars, but it seems like you're very much focused on CPU-bound performance, and the choice of language is not neutral in this case.

domano 5y ago
I think finding a good balance between performance and development time & risks is important and C does not seem like a very good choice regarding the second part. At least for web stuff.
studmuffin650 5y ago
Full Disclosure: I do not work at Uber but at another large enterprise that is at Ubers scale and is almost 100% Go across 100s of micro services. Inside an enterprise, there’s more to a language then just the performance (though that is a large factor). You also have to take into account existing tooling (both internal and external), developer experience, whether you can find enough developers to code it and source code maintainability, as well as many other common concerns with a language. Most languages will do well in a few cases but none are best in class in all cases (or everyone would use it). Go does well enough in the performance category while also doing moderately to extremely well in other categories. In CPU bound tasks that don’t rely on CGo, go does extremely well in my experience. I think in general though, for most enterprises, Go strikes a happy medium and makes the right trade offs that most developers are willing to make.
hu3 5y ago
I doubt most of Uber microservices are CPU bound. Yes it's natural for a fleet of this size to also be scrutinized for CPU savings but that's not the only cost involved when choosing a language. Good tooling, dev experience and availability of competent engineers comes to mind.
- akshayshah 5y ago
  At least when I was at Uber (left in mid-2019), this was absolutely the case: most services weren't CPU-bound, but a few were. Rewriting the CPU-bound services in C++, Rust, or some other language might have helped, but would have required a _ton_ of well-understood, reliable libraries for networking, observability, authn/authz, database access, etc. In nearly all cases, there was plenty of room to make the Go service faster. A more careful choice of data structure and algorithm, finer-grained locking, fan-out across a goroutine pool, or just avoiding a zillion heap allocations solved most problems. I don't recall any cases where Go was simply incapable of meeting our performance needs. As a side benefit, services with more stringent performance requirements often exposed inefficiencies in widely-used libraries. Optimizing those libraries made every Go service faster and cut our overall compute spend. Avoiding rewrites in C++ or Rust let those wins add up over time.
heleninboodler 5y ago
As both a C and Go developer, in my experience, 2-5x execution time increase going from C to Go does not sound reasonable at all.
arp242 5y ago
Facebook managed to optimize mercurial (Python) to be faster than git (C) for their huge monorepo use case simply because Python made it so much easier to implement more efficient algorithms. If you have some fairly simple function/task, then yeah, a C version will probably blow the Go version away almost all of the time. But that's not necessarily indicative of real-world performance of a full application. And of course, there are other interests than "maximize performance" too, such as developer efficiency.
- ashkankiani 5y ago
  Ya got a source/article on that? I'm curious about it. Overall I agree. I'd take a speed hit for ease of development most of the time, but there are degrees of speed hit that are acceptable depending on the context.
  - akshayshah 5y ago
    Scaling Mercurial at Facebook: https://engineering.fb.com/2014/01/07/core-data/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-data/scaling-merc...
skybrian 5y ago
You could get a 5x slowdown like that for an interpreter written in Go (versus C), but that's a special case and usually it could be replaced by switching from an interpreter to a compiler. (Code generation.)

domano 5y ago

I think finding a good balance between performance and development time & risks is important and C does not seem like a very good choice regarding the second part. At least for web stuff.

studmuffin650 5y ago

Full Disclosure: I do not work at Uber but at another large enterprise that is at Ubers scale and is almost 100% Go across 100s of micro services. Inside an enterprise, there’s more to a language then just the performance (though that is a large factor). You also have to take into account existing tooling (both internal and external), developer experience, whether you can find enough developers to code it and source code maintainability, as well as many other common concerns with a language. Most languages will do well in a few cases but none are best in class in all cases (or everyone would use it). Go does well enough in the performance category while also doing moderately to extremely well in other categories. In CPU bound tasks that don’t rely on CGo, go does extremely well in my experience. I think in general though, for most enterprises, Go strikes a happy medium and makes the right trade offs that most developers are willing to make.

hu3 5y ago

I doubt most of Uber microservices are CPU bound. Yes it's natural for a fleet of this size to also be scrutinized for CPU savings but that's not the only cost involved when choosing a language. Good tooling, dev experience and availability of competent engineers comes to mind.

akshayshah 5y ago
At least when I was at Uber (left in mid-2019), this was absolutely the case: most services weren't CPU-bound, but a few were. Rewriting the CPU-bound services in C++, Rust, or some other language might have helped, but would have required a _ton_ of well-understood, reliable libraries for networking, observability, authn/authz, database access, etc. In nearly all cases, there was plenty of room to make the Go service faster. A more careful choice of data structure and algorithm, finer-grained locking, fan-out across a goroutine pool, or just avoiding a zillion heap allocations solved most problems. I don't recall any cases where Go was simply incapable of meeting our performance needs. As a side benefit, services with more stringent performance requirements often exposed inefficiencies in widely-used libraries. Optimizing those libraries made every Go service faster and cut our overall compute spend. Avoiding rewrites in C++ or Rust let those wins add up over time.

heleninboodler 5y ago

As both a C and Go developer, in my experience, 2-5x execution time increase going from C to Go does not sound reasonable at all.

arp242 5y ago

Facebook managed to optimize mercurial (Python) to be faster than git (C) for their huge monorepo use case simply because Python made it so much easier to implement more efficient algorithms. If you have some fairly simple function/task, then yeah, a C version will probably blow the Go version away almost all of the time. But that's not necessarily indicative of real-world performance of a full application. And of course, there are other interests than "maximize performance" too, such as developer efficiency.

ashkankiani 5y ago
Ya got a source/article on that? I'm curious about it. Overall I agree. I'd take a speed hit for ease of development most of the time, but there are degrees of speed hit that are acceptable depending on the context.
- akshayshah 5y ago
  Scaling Mercurial at Facebook: https://engineering.fb.com/2014/01/07/core-data/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

skybrian 5y ago

You could get a 5x slowdown like that for an interpreter written in Go (versus C), but that's a special case and usually it could be replaced by switching from an interpreter to a compiler. (Code generation.)