8 ms·

It took months to finally solve a data race in Go. No race detector would see anything. Nobody understood what was happening. It ultimately resulted in a loop

It took months to finally solve a data race in Go. No race detector would see anything. Nobody understood what was happening. It ultimately resulted in a loop counter overflowing, which recomputed the same thing a billion of time (but always the same!). So the visible effect was a request would randomly take 3 min instead of 100ms. I ended up using perf in production, which indirectly lead me to understand the data race. I was called in to help the team because of my experience debugging the weirdest things as a platform dev. Because of this I was exposed to so many races in Go, from my biased point of view, I want Rust everywhere instead. But I guess I am putting myself out of a job? ;)

jchw 1y ago
I think the true answer is that the moment you have to do tricky concurrency in Go, it becomes less desirable. I think that Go is still better at tricky concurrency than C, though there are some downsides too (I think it's a bit easier to sneak in a torn read issue in Go due to the presence of fat pointers and slice headers everywhere.) Go is really good at easy concurrency tasks, like things that have almost no shared memory at all, "shared-nothing" architectures, like a typical web server. Share some resources like database handles with a sync.Pool and call it a day. Go lets you write "async" code as if it were sync with no function coloring, making it decidedly nicer than basically anything in its performance class for this use case. Rust, on the other hand, has to contend with function coloring and a myriad of seriously hard engineering tasks to deal with async issues. Async Rust gets better every year, but personally I still (as of last month at least) think it's quite a mess. Rust is absolutely excellent for traditional concurrency, though. Anything where you would've used a mutex lock, Rust is just way better than everything else. It's beautiful. But I struggle to be as productive in Rust as I am in Go, because Rust, the standard library, and its ecosystem gives the programmer so much to worry about. It sometimes reminds me of C++ in that regard, though it's nowhere near as extremely bad (because at least there's a coherent build system and package manager.) And frankly, a lot of software I write is just boring, and Go does fine for a lot of that. I try Rust periodically for things, and romantically it feels like it's the closest language to "the future", but I think the future might still have a place for languages like Go.
- zozbot234 1y ago
  > And frankly, a lot of software I write is just boring, and Go does fine for a lot of that. I try Rust periodically for things, and romantically it feels like it's the closest language to "the future", but I think the future might still have a place for languages like Go. It's not so much about being "boring" or not; Rust does just fine at writing boring code once you get familiar with the boilerplate patterns (Real-world experience has shown that Rust is not really at a disadvantage wrt. productivity or iteration speed). There is a case for Golang and similar languages, but it has to do with software domains where there literally is no viable alternative to GC, such as when dealing with arbitrary, "spaghetti" reference graphs. Most programs aren't going to look like that though, and starting with Rust will yield a higher quality solution overall.
  - jchw 1y ago
    Rust can yield a higher quality solution, but we can't make a perfect solution, we can only approach perfection. If we want to go further, we could introduce formally-proven code, too. Personally I'm interested in the intersection of proof assistants and Rust, like creusot-rs, and have been investigating it. But as much as I love LARPing about correctness (believe me I do,) it's just simply the case that we won't right perfect software and it's totally OK. It's totally OK that our software will have artificial limitations, like with Go, only accepting filenames that are valid UTF-8, or taking some unnecessary performance/latency hits, or perhaps even crashing in some weird ass edge case. There are very few domains in which correctness issues can't be tolerated. I don't deal with domains that are truly mission critical, where people could die if the code is incorrect. At worst, people could lose some money if my code is incorrect. I still would prefer not to cause that to happen, but those people are generally OK with taking that risk if it means getting features faster. That's why Go has a future really. It's because for most software, some correctness issues are not the end of the world, and so you can rely on not fully sound approaches to finding bugs, like automated testing, race detection, and so on. Rust can also make some types of software more productive to write, but it is unlikely to beat Go in terms of productivity when it comes to a lot of the stuff SaaS shops deal with. And boy, the software industry sure is swamped in fucking SaaS.
norir 1y ago
It is very unfortunate that we use fixed width numbers by default in most programming languages and that common ops will silently overflow. Smarter compilers can work with richer numeric primitives and either automatically promote machine words to big numbers or throw an error on overflow. People talk a lot about the productivity gains of ai, but fixing problems like this at the language level could have an even bigger impact on productivity, but are far less sensational. Think about how much productivity is lost due to obscure but detectable bugs like this one. I don't think rust is a good answer (it doesn't check overflow by default), but at least it points a little bit in the vaguely correct direction.
- recursivecaveat 1y ago
  The situation with numbers in basically every widely used programming language is kind of an indictment of our industry. Silent overflow for incorrect results, no convenient facilities for units, lossy casts everywhere. It's one of those things where standing in 1975 you'd think surely we'll spend some of the next 40 years of performance gains to give ourselves nice, correct numbers to work with, but we never did.
  - tomp 1y ago
    "nice, correct numbers" end somewhere between 1/3 and sqrt(2) so in reality, it's just "pick your own poison" to various degrees...
    - tialaramex 1y ago
      The square root of two is still a computable Real. We choose not to cope with that, but it's not actually impossible it was merely inconvenient. I've mentioned elsewhere that my Rust care realistic is quite happy to work with these numbers e.g. take the square root of ten, and the square root of forty, multiply them together and get the quite ordinary integer twenty. The non computable reals are a huge problem because, as their name suggests, we can't compute them - and in the strict sense that's Almost All reals, but none of the ones you're thinking of are non-computable so you'll likely be fine. For the merely rational numbers like a third, or sixteen hundred and five sevenths, it's even more so a matter of choosing not to address it rather than it being out of reach.
Thaxll 1y ago
Rust does have loop counter overflow.
- swiftcoder 1y ago
  Theoretically you can construct a loop counter that overflows, but I don't that there is any reasonable way to do it accidentally? Within safe rust you would likely need to be using an explicit .wrapping_add() on your counter, and explicitly constructing a for loop that wasn't range-based...
  - tekne 1y ago
    Well, in debug mode yes, but in release mode overflow is wrapping by default unless you explicitly set a flag to make it panic.
- jason_oster 1y ago
  This is irrelevant on 64-bit platforms [^1] [^2]. For platforms with smaller `usize`, enable overflow-checks in your release builds. [^1]: https://www.reddit.com/r/ProgrammerTIL/comments/4tspsn/c_it_will_take_approximately_97_years_to_overflow/ https://www.reddit.com/r/ProgrammerTIL/comments/4tspsn/c_it_... [^2]: https://stackoverflow.com/questions/69375375/is-it-safe-to-assume-64-bit-counter-doesnt-suffer-from-overflow https://stackoverflow.com/questions/69375375/is-it-safe-to-a...
wavemode 1y ago
> It ultimately resulted in a loop counter overflowing, which recomputed the same thing a billion of time (but always the same!). So the visible effect was a request would randomly take 3 min instead of 100ms. This means that multiple goroutines were writing to the same local variable. I've never worked on a Go team where code that is structured in such a way would be considered normal or pass code review without good justification.
- bombela 1y ago
  It happens all the time sadly. It's not because people intentionally write this way. A function takes a parameter (a Go slice for example) and calls another function and so one. Deep down a function copies the pointer to the slice (via closure for example). And then a goroutine is spawned with this closure. The most obvious mistakes are caught quickly. Buu sharing a memory address between two threads can happen very indirectly. And somehow in Go, everybody feels incredibly comfortable spawning millions of coroutines/threads.

jchw 1y ago

I think the true answer is that the moment you have to do tricky concurrency in Go, it becomes less desirable. I think that Go is still better at tricky concurrency than C, though there are some downsides too (I think it's a bit easier to sneak in a torn read issue in Go due to the presence of fat pointers and slice headers everywhere.) Go is really good at easy concurrency tasks, like things that have almost no shared memory at all, "shared-nothing" architectures, like a typical web server. Share some resources like database handles with a sync.Pool and call it a day. Go lets you write "async" code as if it were sync with no function coloring, making it decidedly nicer than basically anything in its performance class for this use case. Rust, on the other hand, has to contend with function coloring and a myriad of seriously hard engineering tasks to deal with async issues. Async Rust gets better every year, but personally I still (as of last month at least) think it's quite a mess. Rust is absolutely excellent for traditional concurrency, though. Anything where you would've used a mutex lock, Rust is just way better than everything else. It's beautiful. But I struggle to be as productive in Rust as I am in Go, because Rust, the standard library, and its ecosystem gives the programmer so much to worry about. It sometimes reminds me of C++ in that regard, though it's nowhere near as extremely bad (because at least there's a coherent build system and package manager.) And frankly, a lot of software I write is just boring, and Go does fine for a lot of that. I try Rust periodically for things, and romantically it feels like it's the closest language to "the future", but I think the future might still have a place for languages like Go.

zozbot234 1y ago
> And frankly, a lot of software I write is just boring, and Go does fine for a lot of that. I try Rust periodically for things, and romantically it feels like it's the closest language to "the future", but I think the future might still have a place for languages like Go. It's not so much about being "boring" or not; Rust does just fine at writing boring code once you get familiar with the boilerplate patterns (Real-world experience has shown that Rust is not really at a disadvantage wrt. productivity or iteration speed). There is a case for Golang and similar languages, but it has to do with software domains where there literally is no viable alternative to GC, such as when dealing with arbitrary, "spaghetti" reference graphs. Most programs aren't going to look like that though, and starting with Rust will yield a higher quality solution overall.
- jchw 1y ago
  Rust can yield a higher quality solution, but we can't make a perfect solution, we can only approach perfection. If we want to go further, we could introduce formally-proven code, too. Personally I'm interested in the intersection of proof assistants and Rust, like creusot-rs, and have been investigating it. But as much as I love LARPing about correctness (believe me I do,) it's just simply the case that we won't right perfect software and it's totally OK. It's totally OK that our software will have artificial limitations, like with Go, only accepting filenames that are valid UTF-8, or taking some unnecessary performance/latency hits, or perhaps even crashing in some weird ass edge case. There are very few domains in which correctness issues can't be tolerated. I don't deal with domains that are truly mission critical, where people could die if the code is incorrect. At worst, people could lose some money if my code is incorrect. I still would prefer not to cause that to happen, but those people are generally OK with taking that risk if it means getting features faster. That's why Go has a future really. It's because for most software, some correctness issues are not the end of the world, and so you can rely on not fully sound approaches to finding bugs, like automated testing, race detection, and so on. Rust can also make some types of software more productive to write, but it is unlikely to beat Go in terms of productivity when it comes to a lot of the stuff SaaS shops deal with. And boy, the software industry sure is swamped in fucking SaaS.

norir 1y ago

It is very unfortunate that we use fixed width numbers by default in most programming languages and that common ops will silently overflow. Smarter compilers can work with richer numeric primitives and either automatically promote machine words to big numbers or throw an error on overflow. People talk a lot about the productivity gains of ai, but fixing problems like this at the language level could have an even bigger impact on productivity, but are far less sensational. Think about how much productivity is lost due to obscure but detectable bugs like this one. I don't think rust is a good answer (it doesn't check overflow by default), but at least it points a little bit in the vaguely correct direction.

recursivecaveat 1y ago
The situation with numbers in basically every widely used programming language is kind of an indictment of our industry. Silent overflow for incorrect results, no convenient facilities for units, lossy casts everywhere. It's one of those things where standing in 1975 you'd think surely we'll spend some of the next 40 years of performance gains to give ourselves nice, correct numbers to work with, but we never did.
- tomp 1y ago
  "nice, correct numbers" end somewhere between 1/3 and sqrt(2) so in reality, it's just "pick your own poison" to various degrees...
  - tialaramex 1y ago
    The square root of two is still a computable Real. We choose not to cope with that, but it's not actually impossible it was merely inconvenient. I've mentioned elsewhere that my Rust care realistic is quite happy to work with these numbers e.g. take the square root of ten, and the square root of forty, multiply them together and get the quite ordinary integer twenty. The non computable reals are a huge problem because, as their name suggests, we can't compute them - and in the strict sense that's Almost All reals, but none of the ones you're thinking of are non-computable so you'll likely be fine. For the merely rational numbers like a third, or sixteen hundred and five sevenths, it's even more so a matter of choosing not to address it rather than it being out of reach.

Thaxll 1y ago

Rust does have loop counter overflow.

swiftcoder 1y ago
Theoretically you can construct a loop counter that overflows, but I don't that there is any reasonable way to do it accidentally? Within safe rust you would likely need to be using an explicit .wrapping_add() on your counter, and explicitly constructing a for loop that wasn't range-based...
- tekne 1y ago
  Well, in debug mode yes, but in release mode overflow is wrapping by default unless you explicitly set a flag to make it panic.
jason_oster 1y ago
This is irrelevant on 64-bit platforms [^1] [^2]. For platforms with smaller `usize`, enable overflow-checks in your release builds. [^1]: https://www.reddit.com/r/ProgrammerTIL/comments/4tspsn/c_it_will_take_approximately_97_years_to_overflow/ https://www.reddit.com/r/ProgrammerTIL/comments/4tspsn/c_it_... [^2]: https://stackoverflow.com/questions/69375375/is-it-safe-to-assume-64-bit-counter-doesnt-suffer-from-overflow https://stackoverflow.com/questions/69375375/is-it-safe-to-a...

wavemode 1y ago

> It ultimately resulted in a loop counter overflowing, which recomputed the same thing a billion of time (but always the same!). So the visible effect was a request would randomly take 3 min instead of 100ms. This means that multiple goroutines were writing to the same local variable. I've never worked on a Go team where code that is structured in such a way would be considered normal or pass code review without good justification.

bombela 1y ago
It happens all the time sadly. It's not because people intentionally write this way. A function takes a parameter (a Go slice for example) and calls another function and so one. Deep down a function copies the pointer to the slice (via closure for example). And then a goroutine is spawned with this closure. The most obvious mistakes are caught quickly. Buu sharing a memory address between two threads can happen very indirectly. And somehow in Go, everybody feels incredibly comfortable spawning millions of coroutines/threads.