5 ms·

> Testing concurrency is extremely hard Writing a non-trivial concurrent system based on your understanding of the 'algorithm' , without relying on testing is

by wubrr 1y ago

> Testing concurrency is extremely hard Writing a non-trivial concurrent system based on your understanding of the 'algorithm' , without relying on testing is much harder. > I truly wonder if there is real world tests around this Of course there are. There are many tools, methods, and test suites out there for concurrency testing, for almost any major language out there. Of course, understanding your algorithm, and the systems involved is required to write a proper test suite. > For instance, get sql queries; You ran them, and you have no issue; Is your code sane ? Take those queries and run them 1000x+ times concurrently in a loop. That will catch most common issues. If you want to go a step further you can build a setup to execute your queries in any desired order.

sarchertech 1y ago
I’ve never worked somewhere (in 20 years from big tech companies to small startups) that was generally and reliably testing for concurrency bugs. And I’ve seen dozens of bugs caused by people assuming that transactions (with the default isolation level) protect against race conditions.
- wubrr 1y ago
  Every place I worked at, that had any kind of reliable, high-throughput concurrent system had an extensive suite of concurrent tests. https://github.com/postgres/postgres/tree/master/src/test/isolation https://github.com/postgres/postgres/tree/master/src/test/is... https://muratbuffalo.blogspot.com/2023/08/distributed-transactions-at-scale-in.html https://muratbuffalo.blogspot.com/2023/08/distributed-transa... https://learn.microsoft.com/en-us/archive/msdn-magazine/2008/june/tools-and-techniques-to-identify-concurrency-issues https://learn.microsoft.com/en-us/archive/msdn-magazine/2008... https://go.dev/blog/synctest https://go.dev/blog/synctest https://learntla.com/core/concurrency.html https://learntla.com/core/concurrency.html
  - sarchertech 1y ago
    > Every place I worked at, that had any kind of reliable, high-throughput concurrent system Pretty much anyone with high throughput is running a high throughput concurrent system, and very few companies have an extensive suite of concurrency tests unless you just mean load tests (that aren’t setup to catch race conditions). The “reliable” part of that statement might be doing a lot of heavy lifting depending on what exactly you mean by that.
    - wubrr 1y ago
      I gave you several concrete examples. Your claims of 'very few companies have...' aren't very convincing, and the apparent popularity of concurrency testing isn't really a strong argument for or against it's effectiveness or do-ability.
      - sarchertech 1y ago
        Did you Google “concurrency testing” and send me the top 5 results?
        fn-mote 1y ago
        Kind of looks like it … the supporting evidence includes work from Microsoft: learning how to write concurrent programs. Surely not evidence that Microsoft is testing for concurrency bugs (of course they are).
        ongy 1y ago
        > In Go 1.24, we are introducing a new, experimental testing/synctest package Clearly a mature mechanism we'd see in large companies...
        wubrr 1y ago
        Maybe you should have googled 'concurrency testing' before telling me a story about how you worked at every tech company for 76000 years and never saw any concurrency testing lmao.
- nijave 1y ago
  Not sure about other languages but I believe the stock test tooling for `go` generates a random sorting seed and has a flag to run tests concurrently. You can also manually pass the seed to simulate a certain ordering. While not perfect, our e2e tests caught a couple bugs running them with concurrency.
JackSlateur 1y ago
Running 1000x queries in a loop is called luck.
- wubrr 1y ago
  No, it's called testing many concurrent operations. Implementing a complex concurrent algorithm based on your understanding of it, without proper testing is called luck, and often called delusion.
  - JackSlateur 1y ago
    What algorithm ? The whole idea is that algorithms are useless, and you should just write a bunch of tests and go with it Yes, if I write stuff with locks, I shall ensure that my code acquires and releases locks correctly This is completely off-topic with the original post; Also, you cannot prove something by tests; Just because you found 100000 cases where your code works does not mean there is not a case where is does not (just as you cannot prove that unicorn does not exist) :)
    - sarchertech 1y ago
      > Also, you cannot prove something by tests; Just because you found 100000 cases where your code works does not mean there is not a case where is does not (just as you cannot prove that unicorn does not exist) :) That’s exactly it. For any non trivial program, there exists an infinite number of ways your program can be wrong and still pass all your tests. Unless you can literally test every possible input and every bit of state this holds true.
      - wubrr 1y ago
        For any 'non trivial' program there exists an infinite number of ways your program can be wrong but you still believe it's right. Testing is not a perfect solution to catch all bugs. It's a relatively easy, efficient and reliable way to catch many common bugs though.
pixl97 1y ago
I mean if you're talking SQL on a large real database vs a small test db you can get some pretty big differences in performance and behavior. Of course query planning is something that should be monitored as an app is deployed and used, but testing never does seem to catch the edge cases.

by wubrr 1y ago

sarchertech 1y ago

I’ve never worked somewhere (in 20 years from big tech companies to small startups) that was generally and reliably testing for concurrency bugs. And I’ve seen dozens of bugs caused by people assuming that transactions (with the default isolation level) protect against race conditions.

wubrr 1y ago
Every place I worked at, that had any kind of reliable, high-throughput concurrent system had an extensive suite of concurrent tests. https://github.com/postgres/postgres/tree/master/src/test/isolation https://github.com/postgres/postgres/tree/master/src/test/is... https://muratbuffalo.blogspot.com/2023/08/distributed-transactions-at-scale-in.html https://muratbuffalo.blogspot.com/2023/08/distributed-transa... https://learn.microsoft.com/en-us/archive/msdn-magazine/2008/june/tools-and-techniques-to-identify-concurrency-issues https://learn.microsoft.com/en-us/archive/msdn-magazine/2008... https://go.dev/blog/synctest https://go.dev/blog/synctest https://learntla.com/core/concurrency.html https://learntla.com/core/concurrency.html
- sarchertech 1y ago
  > Every place I worked at, that had any kind of reliable, high-throughput concurrent system Pretty much anyone with high throughput is running a high throughput concurrent system, and very few companies have an extensive suite of concurrency tests unless you just mean load tests (that aren’t setup to catch race conditions). The “reliable” part of that statement might be doing a lot of heavy lifting depending on what exactly you mean by that.
  - wubrr 1y ago
    I gave you several concrete examples. Your claims of 'very few companies have...' aren't very convincing, and the apparent popularity of concurrency testing isn't really a strong argument for or against it's effectiveness or do-ability.
    - sarchertech 1y ago
      Did you Google “concurrency testing” and send me the top 5 results?
      - fn-mote 1y ago
        Kind of looks like it … the supporting evidence includes work from Microsoft: learning how to write concurrent programs. Surely not evidence that Microsoft is testing for concurrency bugs (of course they are).
        ongy 1y ago
        > In Go 1.24, we are introducing a new, experimental testing/synctest package Clearly a mature mechanism we'd see in large companies...
      - wubrr 1y ago
        Maybe you should have googled 'concurrency testing' before telling me a story about how you worked at every tech company for 76000 years and never saw any concurrency testing lmao.
nijave 1y ago
Not sure about other languages but I believe the stock test tooling for `go` generates a random sorting seed and has a flag to run tests concurrently. You can also manually pass the seed to simulate a certain ordering. While not perfect, our e2e tests caught a couple bugs running them with concurrency.

JackSlateur 1y ago

Running 1000x queries in a loop is called luck.

wubrr 1y ago
No, it's called testing many concurrent operations. Implementing a complex concurrent algorithm based on your understanding of it, without proper testing is called luck, and often called delusion.
- JackSlateur 1y ago
  What algorithm ? The whole idea is that algorithms are useless, and you should just write a bunch of tests and go with it Yes, if I write stuff with locks, I shall ensure that my code acquires and releases locks correctly This is completely off-topic with the original post; Also, you cannot prove something by tests; Just because you found 100000 cases where your code works does not mean there is not a case where is does not (just as you cannot prove that unicorn does not exist) :)
  - sarchertech 1y ago
    > Also, you cannot prove something by tests; Just because you found 100000 cases where your code works does not mean there is not a case where is does not (just as you cannot prove that unicorn does not exist) :) That’s exactly it. For any non trivial program, there exists an infinite number of ways your program can be wrong and still pass all your tests. Unless you can literally test every possible input and every bit of state this holds true.
    - wubrr 1y ago
      For any 'non trivial' program there exists an infinite number of ways your program can be wrong but you still believe it's right. Testing is not a perfect solution to catch all bugs. It's a relatively easy, efficient and reliable way to catch many common bugs though.

pixl97 1y ago

I mean if you're talking SQL on a large real database vs a small test db you can get some pretty big differences in performance and behavior. Of course query planning is something that should be monitored as an app is deployed and used, but testing never does seem to catch the edge cases.