4 ms·
I mean, in your example you just have an incomplete test suite. (Though writing a complete one is often unrealistic) While understanding algorithms and data st
by wubrr 1y ago
I mean, in your example you just have an incomplete test suite. (Though writing a complete one is often unrealistic)
While understanding algorithms and data structures is important, the only way you really know how well it works, and how well it's implemented is by thoroughly testing it. There are an infinite amount of clever algorithms out there with terrible implementations.
You need both.
- JackSlateur 1y agoTesting concurrency is extremely hard For instance, get sql queries; You ran them, and you have no issue; Is your code sane ? Or is it because one query ran 10ms earlier and, thus, you avoided the issue ? I truly wonder if there is real world tests around this; I bet there is only algorithm and fuzzing;
- wubrr 1y ago> Testing concurrency is extremely hard Writing a non-trivial concurrent system based on your understanding of the 'algorithm' , without relying on testing is much harder. > I truly wonder if there is real world tests around this Of course there are. There are many tools, methods, and test suites out there for concurrency testing, for almost any major language out there. Of course, understanding your algorithm, and the systems involved is required to write a proper test suite. > For instance, get sql queries; You ran them, and you have no issue; Is your code sane ? Take those queries and run them 1000x+ times concurrently in a loop. That will catch most common issues. If you want to go a step further you can build a setup to execute your queries in any desired order.
- sarchertech 1y agoI’ve never worked somewhere (in 20 years from big tech companies to small startups) that was generally and reliably testing for concurrency bugs. And I’ve seen dozens of bugs caused by people assuming that transactions (with the default isolation level) protect against race conditions.
- wubrr 1y agoEvery place I worked at, that had any kind of reliable, high-throughput concurrent system had an extensive suite of concurrent tests. https://github.com/postgres/postgres/tree/master/src/test/isolation https://github.com/postgres/postgres/tree/master/src/test/is... https://muratbuffalo.blogspot.com/2023/08/distributed-transactions-at-scale-in.html https://muratbuffalo.blogspot.com/2023/08/distributed-transa... https://learn.microsoft.com/en-us/archive/msdn-magazine/2008/june/tools-and-techniques-to-identify-concurrency-issues https://learn.microsoft.com/en-us/archive/msdn-magazine/2008... https://go.dev/blog/synctest https://go.dev/blog/synctest https://learntla.com/core/concurrency.html https://learntla.com/core/concurrency.html
- sarchertech 1y ago> Every place I worked at, that had any kind of reliable, high-throughput concurrent system Pretty much anyone with high throughput is running a high throughput concurrent system, and very few companies have an extensive suite of concurrency tests unless you just mean load tests (that aren’t setup to catch race conditions). The “reliable” part of that statement might be doing a lot of heavy lifting depending on what exactly you mean by that.
- wubrr 1y agoI gave you several concrete examples. Your claims of 'very few companies have...' aren't very convincing, and the apparent popularity of concurrency testing isn't really a strong argument for or against it's effectiveness or do-ability.
- sarchertech 1y agoDid you Google “concurrency testing” and send me the top 5 results?
- fn-mote 1y agoKind of looks like it … the supporting evidence includes work from Microsoft: learning how to write concurrent programs. Surely not evidence that Microsoft is testing for concurrency bugs (of course they are).
- JackSlateur 1y agoRunning 1000x queries in a loop is called luck.
- wubrr 1y agoNo, it's called testing many concurrent operations. Implementing a complex concurrent algorithm based on your understanding of it, without proper testing is called luck, and often called delusion.
- JackSlateur 1y agoWhat algorithm ? The whole idea is that algorithms are useless, and you should just write a bunch of tests and go with it Yes, if I write stuff with locks, I shall ensure that my code acquires and releases locks correctly This is completely off-topic with the original post; Also, you cannot prove something by tests; Just because you found 100000 cases where your code works does not mean there is not a case where is does not (just as you cannot prove that unicorn does not exist) :)
- sarchertech 1y ago> Also, you cannot prove something by tests; Just because you found 100000 cases where your code works does not mean there is not a case where is does not (just as you cannot prove that unicorn does not exist) :) That’s exactly it. For any non trivial program, there exists an infinite number of ways your program can be wrong and still pass all your tests. Unless you can literally test every possible input and every bit of state this holds true.
- wubrr 1y agoFor any 'non trivial' program there exists an infinite number of ways your program can be wrong but you still believe it's right. Testing is not a perfect solution to catch all bugs. It's a relatively easy, efficient and reliable way to catch many common bugs though.
- pixl97 1y agoI mean if you're talking SQL on a large real database vs a small test db you can get some pretty big differences in performance and behavior. Of course query planning is something that should be monitored as an app is deployed and used, but testing never does seem to catch the edge cases.
- hxtk 1y agoI’ve long wished for an SQL error model: given a schema, query, and transaction isolation mode, what errors are theoretically possible? I have a hard time answering this for Postgres, which disappoints me because I don’t see any reason it sounds very easy to answer, like there could be an extension to EXPLAIN that would dry run the query and list all the error states reachable.
- jiggawatts 1y agoComputer Science is incredibly immature. Many of its "founding fathers" are still alive! Something akin to this that blew my mind recently was an IDE for a functional language that used typed holes, the programming equivalent of a semiconductor electron "vacancy", a quasi-particle with real properties that is actually just the lack of a particle. The concept was that if you delete (or haven't yet typed) some small segment out of an otherwise well-typed program, the compiler can figure out the type that the missing part must have. This can be used for rapid development because many such "holes" can only have one possible type. This kind of mechanistic development with tool assistance is woefully under-developed.
- lucianbr 1y agohttps://jepsen.io/ https://jepsen.io/
- toolslive 1y agoThere are "lightweight formal methods". Most problems can be produced via small models. Tools like alloy are built around this idea. (IIRC alloy was used to show that a famous DHT had issues with the churn protocol) https://en.wikipedia.org/wiki/Alloy_(specification_language) https://en.wikipedia.org/wiki/Alloy_(specification_language)
- terpimost 1y agohttps://antithesis.com/ https://antithesis.com/ was made to deal with this. You can think of its as a fuzzing but it has overall determinism for the whole system, so there is a time travel and interactive debugging.
- eevmanu 1y agodeterministic simulation testing[1] (DST) [1] https://notes.eatonphil.com/2024-08-20-deterministic-simulation-testing.html https://notes.eatonphil.com/2024-08-20-deterministic-simulat...