5 ms·

Solving Concurrency Bugs Using Schedules and Imagination

58 points by signa11 2y ago

gamegoblin 2y ago
If you happen to be coding in Rust, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle https://github.com/awslabs/shuttle) which can find insanely complicated race conditions. What the Shuttle library is doing is basically automatically going through all the permutations of the schedule diagrams described int his blog post. We used it at AWS to verify the custom filesystem we wrote to power AWS S3. If you're curious, I wrote a little tutorial on it here: https://grantslatton.com/shuttle https://grantslatton.com/shuttle
ibash 2y ago
As someone who’s spent a lot of time with javascript, debugging concurrency issues is second nature. No better training than a great spaghetti ball of promise chains.
SillyUsername 2y ago
Looks a lot like a derivate of a truth table, which is often used to debug multiple input combinations and expected output.
rand03853 2y ago
I insert debug macros with random usleep intervals in critical multithreaded code to expose race conditions. In production they ifdef to nothing.
Groxx 2y ago
It's worth keeping in mind that updates that are not guarded by some kind of barrier are generally not guaranteed to be visible cross-thread / process / server / etc in the order you wrote them. And reorders are rather common in many languages / hardware types / storage systems / log collectors (including stdout because "do thing" and "log that you did it" are evidently not guarded together if there's a race happening), it's not just a theoretical concern. Generally speaking though: yes, writing it down can help A LOT, and starting with what you can see is one of those obvious-in-retrospect things that are easily forgotten when under pressure. There are often a LOT of possibilities, and getting it out of your head so you can enumerate them more precisely can super duper important. Intuition for problematic sequences to check first will come with time.

58 points by signa11 2y ago

If you happen to be coding in Rust, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle https://github.com/awslabs/shuttle) which can find insanely complicated race conditions. What the Shuttle library is doing is basically automatically going through all the permutations of the schedule diagrams described int his blog post. We used it at AWS to verify the custom filesystem we wrote to power AWS S3. If you're curious, I wrote a little tutorial on it here: https://grantslatton.com/shuttle https://grantslatton.com/shuttle

ibash 2y ago

As someone who’s spent a lot of time with javascript, debugging concurrency issues is second nature. No better training than a great spaghetti ball of promise chains.

SillyUsername 2y ago

Looks a lot like a derivate of a truth table, which is often used to debug multiple input combinations and expected output.

rand03853 2y ago

I insert debug macros with random usleep intervals in critical multithreaded code to expose race conditions. In production they ifdef to nothing.

Groxx 2y ago

It's worth keeping in mind that updates that are not guarded by some kind of barrier are generally not guaranteed to be visible cross-thread / process / server / etc in the order you wrote them. And reorders are rather common in many languages / hardware types / storage systems / log collectors (including stdout because "do thing" and "log that you did it" are evidently not guarded together if there's a race happening), it's not just a theoretical concern. Generally speaking though: yes, writing it down can help A LOT, and starting with what you can see is one of those obvious-in-retrospect things that are easily forgotten when under pressure. There are often a LOT of possibilities, and getting it out of your head so you can enumerate them more precisely can super duper important. Intuition for problematic sequences to check first will come with time.