3 ms·
The systems I work with seem to choose the even worse "PA/EA", to use the article's terminology, i.e., consistency never, even during apparent normal operation.
by deathanatos 3y ago
The systems I work with seem to choose the even worse "PA/EA", to use the article's terminology, i.e., consistency never, even during apparent normal operation.
We had whole slew of bugs that got discovered by this: we use a vendor for storing Docker images, and AFAICT empirically, their system does not obey the property "Read-Your-Writes". We knew this already about it for images, but what tripped the other day seemed to be that layers are also susceptible: so when the manifest of a new image was uploaded, it referenced layers that the client has acknowledged writes for, but the write of the manifest apparently failed because the layer "didn't exist".
(The script that was uploading the images was also written in Typescript, and Deno's API for executing child processes uses C-style "return the error" handling¹. The script forgot to check, so our CI run "passed", all while missing an image. A deployment then bumped the image versions to the new (non-existent) version, and that deploy failed. Thankfully k8s keeps running the old versions, but the number of swiss cheese holes that went through was pretty astounding.)
So many systems fail read-your-writes, and it just makes them all the more annoying to deal with. Inevitably you are writing to a system because you want to do something with that write: I'm not persisting data into a black hole, never to be see again, after all. But sanely serializing the next action in the sequence when you do not know if the write is "good" yet is impossible, and makes things brittle.
(We asked the vendor about this, and the response from support types / vendors is just so utterly disappointing. "Can't you just write a loop to check if the image appears to exist?" This is one of the many spots where CS theory matters in the actual engineering! No, no I cannot "just" write a "simple" loop to check, and the theory can tell you why that doesn't work. Not to mention the layers/manifest part means I need to re-write `docker push`…)
(¹but child process APIs are rife with this. Heck, even Rust's would bite you all the same. AFAIK, I think only Python exposes safe-ish APIs in this area?)