4 ms·
Troubleshooting an intermittent failure in CI tests on ARM64
- steveBK123 3y ago[flagged]
- deleted 3y ago[deleted]
- vardump 3y agoAnother nice demonstration of power of rr, reversible debugging. https://rr-project.org/ https://rr-project.org/
- nicklecompte 3y agoThis is a very well-written blog post: concise, lucid, and it does a good job speaking to programmer audiences with varying backgrounds.
- tmiku 3y agoI never get tired of funny misreads of product names. I thought "our own busted framework" was self-deprecation until it showed up in slab serif on the next line.
- candiddevmike 3y agoThis is a good cautionary tale for folks jumping on cheap ARM cloud instances. Different architectures mean different bugs, and depending on how your infra is provisioned, could make reproducing bugs even harder.
- mark_undoio 3y agoThis is cool - time travel debugging is potentially really helpful in these flaky situations. Having tests fail unpredictably basically means you're not controlling the thing they test, which can be scary. But I find the most annoying failures in CI are the ones I can't reproduce any other way and somehow always happen when I'm not trying. Run locally? Fine. Run on a cloud machine that's identical to the CI system. Fine. Run multiple instances of the test on the cloud machine to generate more load. Fine. Run in the overnight tests - blam. This doesn't always make sense, even once I've found the bug - sometimes the timing just shakes out that way. Sometimes we record stubborn tests that are acting weird, so we're ready when they next fail.
- byyll 3y agoIs that the company that fucked up insomnia.rest?