12 ms·

How do you measure robustness except with failures / years of service x times deployed ? You cannot make edge cases and strange unique behavior happen in a cont

by 0x008 4y ago

How do you measure robustness except with failures / years of service x times deployed ? You cannot make edge cases and strange unique behavior happen in a controlled environment.

Cthulhu_ 4y ago
One tool that (I think?) wasn't around, wasn't used as much, or couldn't be deployed at scale back then was fuzzing tools; modern-day application of fuzzers on existing libraries and tools has revealed thousands of potential bugs. If fuzzing, unit tests and things like mutation tests are written and automatically applied to these libraries/utils, on top of being written in a proven memory-safe language like Rust, I would assert they're pretty robust. Second, there's more known knowns this time around; they can write test cases for the decades of bugs and rare occurrences found in the tools they replace. That said, I'm sure there's still plenty of unknown unknowns that can only be uncovered with, as you said, years of service and times deployed.
yjftsjthsd-h 4y ago
> You cannot make edge cases and strange unique behavior happen in a controlled environment. Sure you can. Go iterate through the GNU Coreutils bug tracker, find weird bugs, create test cases, feed them to your implementation. Bonus points if you can find an existing test suite (ex. xfstests turns out to be good for testing arbitrary filesystems). Granted, there will always be edge cases that you don't catch until they show up live in prod, but you can hit a chunk of the space without that.
- kaba0 4y ago
  Also, there are plenty of package system that uses coreutils for their build instructions. Just build the whole thing with coreutils aliased to the rust implementation and check for errors.
- retcond 4y ago
  You have just contradicted yourself by saying, there will always be edge cases that you don't catch until they show up live in in prod.." in reply to, "How do you measure robustness except with failures / years of service x times deployed ?" The only viable live testing environment that springs to mind might be running your test code synchronously at the atomic level with production, which I'm convinced only IBM Z/OS on a Parallel Sysplex cluster running CICS can do. Ed.spelling
  - yjftsjthsd-h 4y ago
    I'm not seeing a contradiction. The original claim was that you can't measure robustness without testing live, because you can't reproduce edge cases in a lab. But that's not true; you can reproduce edge cases that are known in a lab. This isn't 100% effective, granted, but it's effective enough that you certainly can test robustness to a reasonable degree. It's like saying that you don't know how safe a car is until you've drive it 100,000 miles on the real roads; Real Life™ will find things you missed in testing, but you can still run enough crashes to get a decent idea of how safe the car is.

by 0x008 4y ago

How do you measure robustness except with failures / years of service x times deployed ? You cannot make edge cases and strange unique behavior happen in a controlled environment.

Cthulhu_ 4y ago

One tool that (I think?) wasn't around, wasn't used as much, or couldn't be deployed at scale back then was fuzzing tools; modern-day application of fuzzers on existing libraries and tools has revealed thousands of potential bugs. If fuzzing, unit tests and things like mutation tests are written and automatically applied to these libraries/utils, on top of being written in a proven memory-safe language like Rust, I would assert they're pretty robust. Second, there's more known knowns this time around; they can write test cases for the decades of bugs and rare occurrences found in the tools they replace. That said, I'm sure there's still plenty of unknown unknowns that can only be uncovered with, as you said, years of service and times deployed.

yjftsjthsd-h 4y ago

> You cannot make edge cases and strange unique behavior happen in a controlled environment. Sure you can. Go iterate through the GNU Coreutils bug tracker, find weird bugs, create test cases, feed them to your implementation. Bonus points if you can find an existing test suite (ex. xfstests turns out to be good for testing arbitrary filesystems). Granted, there will always be edge cases that you don't catch until they show up live in prod, but you can hit a chunk of the space without that.

kaba0 4y ago
Also, there are plenty of package system that uses coreutils for their build instructions. Just build the whole thing with coreutils aliased to the rust implementation and check for errors.
retcond 4y ago
You have just contradicted yourself by saying, there will always be edge cases that you don't catch until they show up live in in prod.." in reply to, "How do you measure robustness except with failures / years of service x times deployed ?" The only viable live testing environment that springs to mind might be running your test code synchronously at the atomic level with production, which I'm convinced only IBM Z/OS on a Parallel Sysplex cluster running CICS can do. Ed.spelling
- yjftsjthsd-h 4y ago
  I'm not seeing a contradiction. The original claim was that you can't measure robustness without testing live, because you can't reproduce edge cases in a lab. But that's not true; you can reproduce edge cases that are known in a lab. This isn't 100% effective, granted, but it's effective enough that you certainly can test robustness to a reasonable degree. It's like saying that you don't know how safe a car is until you've drive it 100,000 miles on the real roads; Real Life™ will find things you missed in testing, but you can still run enough crashes to get a decent idea of how safe the car is.