4 ms·

You're going to be very sad to discover the sentiments around software-intensive systems testing in a lot of places & industries that you'd rightfully assume sh

You're going to be very sad to discover the sentiments around software-intensive systems testing in a lot of places & industries that you'd rightfully assume should know better and do better. I had a safety critical embedded engineer who had worked on everything from weapons systems to automobiles first not have any idea what property-based testing was and then proceed to tell me at length how it was pointless because unit testing is all you need as long as you keep making your units smaller and smaller. At the time I worked in FinTech and was aghast at what I was hearing as they worked on self-driving. If I had to pin down what made me shift my career path into "cyber-physical systems", it was probably that conversation. Since making the transition and later founding a company dedicated to testing in that domain, I've heard dozens and dozens of sentiments even more concerning.

deleted 3y ago
[deleted]
JohnFen 3y ago
> and then proceed to tell me at length how it was pointless because unit testing is all you need as long as you keep making your units smaller and smaller. I've encountered devs that think this nonsense before, but always assumed that this attitude couldn't be held with systems that are actually critical. I'm a bit less naive now.
- shermantanktop 3y ago
  I thought that too, early on. Sort of a perfect-bricks-make-perfect-walls idea. And when problems emerge at the integration stage, it’s often traceable to a gap that a lower-level test could have caught. So the mentality is a bit self-reinforcing.
  - im_down_w_otp 3y ago
    Exactly. All the products we make are focused at the integration and system level because it's crystal clear that's where enormous amounts of issues arise and unbounded numbers of 11th hour fire drills occur. The various industries who've had multiple product cycles with lots of software are starting to figure this out, but it took awhile. Despite how obvious it is with a little outside observer perspective, the "perfect bricks" way of thinking is pervasive because that's how the builders and supply chains are organized.
  - chernevik 3y ago
    Which is silly, because the whole point of integration testing is to show your bricks don't fit together as expected.
    - shermantanktop 3y ago
      Or that your bricks aren't square, or they are actually made of sand, or that they collapse under load. To leave the analogy behind, any non-trivial component has a testable surface area, but typically has additional modes of behavior associated with internal state, environmental conditions, or other areas that well-meaning unit test writers didn't think about. I have often found issues in simple caller/callee pairs of two components, both of which are tested, but the caller contains subtle expectations of the callee that the unit tests don't match up with.
      - bumby 3y ago
        Software testing needs a unique mindset compared to traditional hardware reliability testing. The software doesn't "wear out" like the hardware but tends to fail in coordinating functions. Too often, we rely on simplistic measures like a hard restart to manage coordination failures, but that isn't always an option on safety-critical applications.
  - grumpyprole 3y ago
    Taking that argument to the extreme, why test any new software at all when the underlying instructions have all been thoroughly tested!
    - shermantanktop 3y ago
      Well we do that all day long when we write to a file system and expect it to actually persist data to disk. It's usually only db engines and loggers that attempt to flush, and even then the disk will usually lie in response.
    - deleted 3y ago
      [deleted]
  - bumby 3y ago
    Ariane V is an interesting case study in that regard [1]. It failed because they didn't feel the need to retest the software reused from the Ariane IV design. No need to re-test the bricks that already worked before... [1] https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88

deleted 3y ago

[deleted]

JohnFen 3y ago

> and then proceed to tell me at length how it was pointless because unit testing is all you need as long as you keep making your units smaller and smaller. I've encountered devs that think this nonsense before, but always assumed that this attitude couldn't be held with systems that are actually critical. I'm a bit less naive now.

shermantanktop 3y ago
I thought that too, early on. Sort of a perfect-bricks-make-perfect-walls idea. And when problems emerge at the integration stage, it’s often traceable to a gap that a lower-level test could have caught. So the mentality is a bit self-reinforcing.
- im_down_w_otp 3y ago
  Exactly. All the products we make are focused at the integration and system level because it's crystal clear that's where enormous amounts of issues arise and unbounded numbers of 11th hour fire drills occur. The various industries who've had multiple product cycles with lots of software are starting to figure this out, but it took awhile. Despite how obvious it is with a little outside observer perspective, the "perfect bricks" way of thinking is pervasive because that's how the builders and supply chains are organized.
- chernevik 3y ago
  Which is silly, because the whole point of integration testing is to show your bricks don't fit together as expected.
  - shermantanktop 3y ago
    Or that your bricks aren't square, or they are actually made of sand, or that they collapse under load. To leave the analogy behind, any non-trivial component has a testable surface area, but typically has additional modes of behavior associated with internal state, environmental conditions, or other areas that well-meaning unit test writers didn't think about. I have often found issues in simple caller/callee pairs of two components, both of which are tested, but the caller contains subtle expectations of the callee that the unit tests don't match up with.
    - bumby 3y ago
      Software testing needs a unique mindset compared to traditional hardware reliability testing. The software doesn't "wear out" like the hardware but tends to fail in coordinating functions. Too often, we rely on simplistic measures like a hard restart to manage coordination failures, but that isn't always an option on safety-critical applications.
- grumpyprole 3y ago
  Taking that argument to the extreme, why test any new software at all when the underlying instructions have all been thoroughly tested!
  - shermantanktop 3y ago
    Well we do that all day long when we write to a file system and expect it to actually persist data to disk. It's usually only db engines and loggers that attempt to flush, and even then the disk will usually lie in response.
  - deleted 3y ago
    [deleted]
- bumby 3y ago
  Ariane V is an interesting case study in that regard [1]. It failed because they didn't feel the need to retest the software reused from the Ariane IV design. No need to re-test the bricks that already worked before... [1] https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88