2 ms·

In the vast majority of fields of engineering, it's possible to exhaustively test your product. Your power supply is specified to accept 90v-260v 50-60Hz and d

In the vast majority of fields of engineering, it's possible to exhaustively test your product. Your power supply is specified to accept 90v-260v 50-60Hz and deliver 12v at 1 to 20 watts, in ambient temperatures between -5°C and 55°C? You can test over the entire input range, your expert knowledge assuring you that if it's tested at 20°C and 22°C there's no need to test at 21°C. On the other hand, if your software is specified to correctly validate a SAML assertion? It's simply not possible to enumerate all the states the system might encounter.

rcxdude 3y ago
I wouldn't say this is actually true: while there are generally fewer dimensions in hardware testing, it's not actually a given that if you have tested over the range in each of them that you will certainly be OK across the entire space (in the same way that having unit tested each of your functions at extremes doesn't mean the whole system will work as intended), not to mention it's not very common that you actually have anywhere near proper coverage of each of those dimensions (and don't get me started on lifetime and aging testing). Also, testing is often very expensive so it's also very common that you don't actually have test results for the version you're actually building (one of the reasons safety critical hardware is so expensive is because regulations generally require you do this as opposed to handwaving). (This isn't to say software isn't more complex than hardware: just that I think software engineers have a tendency to romanticize the level of competence and confidence other engineering disciplines have in their designs)
- michaelt 3y ago
  > I wouldn't say this is actually true: while there are generally fewer dimensions in hardware testing, it's not actually a given that if you have tested over the range in each of them that you will certainly be OK across the entire space I agree that the point where the power supply dissipates the most power, or the point where it starts making an audible whine, might be in the middle of its operating range rather than at its limits. You would certainly want an experienced engineer who can say what density of parameter sweep is appropriate. This is what I was trying to say when I said that when testing a system designed to operate between -5°C and 55°C, expert knowledge might assure you there was no need to test at 21°C if it had been tested at 20°C and 22°C successfully.
  - nunuvit 3y ago
    I've designed power supplies and I can tell you that the "density" of the test doesn't matter so much as the validity of the test. One way we assess stability is by doing frequency sweeps. What allows us to infer stability from that test is the assumption that the system is linear time invariant (LTI). We assume it's LTI because we tried to design it to mostly act that way, even though it's really not. For an LTI system, a frequency sweep and a step response are completely equivalent and interchangeable, it doesn't matter which one you use. But what actually happens is the step response test reveals different information. This can only happen if the system is not LTI. Therefore neither test is conclusive about stability, though they're still informative, which is why we do them. 10x density would be no less inclusive. Another problem is that some stability issues have very low observability. The blip or offset they cause in testing is indistinguishable from normal switching noise and measurement imperfections. They only really reveal themselves when they become a problem in the field. 10x density wouldn't find the issue because it's caused by some other combination. Explaining and fixing those issues is tough, and anticipating them requires an arsenal of models developed from someone's hard earned experience. I don't mean to diminish the density issue. That's still a real thing, but it's kind of an orthogonal problem. The formal term is optimal design of experiments, and in simulation we use search methods like Monte Carlo because it's not feasible to enumerate the design space even on a computer, much less in a real test.
naniwaduni 3y ago
It's way more possible to exhaustively enumerate the states a software system might be in than any nontrivial hardware system.
deleted 3y ago
[deleted]
nunuvit 3y ago
It doesn't work like that. Testing doesn't enumerate the physical design space. It only enumerates a model. Only very simple theoretical models can be enumerated in practice. We have to design with much more sophisticated and accurate models that can't be enumerated, and we have to settle for knowing that even those models are inadequate to describe everything we need to account for. We do our best to design in a way that makes the simple, testable models valid and informative, but it's impossible to completely succeed, so we only ever test "enough" (to make money).

rcxdude 3y ago

I wouldn't say this is actually true: while there are generally fewer dimensions in hardware testing, it's not actually a given that if you have tested over the range in each of them that you will certainly be OK across the entire space (in the same way that having unit tested each of your functions at extremes doesn't mean the whole system will work as intended), not to mention it's not very common that you actually have anywhere near proper coverage of each of those dimensions (and don't get me started on lifetime and aging testing). Also, testing is often very expensive so it's also very common that you don't actually have test results for the version you're actually building (one of the reasons safety critical hardware is so expensive is because regulations generally require you do this as opposed to handwaving). (This isn't to say software isn't more complex than hardware: just that I think software engineers have a tendency to romanticize the level of competence and confidence other engineering disciplines have in their designs)

michaelt 3y ago
> I wouldn't say this is actually true: while there are generally fewer dimensions in hardware testing, it's not actually a given that if you have tested over the range in each of them that you will certainly be OK across the entire space I agree that the point where the power supply dissipates the most power, or the point where it starts making an audible whine, might be in the middle of its operating range rather than at its limits. You would certainly want an experienced engineer who can say what density of parameter sweep is appropriate. This is what I was trying to say when I said that when testing a system designed to operate between -5°C and 55°C, expert knowledge might assure you there was no need to test at 21°C if it had been tested at 20°C and 22°C successfully.
- nunuvit 3y ago
  I've designed power supplies and I can tell you that the "density" of the test doesn't matter so much as the validity of the test. One way we assess stability is by doing frequency sweeps. What allows us to infer stability from that test is the assumption that the system is linear time invariant (LTI). We assume it's LTI because we tried to design it to mostly act that way, even though it's really not. For an LTI system, a frequency sweep and a step response are completely equivalent and interchangeable, it doesn't matter which one you use. But what actually happens is the step response test reveals different information. This can only happen if the system is not LTI. Therefore neither test is conclusive about stability, though they're still informative, which is why we do them. 10x density would be no less inclusive. Another problem is that some stability issues have very low observability. The blip or offset they cause in testing is indistinguishable from normal switching noise and measurement imperfections. They only really reveal themselves when they become a problem in the field. 10x density wouldn't find the issue because it's caused by some other combination. Explaining and fixing those issues is tough, and anticipating them requires an arsenal of models developed from someone's hard earned experience. I don't mean to diminish the density issue. That's still a real thing, but it's kind of an orthogonal problem. The formal term is optimal design of experiments, and in simulation we use search methods like Monte Carlo because it's not feasible to enumerate the design space even on a computer, much less in a real test.

naniwaduni 3y ago

It's way more possible to exhaustively enumerate the states a software system might be in than any nontrivial hardware system.

deleted 3y ago

[deleted]

nunuvit 3y ago

It doesn't work like that. Testing doesn't enumerate the physical design space. It only enumerates a model. Only very simple theoretical models can be enumerated in practice. We have to design with much more sophisticated and accurate models that can't be enumerated, and we have to settle for knowing that even those models are inadequate to describe everything we need to account for. We do our best to design in a way that makes the simple, testable models valid and informative, but it's impossible to completely succeed, so we only ever test "enough" (to make money).