4 ms·

Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is

Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is paramount! Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.

StavrosK 5y ago
The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?
- stonemetal12 5y ago
  > If "check expected outputs" is a subjective step anyway It isn't. It is the evidence that calculations in the code were done correctly. It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.
  - StavrosK 5y ago
    How do you tell that the calculations were done correctly? Presumably you have some way of doing that. Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.
    - pigeonhole123 5y ago
      Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.
    - stonemetal12 5y ago
      >How do you tell that the calculations were done correctly? Presumably you have some way of doing that. Isn't that what we are discussing, how to tell if a piece of code works? You could do a formal proof that the code works (which is long and tedious) or you can test it (less long and tedious but less rigorous). >You don't need tests if you only need to do it once. It doesn't matter how many times you plan on running it. What counts is how much we value the output. I wouldn't bet $20 that untested code works correctly.
woeirua 5y ago
Synthetics only take you so far though. Real data almost always has unusual data (outliers), or the data will violate various assumptions that you've made along the way. In many cases there is no way to know ahead of time how the data will break your model. Which is why monitoring for model drift is so important.

StavrosK 5y ago

The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?

stonemetal12 5y ago
> If "check expected outputs" is a subjective step anyway It isn't. It is the evidence that calculations in the code were done correctly. It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.
- StavrosK 5y ago
  How do you tell that the calculations were done correctly? Presumably you have some way of doing that. Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.
  - pigeonhole123 5y ago
    Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.
  - stonemetal12 5y ago
    >How do you tell that the calculations were done correctly? Presumably you have some way of doing that. Isn't that what we are discussing, how to tell if a piece of code works? You could do a formal proof that the code works (which is long and tedious) or you can test it (less long and tedious but less rigorous). >You don't need tests if you only need to do it once. It doesn't matter how many times you plan on running it. What counts is how much we value the output. I wouldn't bet $20 that untested code works correctly.

woeirua 5y ago

Synthetics only take you so far though. Real data almost always has unusual data (outliers), or the data will violate various assumptions that you've made along the way. In many cases there is no way to know ahead of time how the data will break your model. Which is why monitoring for model drift is so important.