2 ms·
You should definitely also try the exact original conditions. Think about reducing a tricky bug: you start with the exact input and code that first caused the p
by andrewflnr 6y ago
You should definitely also try the exact original conditions. Think about reducing a tricky bug: you start with the exact input and code that first caused the problem, then try removing constraints to see what preserves the bug.
Ed; I should clarify: you start with whatever big nasty test scenario you need to make sure you can reproduce the effect at all, and only then can you start changing the conditions and be confident you're still measuring the same thing.
- mxcrossb 6y agoYes but science and computer programming are fundamentally different activities. The programmer tries to reproduce the bug because his main fear is that he can’t see its effect. But the scientist’s main fear is that the experimental setup does not robustly test the underlying hypothesis. If you reproduce exactly the conditions (and remember you will always be criticized for missing X), the best you can learn is whether the previous result was not completely fabricated (or statistical noise). It’s much more productive to subject a result to a wide battery of tests.
- andrewflnr 6y agoIf you're dealing with an unknown, confusing phenomenon, you don't know what the underlying hypothesis is. You don't know yet what constitutes rigorous testing and what is just wandering off into the weeds. So you need to start with seeing if the thing happens at all. You do in fact need to know whether the previous result was garbage before it can be productive to do a wide battery of tests. And "wide" depends on how sensitive to conditions your effect is, which you also don't know yet. Your closest thing to an anchor is the initial claimed setup, so start there.