6 ms·

Forgive my ignorance, but how does this approach differ from traditional unit testing or end to end testing? My point being that unit testing essentially gives

Forgive my ignorance, but how does this approach differ from traditional unit testing or end to end testing? My point being that unit testing essentially gives you the same confidence in your refactoring efforts that Scientist proposes to offer. Tests already demand that your interface remains the same, and that old code does no harm when replacng legacy.

endergen 11y ago
Agreed that writing test coverage and running twin systems against would get you this. But the benefit is that when you have no or low test coverage, if you have a drop in way of just testing behavior matching then it's probably a more economical approach in many projects. Also, I doubt any test suite exists with the variance of inputs that live user testing generates, including error cases.
anentropic 11y ago
I was wondering the same, but I guess the advantage is being able to safely-ish run tests in production against production data and other live services the sort of stuff you usually mock out in unit tests. and may not be feasible to have a whole copy of your prod 'big data' and services just for testing on with Scientist you get a report not just of mismatched results and exceptions (stuff good tests should usually be able to catch) but also if the new code is slower
rhinoceraptor 11y ago
This [1] is a good writeup on how they used it to replace the merge that github uses. You can't really capture the totality of corner cases in unit tests, especially for complicated things. [1]: http://githubengineering.com/move-fast/ http://githubengineering.com/move-fast/
jowiar 11y ago
"100% test coverage" doesn't imply "won't cause real-world things to break". That your tests run all the code doesn't mean that your tests cover all the actual cases. This identifies bugs and performance regressions on a wide assortment of real-world cases.

endergen 11y ago

Agreed that writing test coverage and running twin systems against would get you this. But the benefit is that when you have no or low test coverage, if you have a drop in way of just testing behavior matching then it's probably a more economical approach in many projects. Also, I doubt any test suite exists with the variance of inputs that live user testing generates, including error cases.

anentropic 11y ago

I was wondering the same, but I guess the advantage is being able to safely-ish run tests in production against production data and other live services the sort of stuff you usually mock out in unit tests. and may not be feasible to have a whole copy of your prod 'big data' and services just for testing on with Scientist you get a report not just of mismatched results and exceptions (stuff good tests should usually be able to catch) but also if the new code is slower

rhinoceraptor 11y ago

This [1] is a good writeup on how they used it to replace the merge that github uses. You can't really capture the totality of corner cases in unit tests, especially for complicated things. [1]: http://githubengineering.com/move-fast/ http://githubengineering.com/move-fast/

jowiar 11y ago

"100% test coverage" doesn't imply "won't cause real-world things to break". That your tests run all the code doesn't mean that your tests cover all the actual cases. This identifies bugs and performance regressions on a wide assortment of real-world cases.