5 ms·
http://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ http://randomascii.wordpress.com/2014/01/27/theres-only-four... tes
by dalke 13y ago
http://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ http://randomascii.wordpress.com/2014/01/27/theres-only-four... tested all 4+ billion inputs to ceil(), floor(), and round(), and pointed out that many libraries actually had high error rates, because of incorrect support for rounding, small values, NaN, and -0.0.
Each test is extremely fast. Testing all 12+ billion cases takes 4.5 minutes. Are these unit tests or integration tests?
If they are integration tests, roughly where is the boundary between the two?
Is it meaningful to distinguish between "unit" and "integration" testing based only on the amount of time they takes? That is, if the unit tests take 0.1 second too long then do they suddenly become integration tests?
- twistedpair 13y ago12B test permutations is not your typical scenario, though 4min is pretty damn quick for all that. I'm asserting that for a given project module, it is beneficial to be able to run the test suite in a short time, say 10-15s. If you've got to wait minutes, then it is more integration. The longer your unit tests take, the less likely people will be to use and run them often, which is the whole point. Let the nightly build on the CI machine exercise the long running tests when everyone is asleep.
- dalke 13y agoTrue. Only a few tests are fast enough to run 12B tests within a few minutes. Really, I think the problem is that unit test frameworks are currently incapable of doing the right testing. Unit tests take quadratic time. That is, each new test requires running all previous tests, to get the green. And at some point, a project will have enough tests that it can't finish in 10-15s. One option is to mark "fast" and "slow" tests. Another is to recategorize them as "unit" vs. "functional" tests. These are poorly-defined labels. In this case, the 4.5 minutes of testing is "slow", yes, but it only needs to be run when a specific, small part of the code changes. The problem is, there's no way to determine that automatically. The test runner can't look at the previous test execution path and see that nothing has changed, and there's no way to mark that a test should only be run if code in functions X, Y, or Z of module ABC has changed. Humans are able to figure this out. Well, sometimes. And with lots of mistakes. Get the unit test framework to talk with a coverage analysis tool, plus some static analysis and perhaps a few annotations, and this discussion of how to distinguish one set of tests from another disappears. Blue-sky dreams. I know. :) In real life we toss those functions into their own library, note that the code is static, and do the full test suite only occasionally; mostly when the compiler changes. In other words, bypass CI the same way one does any other third party library. (How often do you run the gcc test suite?)
- twistedpair 13y agoVarious test runners do just this. Maven on TeamCity ranks the tests by their volatility (recently failed first), then by run duration. The point is to run the most likely to fail and historically most brittle tests first and the slow stuff last so you can fail fast.
- dalke 13y agoThe goal is similar, but it isn't the same thing. That still means to run all the tests each time, with re-prioritization to enrich the likelihood of faster feedback. But if none of the code paths used for a test have changed, and the compiler hasn't changed, and there's nothing which depends on random input or timing effects, then why run those tests at all? The reason is we don't have a good way to do that dependency analysis, which is why we run all of the tests all of the time. Or we manually partition them into "slow" and "fast" tests.
- KMag 13y agoCode instrumented for coverage tells you which tests executed which portions of code. As I remember, Google's C++ build/test system was using this by late 2009 to efficiently run all tests on all checkins to HEAD.
- hermannj314 13y agoI call those unit tests. I think the problem is that definitions that should reflect the granularity of what is being tested have become too interconnected with assumptions about frequency of testing / time to run and it has created completely mangled definitions. The problem is so bad that some developers I've crossed have the firm opinion that mstest should only be used for "unit" tests. It is frustrating. If you have slow unit tests, don't run them on every compile. If you have fast integration tests, run them as much as you'd like. I've personally never defined unit tests as "things that run fast", despite that being a valuable property it does not seem essential to the definition, but perhaps I have the wrong understanding of what a unit test is.