5 ms·
In our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, te
by httpsterio 7y ago
In our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, tests were missing or tests didn't catch the issue.
In the first two cases, we revert to the commit where untested or failed code was introduced to the master. This basically never happens. In the third case, you need to do some debugging and try to figure out why it's broken and either fix it or revert it. Basically just look at the git history. If the code has dependencies then you might need to do code triage and produce a hotfix. If it's a relatively isolated piece, then just revert and fix it at a better time.
We use semver and gitlab's tags so we know just by the versioning if the code that is broken is important or not and if we can roll back.
- StreamBright 7y agoWhat about logical errors? math.pow(2, 4) vs math.pow(4, 2)
- ISL 7y agoA test can test for those: "does the code give mathematically correct results?"
- StreamBright 7y agoI meant pow as an example to point out that there are logical errors in code that you cannot catch with unit tests.
- weaksauce 7y agothose kinds of errors should be caught a little higher in the hierarchy... some kind of feature or integration test.
- Scarblac 7y agoEven the best tests only catch like 50% of the bugs though.
- WrtCdEvrydy 7y agoThat's not really correct. Properly input space partitioned tests have something like 90 to 95 accuracy if properly written. The issues always come from people not wanting to add sufficient tests.
- cortesoft 7y agoDoes that mean that 1 in 20 deploys breaks, then?
- WrtCdEvrydy 7y ago1 in 20 deploys has a false negative (IE, something went out without working). Continous Integration is mostly about regression, so the new tests have less value than running old tests. We tend to fix issues in production (bugs) by mandating a unit test with the failing issue to be written as part of the fix.
- aasasd 7y agoBahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’
- not_kurt_godel 7y agoThis is good in theory, and often in practice, but not always. If this practice is followed blindly, unit tests can become too tightly bound to the code to the point where every code change is assumed to require corresponding 1:1 unit test changes and it becomes difficult to distinguish between a broken test that signals broken functionality versus one that simply hasn't been 'synced' yet.
- ouaiueoaieaeoiu 7y agoSemantic errors? I don't think a unit test will be able to distinguish 16 from 16.
- ricardobeat 7y agoImagine this: the spec is Math.pow(x,y). It was implemented as Math.pow(y, x). The test was written with x=2, y=4. It is accidentally correct. This is why any unit tests dealing with absolute or enum values needs to run multiple iterations.