4 ms·

One of the mistake I see a lot around this is when people start to write intelligent mocks. Usually this is where test bugs reside.

by neo2006 8y ago

p1necone 8y ago
The best is when someone writes an entirely new implementation of something just for tests, and only calls that from their tests instead of the actual product code! Then you just get blank stares when you try to explain why what they've just done is completely pointless.
- loup-vaillant 8y ago
  The way they do it may be pointless, but the intent is not. The crux is making sure your two implementation are independent. There are two ways to achieve this. The first is clean room. Just have independent persons or teams implement the same thing. With different tools and programming languages if you can. The second is use an entirely different approach for both. The test code for instance can be constructed around the tests, and for instance rely on the fact that everything happens at the same place at the same time (when you run the test). --- I'm currently writing on a secure handshakes generator. https://github.com/LoupVaillant/Monokex https://github.com/LoupVaillant/Monokex It generates specifications and source code from a Noise pattern. A bit like Noise Explorer, only with less features. Previous versions of the code were hand written, and the generated code is doing its best to look like it was hand written. Here's the result: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/monokex.c https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... Now how do I ensure the code match the specs? (The correctness of the specs itself is currently checked by hand.) I could write another implementation, but I'm only me, and I can't go erase my own memory to make a clean room implementation and compare the two. Instead, I took the specs, and wrote code that takes all the inputs, and spits out all the intermediate buffers and outputs, paying no heed to stuff like order of execution, or who does what. I only concentrated on generating the test vectors: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/vectors.c#L81 https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... The structure of the vector generating code and the actual production code are very different. This is how I decorrelate mistakes, and make sure that if the two "implementations" agree, I'm very likely to have something that works. --- Of course, this is all a lot of effort, so I wouldn't do this for non-critical code.

One of the mistake I see a lot around this is when people start to write intelligent mocks. Usually this is where test bugs reside.

p1necone 8y ago

The best is when someone writes an entirely new implementation of something just for tests, and only calls that from their tests instead of the actual product code! Then you just get blank stares when you try to explain why what they've just done is completely pointless.

loup-vaillant 8y ago
The way they do it may be pointless, but the intent is not. The crux is making sure your two implementation are independent. There are two ways to achieve this. The first is clean room. Just have independent persons or teams implement the same thing. With different tools and programming languages if you can. The second is use an entirely different approach for both. The test code for instance can be constructed around the tests, and for instance rely on the fact that everything happens at the same place at the same time (when you run the test). --- I'm currently writing on a secure handshakes generator. https://github.com/LoupVaillant/Monokex https://github.com/LoupVaillant/Monokex It generates specifications and source code from a Noise pattern. A bit like Noise Explorer, only with less features. Previous versions of the code were hand written, and the generated code is doing its best to look like it was hand written. Here's the result: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/monokex.c https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... Now how do I ensure the code match the specs? (The correctness of the specs itself is currently checked by hand.) I could write another implementation, but I'm only me, and I can't go erase my own memory to make a clean room implementation and compare the two. Instead, I took the specs, and wrote code that takes all the inputs, and spits out all the intermediate buffers and outputs, paying no heed to stuff like order of execution, or who does what. I only concentrated on generating the test vectors: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/vectors.c#L81 https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... The structure of the vector generating code and the actual production code are very different. This is how I decorrelate mistakes, and make sure that if the two "implementations" agree, I'm very likely to have something that works. --- Of course, this is all a lot of effort, so I wouldn't do this for non-critical code.