3 ms·

My opinion: Linux Kernel coders are real programmers. Do you think Mel wrote unit tests?[1] Real programmers are discrete mathematicians in their head. With p

My opinion: Linux Kernel coders are real programmers. Do you think Mel wrote unit tests?[1] Real programmers are discrete mathematicians in their head. With pointers. Their code isn't perfect. By analogy, when Andrew Wiles published his some 109 page proof of Fermat's Last Theorem it had bugs. [2] The mathematics community tested it and he eventually corrected it. The Linux Kernel is like that. There are no unit tests for a^n + b^n = c^n because no integers above n=2 satisfy them. You can't unit test your way to secure, correct code in the kernel either. Only a community of testers and verifiers can do that. "given enough eyeballs, all bugs are shallow"[3] [1] http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html [2] https://en.wikipedia.org/wiki/Wiles%27s_proof_of_Fermat%27s_Last_Theorem https://en.wikipedia.org/wiki/Wiles%27s_proof_of_Fermat%27s_... [3] https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law

saagarjha 4y ago
When Andrew Wiles writes a flawed proof he gets to think for a while and try to fix it. When someone puts broken code into Linux someone else gets to think for a whole and try to fix it. Except they’re losing thousands per hour or someone is having their phone hacked. Unit tests are just one part of the story, but they can be very important.
- Xelynega 4y ago
  I think their point is that Andrew Wiles' flawed proof was shown to a community and improved without any automated testing because something can be verified before it's integrated and that verification never changes. You don't need testing for verified code interacting with verified code in an environment verified to run verified code, but testing takes half the time for "good enough" reliability so we don't spend the time.
- MaxBarraclough 4y ago
  I'm not sure I see the point of this comparison. Publication of a bad proof with an untrue conclusion may lead to man-years of wasted research effort. The dollar figure of the cost of a bad proof, or of a kernel bug, will always depend on the specifics of the case.
thesuperbigfrog 4y ago
>> You can't unit test your way to secure, correct code in the kernel either. Only a community of testers and verifiers can do that. Unit tests cannot show the presence of bugs that you are not teating for, but they can show the absence of bugs you've fixed and do not want to re-introduce / regressions. Not having unit tests strikes me as either hubris or lack of resources. Are you really going to have people run repeated tests even if some of the tests could be automated? If a test cannot be automated, then that's understandable, but why wouldn't you automate what you can with unit tests and integration tests? I would further argue that even if unit tests, integration tests, and a community of testers and verifiers are working together there will still be bugs, some of which will be critical. Why wouldn't you use every tool at your disposal in addition to community / people to find and fix the bugs?
- Xelynega 4y ago
  If interface A is verified to be correct code(not tested) and it's replaced with interface B that's also verified to be correct, why would there be any regressions? The reason I see not to do it is it takes time away from writing correct code. I could spend half my time writing unit tests or I can spend twice the time writing the code in a verifiable language.
  - thesuperbigfrog 4y ago
    >> If interface A is verified to be correct code (not tested) and it's replaced with interface B that's also verified to be correct, why would there be any regressions? How are you verifying that interface A and interface B are correct? You would either need to test manually, test automatically (i.e. test automation, unit tests, integration tests, fuzzing, and perhaps even mutation testing or design by contract), or maybe you are using formal methods (proving correctness with mathematical rigor). If you are not testing, how are you verifying that the code is correct? "It works on my machine" or "my teammate peer reviewed it and did not see anything wrong" are pretty low standards and how a lot of bugs get shipped and discovered in production. >> The reason I see not to do it is it takes time away from writing correct code. Again, how do you know the code is correct? Spending more time writing code is pointless if the code has bugs and exploits.

saagarjha 4y ago

When Andrew Wiles writes a flawed proof he gets to think for a while and try to fix it. When someone puts broken code into Linux someone else gets to think for a whole and try to fix it. Except they’re losing thousands per hour or someone is having their phone hacked. Unit tests are just one part of the story, but they can be very important.

Xelynega 4y ago
I think their point is that Andrew Wiles' flawed proof was shown to a community and improved without any automated testing because something can be verified before it's integrated and that verification never changes. You don't need testing for verified code interacting with verified code in an environment verified to run verified code, but testing takes half the time for "good enough" reliability so we don't spend the time.
MaxBarraclough 4y ago
I'm not sure I see the point of this comparison. Publication of a bad proof with an untrue conclusion may lead to man-years of wasted research effort. The dollar figure of the cost of a bad proof, or of a kernel bug, will always depend on the specifics of the case.

thesuperbigfrog 4y ago

>> You can't unit test your way to secure, correct code in the kernel either. Only a community of testers and verifiers can do that. Unit tests cannot show the presence of bugs that you are not teating for, but they can show the absence of bugs you've fixed and do not want to re-introduce / regressions. Not having unit tests strikes me as either hubris or lack of resources. Are you really going to have people run repeated tests even if some of the tests could be automated? If a test cannot be automated, then that's understandable, but why wouldn't you automate what you can with unit tests and integration tests? I would further argue that even if unit tests, integration tests, and a community of testers and verifiers are working together there will still be bugs, some of which will be critical. Why wouldn't you use every tool at your disposal in addition to community / people to find and fix the bugs?

Xelynega 4y ago
If interface A is verified to be correct code(not tested) and it's replaced with interface B that's also verified to be correct, why would there be any regressions? The reason I see not to do it is it takes time away from writing correct code. I could spend half my time writing unit tests or I can spend twice the time writing the code in a verifiable language.
- thesuperbigfrog 4y ago
  >> If interface A is verified to be correct code (not tested) and it's replaced with interface B that's also verified to be correct, why would there be any regressions? How are you verifying that interface A and interface B are correct? You would either need to test manually, test automatically (i.e. test automation, unit tests, integration tests, fuzzing, and perhaps even mutation testing or design by contract), or maybe you are using formal methods (proving correctness with mathematical rigor). If you are not testing, how are you verifying that the code is correct? "It works on my machine" or "my teammate peer reviewed it and did not see anything wrong" are pretty low standards and how a lot of bugs get shipped and discovered in production. >> The reason I see not to do it is it takes time away from writing correct code. Again, how do you know the code is correct? Spending more time writing code is pointless if the code has bugs and exploits.