4 ms·

GnuTLS vulnerability: is unit testing a matter of language culture?

michaelfeathers 13y ago
Unit testing is definitely influenced by language culture but there's also the issue of how easy it is to do. JS doesn't have the unit testing uptake that Java/C#/Ruby/Python have, and I think that largely it is because JS is often used in the front-end and people aren't used to having any sort of an abstraction layer. The issue with C is that it is hard to mock. You can use the preprocessor, the linker, or function pointers. The latter is the best option but your code ends up looking quasi-OO with structs of function pointers. It's a style people are used to in some OS programming but it does freak some people out.
dalke 13y ago
I once did a security audit of Fitnesse, perhaps the best known software developed under the red-green-refactor cycle of TDD. I found a number of security vulnerabilities. A path traversal vulnerability meant I could read an arbitrary file on the server, including the password file. The password hash encoding is weak, and given a hash I was able to find a password which would match the hash. Once I had upload access to the server, another path traversal vulnerability meant I could upload to an arbitrary spot on the server, so long as the process had permissions to it. By design, the server can execute Java code. Combined, this means an attacker could run arbitrary code on the server. While the worst vulnerabilities have since been fixed, the server still contains a CSRF vulnerability. These all suggest that unit testing/TDD doesn't really add much to security protection.
- michaelfeathers 13y ago
  > These all suggest that unit testing/TDD doesn't really add much to security protection. I find it hard to believe that anyone would find that surprising. Security is a thing you either have in mind when you are working or not.
  - dalke 13y ago
    The article is titled "GnuTLS vulnerability: is unit testing a matter of language culture". The coupling of vulnerability to unit testing is, IMO, a red herring. I've never seen any evidence that unit testing leads to more secure systems than any other commonly used method, including code review and coverage based functional tests. My observation is that unit testing/TDD doesn't significantly reduce security failures (ie, it may be the same, or it may be worse), and I use my Fitnesse pen test as a real-world example. You seem to agree. The article goes on to say "An automated test suite should have immediately spotted that invalid commit, right." I just downloaded the package. GnuTLS has an automated test suite. There's nearly 22KLOC in the tests/ subdirectory, with 75KLOC in gl/ lib/ and src/ combined. The rest of the article slags on some perceived lack of testing based on C culture ("did you really, honestly, expect a C code base that reaches back more than a decade to be under surveillance of ideal unit-tests, by modern standards"), and implies that "ideal unit-tests, by modern standards" would have caught error. This is unjustified. Moreover, we know what ideal non-unit tests ca. 10+ years ago looks like. For example, SQLite is the exemplar of how a C library, without low-level unit tests, and coupled with strong code coverage, can help produce high quality code. (To tie in with your other post, in part because SQLite uses "structs of function pointers" to make a plug-in filesystem architecture for users, which also allows emulation of unusual file system failures.) If GnuTLS had the ideal tests of the standard of 10+ years ago, then these bugs would still have been found, unit tests or not. As its authors point out in the README, "Thorough testing is very important and expensive." So, "unit tests" isn't really is issue, is it? I would have been happier with an article titled "GnuTLS vulnerability: we are all cheapskates".
    - michaelfeathers 13y ago
      > The article goes on to say "An automated test suite should have immediately spotted that invalid commit, right." I just downloaded the package. GnuTLS has an automated test suite. There's nearly 22KLOC in the tests/ subdirectory, with 75KLOC in gl/ lib/ and src/ combined. Did you see a test for the error? Maybe check to see if there were tests for other parts of code that prevented vulnerabilities? I'm sure there are. First and foremost, people need to be conscious of security to do well. No amount of testing will overcome that. But, if you have security consciousness, I'm sure that TDD can augment it. It's a practice that encourages reflection. Re the article itself, fair enough, it uses a flawed example but I can back up the assertion. I see less unit testing in C codebases.

Unit testing is definitely influenced by language culture but there's also the issue of how easy it is to do. JS doesn't have the unit testing uptake that Java/C#/Ruby/Python have, and I think that largely it is because JS is often used in the front-end and people aren't used to having any sort of an abstraction layer. The issue with C is that it is hard to mock. You can use the preprocessor, the linker, or function pointers. The latter is the best option but your code ends up looking quasi-OO with structs of function pointers. It's a style people are used to in some OS programming but it does freak some people out.

dalke 13y ago

I once did a security audit of Fitnesse, perhaps the best known software developed under the red-green-refactor cycle of TDD. I found a number of security vulnerabilities. A path traversal vulnerability meant I could read an arbitrary file on the server, including the password file. The password hash encoding is weak, and given a hash I was able to find a password which would match the hash. Once I had upload access to the server, another path traversal vulnerability meant I could upload to an arbitrary spot on the server, so long as the process had permissions to it. By design, the server can execute Java code. Combined, this means an attacker could run arbitrary code on the server. While the worst vulnerabilities have since been fixed, the server still contains a CSRF vulnerability. These all suggest that unit testing/TDD doesn't really add much to security protection.

michaelfeathers 13y ago
> These all suggest that unit testing/TDD doesn't really add much to security protection. I find it hard to believe that anyone would find that surprising. Security is a thing you either have in mind when you are working or not.
- dalke 13y ago
  The article is titled "GnuTLS vulnerability: is unit testing a matter of language culture". The coupling of vulnerability to unit testing is, IMO, a red herring. I've never seen any evidence that unit testing leads to more secure systems than any other commonly used method, including code review and coverage based functional tests. My observation is that unit testing/TDD doesn't significantly reduce security failures (ie, it may be the same, or it may be worse), and I use my Fitnesse pen test as a real-world example. You seem to agree. The article goes on to say "An automated test suite should have immediately spotted that invalid commit, right." I just downloaded the package. GnuTLS has an automated test suite. There's nearly 22KLOC in the tests/ subdirectory, with 75KLOC in gl/ lib/ and src/ combined. The rest of the article slags on some perceived lack of testing based on C culture ("did you really, honestly, expect a C code base that reaches back more than a decade to be under surveillance of ideal unit-tests, by modern standards"), and implies that "ideal unit-tests, by modern standards" would have caught error. This is unjustified. Moreover, we know what ideal non-unit tests ca. 10+ years ago looks like. For example, SQLite is the exemplar of how a C library, without low-level unit tests, and coupled with strong code coverage, can help produce high quality code. (To tie in with your other post, in part because SQLite uses "structs of function pointers" to make a plug-in filesystem architecture for users, which also allows emulation of unusual file system failures.) If GnuTLS had the ideal tests of the standard of 10+ years ago, then these bugs would still have been found, unit tests or not. As its authors point out in the README, "Thorough testing is very important and expensive." So, "unit tests" isn't really is issue, is it? I would have been happier with an article titled "GnuTLS vulnerability: we are all cheapskates".
  - michaelfeathers 13y ago
    > The article goes on to say "An automated test suite should have immediately spotted that invalid commit, right." I just downloaded the package. GnuTLS has an automated test suite. There's nearly 22KLOC in the tests/ subdirectory, with 75KLOC in gl/ lib/ and src/ combined. Did you see a test for the error? Maybe check to see if there were tests for other parts of code that prevented vulnerabilities? I'm sure there are. First and foremost, people need to be conscious of security to do well. No amount of testing will overcome that. But, if you have security consciousness, I'm sure that TDD can augment it. It's a practice that encourages reflection. Re the article itself, fair enough, it uses a flawed example but I can back up the assertion. I see less unit testing in C codebases.