7 ms·

Is it that they just now started actually testing these implementations, for the first time? Even if you wouldn't follow TDD practices, it's obvious that anythi

by Geee 13y ago

Is it that they just now started actually testing these implementations, for the first time? Even if you wouldn't follow TDD practices, it's obvious that anything that implements a protocol of some sort (particularly a security protocol), should be tested to actually follow that protocol. Isn't that common sense? What is going on?

tptacek 13y ago
TLS stacks aren't built with TDD practices for the same reason that the kernel TCP/IP stack isn't built with TDD: the things that go wrong in TLS involve the state machines of a distributed protocol and somewhat elaborate mathematical operations. Read Schneier and Ferguson describing in _Cryptographic Engineering_ the steps required to validate a bignum arithmetic library. That's one of the smaller problems in getting a TLS stack right, and it's complicated. If you demand that your browser implement protocols the same way ActiveRecord implements SQL join generation, you should just stop using browsers; none of them are built the way you'd want them to be. I have a hard time not being dismissive about TDD's role in improving systems software security. If you're going to propose a radical change in the way systems code is written (or, worse, hold systems developers to a nonexistent standard), be intellectually coherent about it: demand that security-critical code be implemented in a rigorous language, like idiomatic Haskell. Don't propose monumental changes that promise only that the resulting code will be asymptotically secure as Ruby on Rails.
- jsmthrowaway 13y ago
  That's not what he said at all and your comment dips into "the elite few know more than you do" territory, particularly with the unnecessary dig at Web development. His only mention of TDD was "even if you don't," yet you jumped on the acronym TDD and took him to task for it. What I read him as saying (and agree with) is that he's surprised that GnuTLS isn't tested against known-bad certificates in a simple integration test that doesn't require a Ph.D. to set up, as you imply.
  - tptacek 13y ago
    What "known-bad" certificate? You appear to be suffering from hindsight bias. We know what the bad certs look like now, but they're idiosyncratically related to a very, very specific bug: having the test case that checks for this bug implies that you don't have the bug to begin with. The fallacy that you could have easily written a test case for this bug appears neatly encapsulates the weakness of "TDD" as a mitigation for programming flaws.
    - jsmthrowaway 13y ago
      My point wasn't the specificity of the argument, merely that you had misinterpreted the argument and gone after its originator misguidedly, to the conversation's detriment, something you're continuing to do with me by accusing me of hindsight bias. Your résumé is well-known around here and you don't have to set the terms of the discussion about every security-related issue on Hacker News, particularly when it's this heavy-handed and you get recognition upvotes. All I'm saying.
      - tptacek 13y ago
        I no longer have any idea what you're talking about. If you want to pose an argument about how TDD applies to building things like TLS stacks, I will read, consider, and respond. Debating TDD for TLS stacks isn't a waste of time, even if I do think the debate has a foregone conclusion. If you want to talk about me personally, I'm not interested.
        jsmthrowaway 13y ago
        Okay, small words: Him: Can't you test for this somehow? Even without TDD... You: TDD is bad because math and wall of text. Go read a book and realize why you can't, Web hacker. Me: You jumped on TDD unnecessarily, there. You: Now I'm going to redirect the argument to you! You're browbeating anybody that comments here, rather unnecessarily, almost as a display of expertise.
- strictfp 13y ago
  With all due respect I must say that I find that you underestimate how much tests help. If I implement a network protocol I will make sure to write automated test involving clients and servers. Setting this up on localhost or on a virtual network using TUN/TAP is not that hard. And this has made me find TONS of bug ahead of time. I hear a lot of arguments as to why testing networked or otherwise distributed things isn't necessary. But just look at Aphyrs complete destruction of well-known distributed systems by using realistic testing. And don't claim that the kernel network stack isn't tested. It's just tested by hand. Plus, there are projects like autotest. Without testing, I don't think there's any chance that the kernel devs could release any more versions. Safe languages, formal verification, using proven methodologies and testing are all separate methods of improving code quality. But I don't believe any one precludes the other. I'll end with a famous quote from Knuth: "Beware of bugs in the above code; I have only proved it correct, not tried it."
  - cube13 13y ago
    >If I implement a network protocol I will make sure to write automated test involving clients and servers. Setting this up on localhost or on a virtual network using TUN/TAP is not that hard. And this has made me find TONS of bug ahead of time. I agree that it's helpful, but the problem is coverage. Your test code, most likely, doesn't cover EVERY single possible condition that can happen with a simple TCP/IP connection. Especially once you get out of localhost land, where you're dealing not only with your code, but all the hardware and software between the two systems The fundamental problem is that even with a rigorous test suite, you're probably going to run into things you didn't even think were possible once the code is out in the wild. For example, we just ran into a scenario where we were seeing corruption through a TCP connection. Knowing the wire, it was impossible for the packets to be appearing in the way that they were(there was some packet level corruption). After going through multiple wireshark logs, we found that the culprit was a hardware firewall in between the server and client. Thankfully, our code didn't crash, but it's also something that we never tested for, because(in theory, at least), it should never be possible for that specific corruption to be sent in the first place.
- acqq 13y ago
  It's definitely not a rocket science to have the test examples with the crafted certificates that are not supposed to pass the tests and to have such tests run at every build. Most CPAN Perl modules for almost two decades have such tests for much less critical stuff. There's no excuse for not having such methods for security critical code. Tptacek is muddying the waters by claiming it's hard. It's not. We have more different implementations of the same protocols so it's even easier to cross-verify.
  - tptacek 13y ago
    That would be a more compelling argument if any other TLS stack had ever had this particular validation bug in it. As it stands, the test case that you suppose should have existed is specific to exactly the code in GnuTLS, and applies to no other stack.
    - acqq 13y ago
      What is GnuTLS specific in "The failure may allow attackers using a self-signed certificate to pose as the cryptographically authenticated operator of a vulnerable website and to decrypt protected communications"? [1] Apparently just a self signed cert. It was accepted as the "CA signed." Since 2005. 1) http://arstechnica.com/security/2014/03/critical-crypto-bug-leaves-linux-hundreds-of-apps-open-to-eavesdropping/ http://arstechnica.com/security/2014/03/critical-crypto-bug-...
      - tptacek 13y ago
        No, that is not the bug.
  - jsmthrowaway 13y ago
    Seriously, all that's being said is: - Here's a library that claims to validate certificates. - Here's a TLS httpd with a forged commonName. - for ciphersuite in $supported; do Completing this test is an exercise for the reader, yet we're waxing philosophical about the perils of Ruby on Rails in this thread for some reason. If I'm being told by a security expert that such a framework would not be helpful, that's concerning and makes me wonder how many bugs such a framework would uncover (given that this one remained untouched for a decade).
- ambrop7 13y ago
  This is bullshit. You can write tests for TLS libraries just fine. And yes, I'm saying that without ever doing any programming of TLS libraries (just using them). For certificate verification, you craft various correct and incorrect certificates to cover the various scenarios involved in certificate verification. For communication related code, you talk to the tested code over TCP or internally as a transport layer implementation (assuming the TLS library supports user defined transport layers). Same as above, you identify the various code paths and attempt to test them all. > for the same reason that the kernel TCP/IP stack isn't built with TDD I believe the primary reason why testing kernels is hard is that they are intended to run on real hardware and often with strict timing expectations. This is not the case for TLS libraries, which normally run in user space.
  - tptacek 13y ago
    If you knew every possible incorrect certificate format, or even every possible form a valid certificate could take, your comment would be more compelling. But of course, you don't. Nobody does.
    - ambrop7 13y ago
      If you're writing that code, you SHOULD know every possible incorrect certificate format. If you don't, you should learn from the relevant standards and other materials, or admit that you're not capable of writing that kind of code. But in this case this is not about that. It's about testing code already written. All you need to do is go through that code and verify that every branch does what is supposed to do. Which is pretty straightforward, if time consuming.
      - clogston 13y ago
        > If you're writing that code, you SHOULD know every possible incorrect certificate format. If we define a correct format there are likely an infinite number of incorrect formats, no? A test explicitly checking for this bug would prevent a fix from regressing, but it seems to write a test that exploits this bug before understanding the bug itself would require quite a bit of luck. EDIT: I'm re-reading the initial advisory and trying to decide if this applies to any cert with a ROOT CA fail or just a specifically crafted one. If it's the former my initial comment is garbage.
- viraptor 13y ago
  I find this comment confusing. You claim the testing isn't done because the issues are either in state machines or in the math... yet we're looking right now at a case where neither failed and some simple unit testing would catch the issue. Isn't it a false dichotomy to say it's not worth unit testing (not TDD) unless we switch to safer languages? If the code is less secure than RoR because of lack of testing, then what's wrong with getting it to that level first? At least it won't require a rewrite from scratch.
  - tptacek 13y ago
    This bug is entirely random. It causes certificate validation to break at a point where certificate validation never breaks. Obviously, once you know it's there, a unit test keeps you from ever having the bug again. It's less apparent to me that it helps you before you realize that the bug could exist.
    - bguthrie 13y ago
      Unit tests may be written before or after the production code in question has been written. They provide no guarantee of the correctness of the final code, but they may find small things. Corrective unit tests as you've described them will not help you find this bug. But finding bugs like the above is TDD's bread and butter. TDD dictates specifically that the test must be written first, that it must be isolated to a particular spot as much as possible, and the dumbest piece of code possible must be written in order to allow the author to move on to writing the next test. Someone TDDing this method would stub out any required external state in order to focus only on the piece of code in front of them. The final system may not be correct––you must still perform high-level testing as you always would, and you must understand the rules of the system you're building. But you're apt to avoid the simple stuff.
    - viraptor 13y ago
      I don't think i can agree with "random". Anyone going for maximum branch coverage would find this problem. Also every point is where validation never breaks... until it does. It's like the apple openssl bug - trivial to find, provided anyone actually tests the code they release, trying to exercise all code paths.
- diminoten 13y ago
  You don't have to follow TDD at all in order to build a test suite for a project. I think all anyone's suggesting is that some of the complicated nature of getting crypto right actually can be put into a suite of tests. I know I could come up with a very basic set of test cases, and my only experience with TLS is reading parts of some book I found in my company's library. Heck, OpenSSL has a ton of tests it runs on every build, I know this much. And even in Haskell you'd need tests. At no time will you ever be able to write code without testing it.
  - codygman 13y ago
    I do wonder if using Haskell and property based testing in a TLS library would catch many of these bugs. Would there be no value in testing against randomly generated certificates?
- leoc 13y ago
  The bug in this case was in fact one which could readily have been caught by unit tests, no? I don't see why demanding test coverage is intellectually incoherent. Arguably it's somewhat half-assed, but not incoherent. Why shouldn't systems developers at least pick up the low-hanging fruit of tests? The effort isn't too monumental for Rails.
- jsnell 13y ago
  Not sure I can agree with the statement about TCP stacks. Yes, kernel TCP stack aren't built with TDD. But it's not a fundamental property of the problem space, it's just another instance of kernel programming sucking in general. A TCP stack can be written in a way that makes it simple to test, and getting full branch coverage isn't that onerous a task. Does that catch all bugs? Of course not. But it'll still catch many simple logic errors like this one. Such logic errors do creep into code quite often, can be hard to trigger or detect during manual testing, and will be hellish to debug if they manage to get all the way through to a live system. (This isn't idle speculation. I'm the lead engineer for a high performance userspace TCP stack that's handled petabytes of data, probably for millions of endpoints. We have deterministic unit tests for a large proportion of TCP behavior. Getting into a position where we could do proper testing took a while since the initial version was not written with testability in mind, but that work was definitely worth it. But I'll take your word on TLS.)
mikeash 13y ago
Testing error handling is sadly uncommon. Lots of programmers simply run the code in the success case and call it good. Even those who write automated tests only write tests for the success case. I wouldn't be at all surprised if the author of this code either didn't test invalid certificates at all, or only tested them briefly in a way that just happened not to trigger the bug.

by Geee 13y ago

tptacek 13y ago

TLS stacks aren't built with TDD practices for the same reason that the kernel TCP/IP stack isn't built with TDD: the things that go wrong in TLS involve the state machines of a distributed protocol and somewhat elaborate mathematical operations. Read Schneier and Ferguson describing in _Cryptographic Engineering_ the steps required to validate a bignum arithmetic library. That's one of the smaller problems in getting a TLS stack right, and it's complicated. If you demand that your browser implement protocols the same way ActiveRecord implements SQL join generation, you should just stop using browsers; none of them are built the way you'd want them to be. I have a hard time not being dismissive about TDD's role in improving systems software security. If you're going to propose a radical change in the way systems code is written (or, worse, hold systems developers to a nonexistent standard), be intellectually coherent about it: demand that security-critical code be implemented in a rigorous language, like idiomatic Haskell. Don't propose monumental changes that promise only that the resulting code will be asymptotically secure as Ruby on Rails.

jsmthrowaway 13y ago
That's not what he said at all and your comment dips into "the elite few know more than you do" territory, particularly with the unnecessary dig at Web development. His only mention of TDD was "even if you don't," yet you jumped on the acronym TDD and took him to task for it. What I read him as saying (and agree with) is that he's surprised that GnuTLS isn't tested against known-bad certificates in a simple integration test that doesn't require a Ph.D. to set up, as you imply.
- tptacek 13y ago
  What "known-bad" certificate? You appear to be suffering from hindsight bias. We know what the bad certs look like now, but they're idiosyncratically related to a very, very specific bug: having the test case that checks for this bug implies that you don't have the bug to begin with. The fallacy that you could have easily written a test case for this bug appears neatly encapsulates the weakness of "TDD" as a mitigation for programming flaws.
  - jsmthrowaway 13y ago
    My point wasn't the specificity of the argument, merely that you had misinterpreted the argument and gone after its originator misguidedly, to the conversation's detriment, something you're continuing to do with me by accusing me of hindsight bias. Your résumé is well-known around here and you don't have to set the terms of the discussion about every security-related issue on Hacker News, particularly when it's this heavy-handed and you get recognition upvotes. All I'm saying.
    - tptacek 13y ago
      I no longer have any idea what you're talking about. If you want to pose an argument about how TDD applies to building things like TLS stacks, I will read, consider, and respond. Debating TDD for TLS stacks isn't a waste of time, even if I do think the debate has a foregone conclusion. If you want to talk about me personally, I'm not interested.
      - jsmthrowaway 13y ago
        Okay, small words: Him: Can't you test for this somehow? Even without TDD... You: TDD is bad because math and wall of text. Go read a book and realize why you can't, Web hacker. Me: You jumped on TDD unnecessarily, there. You: Now I'm going to redirect the argument to you! You're browbeating anybody that comments here, rather unnecessarily, almost as a display of expertise.
strictfp 13y ago
With all due respect I must say that I find that you underestimate how much tests help. If I implement a network protocol I will make sure to write automated test involving clients and servers. Setting this up on localhost or on a virtual network using TUN/TAP is not that hard. And this has made me find TONS of bug ahead of time. I hear a lot of arguments as to why testing networked or otherwise distributed things isn't necessary. But just look at Aphyrs complete destruction of well-known distributed systems by using realistic testing. And don't claim that the kernel network stack isn't tested. It's just tested by hand. Plus, there are projects like autotest. Without testing, I don't think there's any chance that the kernel devs could release any more versions. Safe languages, formal verification, using proven methodologies and testing are all separate methods of improving code quality. But I don't believe any one precludes the other. I'll end with a famous quote from Knuth: "Beware of bugs in the above code; I have only proved it correct, not tried it."
- cube13 13y ago
  >If I implement a network protocol I will make sure to write automated test involving clients and servers. Setting this up on localhost or on a virtual network using TUN/TAP is not that hard. And this has made me find TONS of bug ahead of time. I agree that it's helpful, but the problem is coverage. Your test code, most likely, doesn't cover EVERY single possible condition that can happen with a simple TCP/IP connection. Especially once you get out of localhost land, where you're dealing not only with your code, but all the hardware and software between the two systems The fundamental problem is that even with a rigorous test suite, you're probably going to run into things you didn't even think were possible once the code is out in the wild. For example, we just ran into a scenario where we were seeing corruption through a TCP connection. Knowing the wire, it was impossible for the packets to be appearing in the way that they were(there was some packet level corruption). After going through multiple wireshark logs, we found that the culprit was a hardware firewall in between the server and client. Thankfully, our code didn't crash, but it's also something that we never tested for, because(in theory, at least), it should never be possible for that specific corruption to be sent in the first place.
acqq 13y ago
It's definitely not a rocket science to have the test examples with the crafted certificates that are not supposed to pass the tests and to have such tests run at every build. Most CPAN Perl modules for almost two decades have such tests for much less critical stuff. There's no excuse for not having such methods for security critical code. Tptacek is muddying the waters by claiming it's hard. It's not. We have more different implementations of the same protocols so it's even easier to cross-verify.
- tptacek 13y ago
  That would be a more compelling argument if any other TLS stack had ever had this particular validation bug in it. As it stands, the test case that you suppose should have existed is specific to exactly the code in GnuTLS, and applies to no other stack.
  - acqq 13y ago
    What is GnuTLS specific in "The failure may allow attackers using a self-signed certificate to pose as the cryptographically authenticated operator of a vulnerable website and to decrypt protected communications"? [1] Apparently just a self signed cert. It was accepted as the "CA signed." Since 2005. 1) http://arstechnica.com/security/2014/03/critical-crypto-bug-leaves-linux-hundreds-of-apps-open-to-eavesdropping/ http://arstechnica.com/security/2014/03/critical-crypto-bug-...
    - tptacek 13y ago
      No, that is not the bug.
- jsmthrowaway 13y ago
  Seriously, all that's being said is: - Here's a library that claims to validate certificates. - Here's a TLS httpd with a forged commonName. - for ciphersuite in $supported; do Completing this test is an exercise for the reader, yet we're waxing philosophical about the perils of Ruby on Rails in this thread for some reason. If I'm being told by a security expert that such a framework would not be helpful, that's concerning and makes me wonder how many bugs such a framework would uncover (given that this one remained untouched for a decade).
ambrop7 13y ago
This is bullshit. You can write tests for TLS libraries just fine. And yes, I'm saying that without ever doing any programming of TLS libraries (just using them). For certificate verification, you craft various correct and incorrect certificates to cover the various scenarios involved in certificate verification. For communication related code, you talk to the tested code over TCP or internally as a transport layer implementation (assuming the TLS library supports user defined transport layers). Same as above, you identify the various code paths and attempt to test them all. > for the same reason that the kernel TCP/IP stack isn't built with TDD I believe the primary reason why testing kernels is hard is that they are intended to run on real hardware and often with strict timing expectations. This is not the case for TLS libraries, which normally run in user space.
- tptacek 13y ago
  If you knew every possible incorrect certificate format, or even every possible form a valid certificate could take, your comment would be more compelling. But of course, you don't. Nobody does.
  - ambrop7 13y ago
    If you're writing that code, you SHOULD know every possible incorrect certificate format. If you don't, you should learn from the relevant standards and other materials, or admit that you're not capable of writing that kind of code. But in this case this is not about that. It's about testing code already written. All you need to do is go through that code and verify that every branch does what is supposed to do. Which is pretty straightforward, if time consuming.
    - clogston 13y ago
      > If you're writing that code, you SHOULD know every possible incorrect certificate format. If we define a correct format there are likely an infinite number of incorrect formats, no? A test explicitly checking for this bug would prevent a fix from regressing, but it seems to write a test that exploits this bug before understanding the bug itself would require quite a bit of luck. EDIT: I'm re-reading the initial advisory and trying to decide if this applies to any cert with a ROOT CA fail or just a specifically crafted one. If it's the former my initial comment is garbage.
viraptor 13y ago
I find this comment confusing. You claim the testing isn't done because the issues are either in state machines or in the math... yet we're looking right now at a case where neither failed and some simple unit testing would catch the issue. Isn't it a false dichotomy to say it's not worth unit testing (not TDD) unless we switch to safer languages? If the code is less secure than RoR because of lack of testing, then what's wrong with getting it to that level first? At least it won't require a rewrite from scratch.
- tptacek 13y ago
  This bug is entirely random. It causes certificate validation to break at a point where certificate validation never breaks. Obviously, once you know it's there, a unit test keeps you from ever having the bug again. It's less apparent to me that it helps you before you realize that the bug could exist.
  - bguthrie 13y ago
    Unit tests may be written before or after the production code in question has been written. They provide no guarantee of the correctness of the final code, but they may find small things. Corrective unit tests as you've described them will not help you find this bug. But finding bugs like the above is TDD's bread and butter. TDD dictates specifically that the test must be written first, that it must be isolated to a particular spot as much as possible, and the dumbest piece of code possible must be written in order to allow the author to move on to writing the next test. Someone TDDing this method would stub out any required external state in order to focus only on the piece of code in front of them. The final system may not be correct––you must still perform high-level testing as you always would, and you must understand the rules of the system you're building. But you're apt to avoid the simple stuff.
  - viraptor 13y ago
    I don't think i can agree with "random". Anyone going for maximum branch coverage would find this problem. Also every point is where validation never breaks... until it does. It's like the apple openssl bug - trivial to find, provided anyone actually tests the code they release, trying to exercise all code paths.
diminoten 13y ago
You don't have to follow TDD at all in order to build a test suite for a project. I think all anyone's suggesting is that some of the complicated nature of getting crypto right actually can be put into a suite of tests. I know I could come up with a very basic set of test cases, and my only experience with TLS is reading parts of some book I found in my company's library. Heck, OpenSSL has a ton of tests it runs on every build, I know this much. And even in Haskell you'd need tests. At no time will you ever be able to write code without testing it.
- codygman 13y ago
  I do wonder if using Haskell and property based testing in a TLS library would catch many of these bugs. Would there be no value in testing against randomly generated certificates?
leoc 13y ago
The bug in this case was in fact one which could readily have been caught by unit tests, no? I don't see why demanding test coverage is intellectually incoherent. Arguably it's somewhat half-assed, but not incoherent. Why shouldn't systems developers at least pick up the low-hanging fruit of tests? The effort isn't too monumental for Rails.
jsnell 13y ago
Not sure I can agree with the statement about TCP stacks. Yes, kernel TCP stack aren't built with TDD. But it's not a fundamental property of the problem space, it's just another instance of kernel programming sucking in general. A TCP stack can be written in a way that makes it simple to test, and getting full branch coverage isn't that onerous a task. Does that catch all bugs? Of course not. But it'll still catch many simple logic errors like this one. Such logic errors do creep into code quite often, can be hard to trigger or detect during manual testing, and will be hellish to debug if they manage to get all the way through to a live system. (This isn't idle speculation. I'm the lead engineer for a high performance userspace TCP stack that's handled petabytes of data, probably for millions of endpoints. We have deterministic unit tests for a large proportion of TCP behavior. Getting into a position where we could do proper testing took a while since the initial version was not written with testability in mind, but that work was definitely worth it. But I'll take your word on TLS.)

mikeash 13y ago

Testing error handling is sadly uncommon. Lots of programmers simply run the code in the success case and call it good. Even those who write automated tests only write tests for the success case. I wouldn't be at all surprised if the author of this code either didn't test invalid certificates at all, or only tested them briefly in a way that just happened not to trigger the bug.