5 ms·

> you wouldn't believe how strong the temptation to punt on those test vectors can be Tests on test vectors is the first thing I write, before I even start fil

by arnarbi 10y ago

> you wouldn't believe how strong the temptation to punt on those test vectors can be Tests on test vectors is the first thing I write, before I even start filling in the implementation of a primitive. I don't do this out of principle, but because I've never managed to get an implementation of a primitive right without it. Parent is right, test vectors are almost part of the declaration - and don't really count as actual testing. Side-note: Props for your humility towards criticism, both here and on reddit. > only one .c and one .h file If your library gains popularity, I predict this will be one of the largest contributing factors. In particular for embedded projects, the ability to just plop in a dependency free file is huge ("programmers are like water" and "path of least resistance" etc..)

loup-vaillant 10y ago
> I don't do this out of principle, but because I've never managed to get an implementation of a primitive right without it. My journey looked a bit like this. First implement all the thing! Cool it compiles! I'm tired… Okay, let's do the test vectors, because crypto-critical-yada-yada. Aw crap. It was all broken. Okay, let's dig it up… By the way, one worry I have right now is that I may not have enough test vectors. Do they hit all the code paths? Do they sufficiently test for overflow? That's why I display the number of test vectors involve in each tests when running the test suite.
- tptacek 10y ago
  Are you maybe using the term "test vectors" differently than we are? The test vectors for a crypto construction are just a series of test transforms, provided to make sure you implemented the same transform function as everyone else. They're not a suite of security checks. Most of the most serious misuse errors with crypto constructions don't have anything like "test vectors"; that's what was so exciting about Google's Wycheproof.
  - loup-vaillant 10y ago
    > Are you maybe using the term "test vectors" differently than we are? I don't think so. Check out the test suite in my tarball, there are files named "vectors", with sets of inputs and expected outputs. In addition, I have written consistency checks for all constructions (encryption-decryption roundtrip yields the original plaintext, legitimate messages are accepted, forgeries are rejected…) I feel I should do more, but I'm not sure what exactly.
    - tptacek 10y ago
      Right, then, so these aren't really security tests. I'm suggesting you reword that bit on your page; passing test vectors doesn't make something more trustworthy; it's table stakes. I'm not saying you shouldn't write tests for misuse cases! You should do that.
      - loup-vaillant 10y ago
        Okay, what's a security test? How is it any different from a mere correctness test? Can you suggest some link to read up, or some tool to use?
        arnarbi 10y ago
        See e.g. https://github.com/google/wycheproof https://github.com/google/wycheproof Fuzzing tests can also uncover weird bugs and edge cases that can lead to vulnerabilities.

by arnarbi 10y ago

loup-vaillant 10y ago

> I don't do this out of principle, but because I've never managed to get an implementation of a primitive right without it. My journey looked a bit like this. First implement all the thing! Cool it compiles! I'm tired… Okay, let's do the test vectors, because crypto-critical-yada-yada. Aw crap. It was all broken. Okay, let's dig it up… By the way, one worry I have right now is that I may not have enough test vectors. Do they hit all the code paths? Do they sufficiently test for overflow? That's why I display the number of test vectors involve in each tests when running the test suite.

tptacek 10y ago
Are you maybe using the term "test vectors" differently than we are? The test vectors for a crypto construction are just a series of test transforms, provided to make sure you implemented the same transform function as everyone else. They're not a suite of security checks. Most of the most serious misuse errors with crypto constructions don't have anything like "test vectors"; that's what was so exciting about Google's Wycheproof.
- loup-vaillant 10y ago
  > Are you maybe using the term "test vectors" differently than we are? I don't think so. Check out the test suite in my tarball, there are files named "vectors", with sets of inputs and expected outputs. In addition, I have written consistency checks for all constructions (encryption-decryption roundtrip yields the original plaintext, legitimate messages are accepted, forgeries are rejected…) I feel I should do more, but I'm not sure what exactly.
  - tptacek 10y ago
    Right, then, so these aren't really security tests. I'm suggesting you reword that bit on your page; passing test vectors doesn't make something more trustworthy; it's table stakes. I'm not saying you shouldn't write tests for misuse cases! You should do that.
    - loup-vaillant 10y ago
      Okay, what's a security test? How is it any different from a mere correctness test? Can you suggest some link to read up, or some tool to use?
      - arnarbi 10y ago
        See e.g. https://github.com/google/wycheproof https://github.com/google/wycheproof Fuzzing tests can also uncover weird bugs and edge cases that can lead to vulnerabilities.