5 ms·

Interesting position on testing: "I generally don't consider my own software adequately enough tested until its tests have turned up a bug in a compiler/toolcha

by mjs 12y ago

Interesting position on testing: "I generally don't consider my own software adequately enough tested until its tests have turned up a bug in a compiler/toolchain. So far I've not encountered a compiler bug for libsecp256k1 [...] so I may have to settle for discovering that a ubiquitous system library couldn't square correctly." (The bug was found via comparing the output of libsecp256k1 and OpenSSL on "specially-constructed random inputs.")

illumen 12y ago
Comparing multiple implementations is a great way to find bugs. Even running the same code with different compilers or platforms helps find bugs. It doesn't find cases where they are both wrong, but often finds cases where one of them is wrong. It is a very common testing strategy with maths libraries.
- nickik 12y ago
  You might be intrested in this [1]. Its a 31C3 talk about exactly this. There idea is to write a runable spec (that only focues on correctness, instead of speed and things like that) that then can be used to test against real world application. There idea is to do this against all the MAJOR interfaces that we all depend on TCP/IP, x86 memory model and stuff like that. [1] http://events.ccc.de/congress/2014/Fahrplan/events/6574.html http://events.ccc.de/congress/2014/Fahrplan/events/6574.html (the talk is on youtube or on ccc video streaming side)
  - Anderkent 12y ago
    The talk on youtube: https://www.youtube.com/watch?v=ca0DWaV9uNc https://www.youtube.com/watch?v=ca0DWaV9uNc (I couldn't find a link on that page)
    - aw3c2 12y ago
      That 'CCCen' account is an imposter and evil. Please do not link to it or the companion 'CCCdeVideos'. Some background on this issue: https://events.ccc.de/2015/01/03/the-youtube-and-stream-dump-problem/ https://events.ccc.de/2015/01/03/the-youtube-and-stream-dump... Here is the official CCC channel: https://www.youtube.com/user/mediacccde https://www.youtube.com/user/mediacccde The official recording: http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_saal_1_-_201412301245_-_why_are_computers_so_and_what_can_we_do_about_it_-_peter_sewell.html#video http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_... The official recording on their official YT channel: https://www.youtube.com/watch?v=MBIHPLFmcgA https://www.youtube.com/watch?v=MBIHPLFmcgA
  - edwintorok 12y ago
    It would help if the videos were automatically linked from the event page: http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_saal_1_-_201412301245_-_why_are_computers_so_and_what_can_we_do_about_it_-_peter_sewell.html#video http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_...
  - pjc50 12y ago
    Runnable/verifiable specs are a good idea. A colleague of mine did some great work with compiling the H265 & VP9 specifications to produce a test suite, in a similar manner to fuzzing: http://www.argondesign.com/products/2014/may/27/streamsvp9/ http://www.argondesign.com/products/2014/may/27/streamsvp9/
  - pilsetnieks 12y ago
    > instead of speed Not when developing crypto. You have to look at execution speed, too, to prevent timing attacks.
    - raverbashing 12y ago
      Execution speed variance It doesn't matter if it takes 1s or 10s, but that it takes 10s every time (to verify a password, for example)
    - azdle 12y ago
      Why would that matter in the case of testing for correctness? I thought these 'runable spec' implementations were only to verify that the output of the 'real' implementation is correct and thus would never even be used anywhere that could/would be attacked. Am I not understanding the purpose correctly?
    - Igglyboo 12y ago
      When testing for correctness why would you care about timing attacks? You would obviously have multiple tests covering correctness, performance, security, etc.
derf_ 12y ago
For those not familiar with Greg's work, this presentation on the testing he did for the Opus codec is informative: https://www.ietf.org/proceedings/82/slides/codec-4.pdf https://www.ietf.org/proceedings/82/slides/codec-4.pdf Particularly slide 20.
kosma 12y ago
That's only because author uses a pretty standard compiler. Try building your flawless library on ARM, Newlib, etc. and you're pretty much fighting your tools all the time. The first compiler/toolchain bug is a rite of passage, yes, but not a measure of the code quality (although they do correlate).
- gcp 12y ago
  It does say "a compiler/toolchain". Opus has both fixed point and floating point implementations. The fixed point one was extensively tested on ARM too, I'm sure.

by mjs 12y ago

illumen 12y ago

Comparing multiple implementations is a great way to find bugs. Even running the same code with different compilers or platforms helps find bugs. It doesn't find cases where they are both wrong, but often finds cases where one of them is wrong. It is a very common testing strategy with maths libraries.

nickik 12y ago
You might be intrested in this [1]. Its a 31C3 talk about exactly this. There idea is to write a runable spec (that only focues on correctness, instead of speed and things like that) that then can be used to test against real world application. There idea is to do this against all the MAJOR interfaces that we all depend on TCP/IP, x86 memory model and stuff like that. [1] http://events.ccc.de/congress/2014/Fahrplan/events/6574.html http://events.ccc.de/congress/2014/Fahrplan/events/6574.html (the talk is on youtube or on ccc video streaming side)
- Anderkent 12y ago
  The talk on youtube: https://www.youtube.com/watch?v=ca0DWaV9uNc https://www.youtube.com/watch?v=ca0DWaV9uNc (I couldn't find a link on that page)
  - aw3c2 12y ago
    That 'CCCen' account is an imposter and evil. Please do not link to it or the companion 'CCCdeVideos'. Some background on this issue: https://events.ccc.de/2015/01/03/the-youtube-and-stream-dump-problem/ https://events.ccc.de/2015/01/03/the-youtube-and-stream-dump... Here is the official CCC channel: https://www.youtube.com/user/mediacccde https://www.youtube.com/user/mediacccde The official recording: http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_saal_1_-_201412301245_-_why_are_computers_so_and_what_can_we_do_about_it_-_peter_sewell.html#video http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_... The official recording on their official YT channel: https://www.youtube.com/watch?v=MBIHPLFmcgA https://www.youtube.com/watch?v=MBIHPLFmcgA
- edwintorok 12y ago
  It would help if the videos were automatically linked from the event page: http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_saal_1_-_201412301245_-_why_are_computers_so_and_what_can_we_do_about_it_-_peter_sewell.html#video http://media.ccc.de/browse/congress/2014/31c3_-_6574_-_en_-_...
- pjc50 12y ago
  Runnable/verifiable specs are a good idea. A colleague of mine did some great work with compiling the H265 & VP9 specifications to produce a test suite, in a similar manner to fuzzing: http://www.argondesign.com/products/2014/may/27/streamsvp9/ http://www.argondesign.com/products/2014/may/27/streamsvp9/
- pilsetnieks 12y ago
  > instead of speed Not when developing crypto. You have to look at execution speed, too, to prevent timing attacks.
  - raverbashing 12y ago
    Execution speed variance It doesn't matter if it takes 1s or 10s, but that it takes 10s every time (to verify a password, for example)
  - azdle 12y ago
    Why would that matter in the case of testing for correctness? I thought these 'runable spec' implementations were only to verify that the output of the 'real' implementation is correct and thus would never even be used anywhere that could/would be attacked. Am I not understanding the purpose correctly?
  - Igglyboo 12y ago
    When testing for correctness why would you care about timing attacks? You would obviously have multiple tests covering correctness, performance, security, etc.

derf_ 12y ago

For those not familiar with Greg's work, this presentation on the testing he did for the Opus codec is informative: https://www.ietf.org/proceedings/82/slides/codec-4.pdf https://www.ietf.org/proceedings/82/slides/codec-4.pdf Particularly slide 20.

kosma 12y ago

That's only because author uses a pretty standard compiler. Try building your flawless library on ARM, Newlib, etc. and you're pretty much fighting your tools all the time. The first compiler/toolchain bug is a rite of passage, yes, but not a measure of the code quality (although they do correlate).

gcp 12y ago
It does say "a compiler/toolchain". Opus has both fixed point and floating point implementations. The fixed point one was extensively tested on ARM too, I'm sure.