4 ms·

Aw man, I was nodding along with the "most examples suck" section and then... it ended :(

pfdietz 1y ago
I'll give you the good example I've been doing for the last two decades: testing a compiler. The complexity here is the complete opposite of the simple toy examples. What are the edge cases of an optimizing compiler? How do you even approach them, if they're buried deep in a chain of transformations? The properties are simple things like "the compiler shouldn't crash, the compiled code shouldn't crash, and code compiled with different optimization levels should do the same thing." This assumes the randomly generated code doesn't touch undefined behavior in the language spec. Here's a recent example of a bug found by this approach. The Common Lisp code stimulating the bug has been automatically minimized: https://bugs.launchpad.net/sbcl/+bug/2109837 https://bugs.launchpad.net/sbcl/+bug/2109837 with the bug fix https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24fa8d1d7eefd36a7b3af/ https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24...
- chipsrafferty 1y ago
  Unfortunately I have no idea what I'm looking at here.
  - pfdietz 1y ago
    The function constructs two lambda expressions (source code for anonymous functions) that should be equivalent. One has some extra declarations. It then compiles the two lambda expressions and calls the compiled code on the same arguments, and gets different values (which is the bug).
hikarudo 1y ago
Here's a great talk by John Hughes, one of the authors of QuickCheck, with real-life examples: https://www.youtube.com/watch?v=zi0rHwfiX1Q https://www.youtube.com/watch?v=zi0rHwfiX1Q
- frogulis 1y ago
  Thanks, that's gotta be one of the best talks I've ever watched. A passionate speaker, talking about fascinating, useful tech, giving specific real world examples of utility, AND showing how to actually apply it for interesting "dirty" situations.
nickpsecurity 1y ago
They should put some good ones in the article.

I'll give you the good example I've been doing for the last two decades: testing a compiler. The complexity here is the complete opposite of the simple toy examples. What are the edge cases of an optimizing compiler? How do you even approach them, if they're buried deep in a chain of transformations? The properties are simple things like "the compiler shouldn't crash, the compiled code shouldn't crash, and code compiled with different optimization levels should do the same thing." This assumes the randomly generated code doesn't touch undefined behavior in the language spec. Here's a recent example of a bug found by this approach. The Common Lisp code stimulating the bug has been automatically minimized: https://bugs.launchpad.net/sbcl/+bug/2109837 https://bugs.launchpad.net/sbcl/+bug/2109837 with the bug fix https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24fa8d1d7eefd36a7b3af/ https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24...

chipsrafferty 1y ago
Unfortunately I have no idea what I'm looking at here.
- pfdietz 1y ago
  The function constructs two lambda expressions (source code for anonymous functions) that should be equivalent. One has some extra declarations. It then compiles the two lambda expressions and calls the compiled code on the same arguments, and gets different values (which is the bug).

hikarudo 1y ago

Here's a great talk by John Hughes, one of the authors of QuickCheck, with real-life examples: https://www.youtube.com/watch?v=zi0rHwfiX1Q https://www.youtube.com/watch?v=zi0rHwfiX1Q

frogulis 1y ago
Thanks, that's gotta be one of the best talks I've ever watched. A passionate speaker, talking about fascinating, useful tech, giving specific real world examples of utility, AND showing how to actually apply it for interesting "dirty" situations.

nickpsecurity 1y ago

They should put some good ones in the article.