5 ms·

Microsoft Code Digger

137 points by scottcha 13y ago

testbro 13y ago
A similar tool exists for Java, although it doesn't rely on symbolic execution (but IIRC it is able to use it for integer constraints) but evolutionary test data generation [1]. There's a plugin for Eclipse too. [1] : http://www.evosuite.org http://www.evosuite.org
nickbarnwell 13y ago
I'll prefix my question by saying I'm unfamiliar with QC beyond knowing of its existence and roughly what it does in terms of generating test values: Am I misunderstanding, or is this akin to QuickCheck for C#?
- danbruc 13y ago
  The QuickCheck manual states that they are randomly generating test cases [1]. This tool is build on top of Pex [2] which analyzes the byte code and uses the Z3 theorem prover [3] to systematically find inputs covering all code paths. This is really a impressive and powerful tool. You can have some fun with it at the Pex for fun site [4]. There is some secret code and you have to reimplement that code based on the test results generated by Pex comparing your implementation to the secret implementation. [1] http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html [2] http://research.microsoft.com/en-us/projects/pex/ http://research.microsoft.com/en-us/projects/pex/ [3] http://research.microsoft.com/en-us/projects/z3m/ http://research.microsoft.com/en-us/projects/z3m/ [4] http://www.pexforfun.com/ http://www.pexforfun.com/
  - 0x0 13y ago
    The pexforfun thing seemed pretty clever: // What values of v and i can cause an exception? Ask Pex to find out! public static void Puzzle(int[] v, int i) { if (v[i + 2] + 5 == v.Length + i) throw new Exception("hidden bug!"); } and then it automatically finds the breaking input: {-6} -2 Exception hidden bug!
- porges 13y ago
  It's better than QuickCheck. It performs whitebox testing - that is, it has knowledge of the code and actively uses this to generate inputs that explore all possible branches.
earlz 13y ago
I tried it. Doesn't usually work all that well though, and basically only works on built in .Net types. If you pass in your own object containing only .Net types, it's not smart enough to fill those in with potential test data. Interesting idea, but I found it completely worthless
- danbruc 13y ago
  I can not test it at the moment but you may have to do the object creation inside a wrapper method. public static void TestWrapper(String foo, Int32 bar, Boolean buzz, Decimal foobar) { var thing = new Thing(foo, bar); thing.Buzz = buzz; Test(thing, blah); } public static Test(Thing thing, Decimal foobar) { thing.DoStuff(); if (thing.State == 42) { thing.DoOtherStuff(foobar); } } In my experience Pex yields really great results when you use it to analyze methods that are close to mathematically functions with not to many side effects and especially state mutation. Tracking changing state over time and finding sequences of operations to prepare an object to a certain state and then checking the behavior of the object in this state is much harder than analyzing a (almost) pure function.
  - georgefox 13y ago
    You can test it online at pex4fun.com. Here are two tests, one using a function that takes built-in types as parameters and instantiates a Thing to operate on, and a second that takes a Thing input and operates on it directly. It seems to generate interesting cases for both, but it's much more easily readable with the more basic input types, as the parameters are easily displayed in the table. http://pex4fun.com/default.aspx?language=CSharp&code=OgPsvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee__997o7nU4n99%2f%2fP1xmZAFs9s5K2smeIYCqyB8%2ffnwfPyLWTbG8SF9fN22_OEyS1XpSFtN0WmZNk76Z03fJL05SelZ1cZm1edq0NV54VlXpL04v8vYwbfDPLwkaFcs2fZLVG1pMqqpMn6x%2f8INIG2koiDAKW9rpeVWNGPYkq_9wK8ENTzsvmjGw_gzNDsPPgctneEs_73ejHTytXrfr8%2fOtLvA6b9f1Mt26n97lcT1Od__kvweIMH5TfbVa5fXWnfSR%2fv28usLfN3b1ZTvPa_lvlk_LRVYC88jQTO%2fo_a62_UAUstZh8pJmocy3Xns0Plu29%2fZAr1H6hCYqz5bphFqN0qcbEb0kvFpMGBF7mV%2fp5DFEtCb2Mi251Zjn%2fzMG7X2nw32TN_0WtxuZ7sx4hsfCL3G3qb66GWPtS9Cxs59_Yj%2fxJqmDxC%2f5fwIAAP%2f%2f http://pex4fun.com/default.aspx?language=CSharp&code=OgP... http://pex4fun.com/default.aspx?language=CSharp&code=_QLsvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee__997o7nU4n99%2f%2fP1xmZAFs9s5K2smeIYCqyB8%2ffnwfPyLWTbG8SF9fN22_OEyS1XpSFtN0WmZNk76Z03fJL05SelZ1cZm1edq0NV54VlXpL04v8vYwbfDPLwkaFcs2fZLVG1pMqqpMn6x%2f8INIG2koiDAKW9rpeVWNGPYkq_9wK8ENTzsvmjGw_gzNDsPPgctneEs_73ejHTytXrfr8%2fOtLvA6b9f1Mt26n97lcT1Od__kvweIMH5TfbVa5fXWnfSR%2fv28usLfN3b1ZTvPa_lvlk_LRVYC88jQTO%2fo_a62_UAUstZh8pJmocy3mNJpO0qf0NTk2TKd0Oej9OlG1Noxz_Fn3NgRXTF_kzftFkHUdw06w6jwC4oI%2fr2pe_2H247t5KWf2E88GneQ_CX%2fTwAAAP%2f%2f http://pex4fun.com/default.aspx?language=CSharp&code=_QL...
Aurel1us 13y ago
Anyone know something similar for C/C++ or Python ?
- monksy 13y ago
  Or for Java.
  - danbruc 13y ago
    https://news.ycombinator.com/item?id=5676010 https://news.ycombinator.com/item?id=5676010
  - grdvnl 13y ago
    Pex is based on the Daikon dynamic invariant. This document explains some results and examples in certain languages. http://groups.csail.mit.edu/pag/daikon/download/doc/daikon.pdf http://groups.csail.mit.edu/pag/daikon/download/doc/daikon.p...
    - rainforest 13y ago
      Is it? I was under the impression it just uses symbolic execution rather than invariant detection.
      - grdvnl 13y ago
        Your are right. I mixed up dynamic symbolic execution with invariant detection. I know there is something called DySy which does do invariant detection using dynamic symbolic execution.
    - danbruc 13y ago
      Pex is based on the Daikon dynamic invariant. That's not true.
- judah 13y ago
  It's based on the Z3 solver, here's Z3Py: http://rise4fun.com/z3py http://rise4fun.com/z3py
Yuioup 13y ago
I've noticed here and on reddit that Microsoft stories don't get much attention. Rightly so?
suyash 13y ago
Anything similar for JavaScript?

137 points by scottcha 13y ago

A similar tool exists for Java, although it doesn't rely on symbolic execution (but IIRC it is able to use it for integer constraints) but evolutionary test data generation [1]. There's a plugin for Eclipse too. [1] : http://www.evosuite.org http://www.evosuite.org

nickbarnwell 13y ago

I'll prefix my question by saying I'm unfamiliar with QC beyond knowing of its existence and roughly what it does in terms of generating test values: Am I misunderstanding, or is this akin to QuickCheck for C#?

danbruc 13y ago
The QuickCheck manual states that they are randomly generating test cases [1]. This tool is build on top of Pex [2] which analyzes the byte code and uses the Z3 theorem prover [3] to systematically find inputs covering all code paths. This is really a impressive and powerful tool. You can have some fun with it at the Pex for fun site [4]. There is some secret code and you have to reimplement that code based on the test results generated by Pex comparing your implementation to the secret implementation. [1] http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html [2] http://research.microsoft.com/en-us/projects/pex/ http://research.microsoft.com/en-us/projects/pex/ [3] http://research.microsoft.com/en-us/projects/z3m/ http://research.microsoft.com/en-us/projects/z3m/ [4] http://www.pexforfun.com/ http://www.pexforfun.com/
- 0x0 13y ago
  The pexforfun thing seemed pretty clever: // What values of v and i can cause an exception? Ask Pex to find out! public static void Puzzle(int[] v, int i) { if (v[i + 2] + 5 == v.Length + i) throw new Exception("hidden bug!"); } and then it automatically finds the breaking input: {-6} -2 Exception hidden bug!
porges 13y ago
It's better than QuickCheck. It performs whitebox testing - that is, it has knowledge of the code and actively uses this to generate inputs that explore all possible branches.

earlz 13y ago

I tried it. Doesn't usually work all that well though, and basically only works on built in .Net types. If you pass in your own object containing only .Net types, it's not smart enough to fill those in with potential test data. Interesting idea, but I found it completely worthless

danbruc 13y ago
I can not test it at the moment but you may have to do the object creation inside a wrapper method. public static void TestWrapper(String foo, Int32 bar, Boolean buzz, Decimal foobar) { var thing = new Thing(foo, bar); thing.Buzz = buzz; Test(thing, blah); } public static Test(Thing thing, Decimal foobar) { thing.DoStuff(); if (thing.State == 42) { thing.DoOtherStuff(foobar); } } In my experience Pex yields really great results when you use it to analyze methods that are close to mathematically functions with not to many side effects and especially state mutation. Tracking changing state over time and finding sequences of operations to prepare an object to a certain state and then checking the behavior of the object in this state is much harder than analyzing a (almost) pure function.
- georgefox 13y ago
  You can test it online at pex4fun.com. Here are two tests, one using a function that takes built-in types as parameters and instantiates a Thing to operate on, and a second that takes a Thing input and operates on it directly. It seems to generate interesting cases for both, but it's much more easily readable with the more basic input types, as the parameters are easily displayed in the table. http://pex4fun.com/default.aspx?language=CSharp&code=OgPsvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee__997o7nU4n99%2f%2fP1xmZAFs9s5K2smeIYCqyB8%2ffnwfPyLWTbG8SF9fN22_OEyS1XpSFtN0WmZNk76Z03fJL05SelZ1cZm1edq0NV54VlXpL04v8vYwbfDPLwkaFcs2fZLVG1pMqqpMn6x%2f8INIG2koiDAKW9rpeVWNGPYkq_9wK8ENTzsvmjGw_gzNDsPPgctneEs_73ejHTytXrfr8%2fOtLvA6b9f1Mt26n97lcT1Od__kvweIMH5TfbVa5fXWnfSR%2fv28usLfN3b1ZTvPa_lvlk_LRVYC88jQTO%2fo_a62_UAUstZh8pJmocy3Xns0Plu29%2fZAr1H6hCYqz5bphFqN0qcbEb0kvFpMGBF7mV%2fp5DFEtCb2Mi251Zjn%2fzMG7X2nw32TN_0WtxuZ7sx4hsfCL3G3qb66GWPtS9Cxs59_Yj%2fxJqmDxC%2f5fwIAAP%2f%2f http://pex4fun.com/default.aspx?language=CSharp&code=OgP... http://pex4fun.com/default.aspx?language=CSharp&code=_QLsvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee__997o7nU4n99%2f%2fP1xmZAFs9s5K2smeIYCqyB8%2ffnwfPyLWTbG8SF9fN22_OEyS1XpSFtN0WmZNk76Z03fJL05SelZ1cZm1edq0NV54VlXpL04v8vYwbfDPLwkaFcs2fZLVG1pMqqpMn6x%2f8INIG2koiDAKW9rpeVWNGPYkq_9wK8ENTzsvmjGw_gzNDsPPgctneEs_73ejHTytXrfr8%2fOtLvA6b9f1Mt26n97lcT1Od__kvweIMH5TfbVa5fXWnfSR%2fv28usLfN3b1ZTvPa_lvlk_LRVYC88jQTO%2fo_a62_UAUstZh8pJmocy3mNJpO0qf0NTk2TKd0Oej9OlG1Noxz_Fn3NgRXTF_kzftFkHUdw06w6jwC4oI%2fr2pe_2H247t5KWf2E88GneQ_CX%2fTwAAAP%2f%2f http://pex4fun.com/default.aspx?language=CSharp&code=_QL...

Aurel1us 13y ago

Anyone know something similar for C/C++ or Python ?

monksy 13y ago
Or for Java.
- danbruc 13y ago
  https://news.ycombinator.com/item?id=5676010 https://news.ycombinator.com/item?id=5676010
- grdvnl 13y ago
  Pex is based on the Daikon dynamic invariant. This document explains some results and examples in certain languages. http://groups.csail.mit.edu/pag/daikon/download/doc/daikon.pdf http://groups.csail.mit.edu/pag/daikon/download/doc/daikon.p...
  - rainforest 13y ago
    Is it? I was under the impression it just uses symbolic execution rather than invariant detection.
    - grdvnl 13y ago
      Your are right. I mixed up dynamic symbolic execution with invariant detection. I know there is something called DySy which does do invariant detection using dynamic symbolic execution.
  - danbruc 13y ago
    Pex is based on the Daikon dynamic invariant. That's not true.
judah 13y ago
It's based on the Z3 solver, here's Z3Py: http://rise4fun.com/z3py http://rise4fun.com/z3py

Yuioup 13y ago

I've noticed here and on reddit that Microsoft stories don't get much attention. Rightly so?

suyash 13y ago

Anything similar for JavaScript?