3 ms·
One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the a
by dahart 7y ago
One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want.
6.3 Grep considered harmful
Simple analysis techniques may be too blunt to provide useful answers. This problem was compounded by the fact that the search for keywords did not look for words and instead captured substrings wholly unrelated to software defects. When the accuracy of classification is as low as 36%, it becomes difficult to argue that results with small effect sizes are meaningful as they may be indistinguishable from noise. If such classification techniques are to be employed, then a careful post hoc validation by hand should be conducted by domain experts.
- stcredzero 7y agoOne of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want. Smalltalk was wonderful in this regard. If you wanted to search for how something was used, you just right-click search for "senders." If you wanted to find implementors, the same. This big difference occurred with chains of senders. In other languages, where there's 2 or 3 ways to get to something or call something, you have this O(2^n) or O(2.5^n) blowup in the work you have to do to properly follow chains of senders. With Smalltalk, it's just O(n). On top of that, there were tools that could let you compose really sophisticated SQL like queries of the code base, dynamically, then pop that up in a browser.