4 ms·

At which point do we get a search that syntactically understands what its searching? Thats such an obvious improvement when you know that your entire data set c

At which point do we get a search that syntactically understands what its searching? Thats such an obvious improvement when you know that your entire data set conforms to a language that an 80s C compiler can understand. A simple grep I can do myself (with the added bonus of knowing what version/commit I'm searching against).

akkartik 14y ago
How would syntactic awareness change search results? Would we have to change the UI as well, move away from 'type in a string'?
- revelation 14y ago
  You could weigh search results. Instead of a random comment in a random device driver as to why we use mmap here, you could directly show me the core mmap implementation when I search for it. You can directly link me to functions, you can show me the definition for a struct object inline, you can show me dependency graphs.
  - akkartik 14y ago
    Ok yeah, that's more of a code browser than just search.
  - boyter 14y ago
    Would you have an example of a tool which you could use to inspect source code to give this output? I imagine a general lexer or some description would be great for this as you could identify functions, comments, classes etc... and use that to add weights.
    - Someone 14y ago
      https://wiki.mozilla.org/DXR https://wiki.mozilla.org/DXR would be a starting point.
- bazzargh 14y ago
  There's the problem of regular expressions not dealing with matching parentheses, for one. That makes a lot of questions about code hard to express in a regexp, even something as simple as 'find foo(anything) + bar(anything)' is busted. False positives within strings are also an issue. I did a code search thing, (which I don't claim is anything more than a hack) described over in this HN thread: http://news.ycombinator.com/item?id=4113348 http://news.ycombinator.com/item?id=4113348 but yes, the syntax wasn't quite 'a string', you'd type the code you're looking for with parts wildcarded, but parentheses in what you typed had to match.
  - akkartik 14y ago
    That's a good idea. Another use case occurred to me after I wrote my comment: to ignore keywords in comments. Sometimes that can be desirable. I keep going back lately to Rob Pike's idea that breaking files into lines is not the best approach (http://news.ycombinator.com/item?id=4111667 http://news.ycombinator.com/item?id=4111667). In that paper he proposes a regular expression for breaking up lines into structured content. But maybe there's another way. Like an ometa grammar? (http://tinlizzie.org/ometa http://tinlizzie.org/ometa) Anyways, just a few random thoughts.
    - johnm 14y ago
      Regular expressions are the wrong tool for this job. For Krugle, we actually parsed the various languages so it actually knew the differences between e.g. definitions, invocations, comments, strings, etc. It affords the capabilities to do all sorts of cool queries and analyses.
johnm 14y ago
We did that with Krugle awhile ago. It's still alive as an enterprise product... http://krugle.com/ http://krugle.com/
eklitzke 14y ago
FWIW I am the author of this tool, and I would like to add something that understands syntax (I think I could start with indexing the code using ctags or cscope, which is hackish but easy to implement). There's obviously a ton of stuff you could do to improve ranking if you could rank results by whether they're a variable, function, etc.

akkartik 14y ago

How would syntactic awareness change search results? Would we have to change the UI as well, move away from 'type in a string'?

revelation 14y ago
You could weigh search results. Instead of a random comment in a random device driver as to why we use mmap here, you could directly show me the core mmap implementation when I search for it. You can directly link me to functions, you can show me the definition for a struct object inline, you can show me dependency graphs.
- akkartik 14y ago
  Ok yeah, that's more of a code browser than just search.
- boyter 14y ago
  Would you have an example of a tool which you could use to inspect source code to give this output? I imagine a general lexer or some description would be great for this as you could identify functions, comments, classes etc... and use that to add weights.
  - Someone 14y ago
    https://wiki.mozilla.org/DXR https://wiki.mozilla.org/DXR would be a starting point.
bazzargh 14y ago
There's the problem of regular expressions not dealing with matching parentheses, for one. That makes a lot of questions about code hard to express in a regexp, even something as simple as 'find foo(anything) + bar(anything)' is busted. False positives within strings are also an issue. I did a code search thing, (which I don't claim is anything more than a hack) described over in this HN thread: http://news.ycombinator.com/item?id=4113348 http://news.ycombinator.com/item?id=4113348 but yes, the syntax wasn't quite 'a string', you'd type the code you're looking for with parts wildcarded, but parentheses in what you typed had to match.
- akkartik 14y ago
  That's a good idea. Another use case occurred to me after I wrote my comment: to ignore keywords in comments. Sometimes that can be desirable. I keep going back lately to Rob Pike's idea that breaking files into lines is not the best approach (http://news.ycombinator.com/item?id=4111667 http://news.ycombinator.com/item?id=4111667). In that paper he proposes a regular expression for breaking up lines into structured content. But maybe there's another way. Like an ometa grammar? (http://tinlizzie.org/ometa http://tinlizzie.org/ometa) Anyways, just a few random thoughts.
  - johnm 14y ago
    Regular expressions are the wrong tool for this job. For Krugle, we actually parsed the various languages so it actually knew the differences between e.g. definitions, invocations, comments, strings, etc. It affords the capabilities to do all sorts of cool queries and analyses.

johnm 14y ago

We did that with Krugle awhile ago. It's still alive as an enterprise product... http://krugle.com/ http://krugle.com/

eklitzke 14y ago

FWIW I am the author of this tool, and I would like to add something that understands syntax (I think I could start with indexing the code using ctags or cscope, which is hackish but easy to implement). There's obviously a ton of stuff you could do to improve ranking if you could rank results by whether they're a variable, function, etc.