4 ms·

I have used regular expressions to search for code, like I guess most devs; but, apart from artificial limits such as maximum time or max number of results, I f

by j1elo 6y ago

I have used regular expressions to search for code, like I guess most devs; but, apart from artificial limits such as maximum time or max number of results, I found regex has never been a reliable search method for me. There is always the case that, when trying to find all instances of some code, it will miss that one case which is not covered by the initial intuition. For example, to find uses of the function "foo()", the first idea is usually to search for ".foo(". But that won't find places where the function is passed as value. Nor places where devs were inconsistent and wrote / didn't write a space between the name and the opening brace, like ".foo (" (this is a common style for Gnome or Glib code). Not to mention if the language is C++ and then you have to distinguish between ".foo", "->foo", and "::foo". In conclusion, regex might be OK to have a superficial look at some code, but never as a reliable source of information, unless a lot of work and ironing out of corner cases has been invested in the regex itself, effectively considering them part of the code, with unit tests and all the fuss that it entails.

indentit 6y ago
Agreed, this is why I like the idea of searching parsed files, for example using scope selectors against code tokenized with a sublime-syntax grammar. Unfortunately, it means it needs to be parsed/indexed first, which is slower than a plain regex search. I wonder if the AST built by Tree-Sitter could also help with this type of search - does anyone know of any existing solutions for this?
- masklinn 6y ago
  > does anyone know of any existing solutions for this? https://semgrep.dev https://semgrep.dev, though it's mostly an analysis tool it can be used as a search tool. IIRC it's not super fast, but for the cases where there is no way to really contort a regex into something suitable (the regex has way too many false positives and / or negatives) it works rather well.
- slimsag 6y ago
  > I like the idea of searching parsed files, for example using scope selectors against code tokenized with a sublime-syntax grammar. I've been working on a trigram-based search engine with support for exactly this (via github.com/trishume/syntect) over the past several months and plan on open-sourcing it soon. Cool to see someone else with this idea! You might also like https://comby.dev https://comby.dev - it is aware of code structure without parsing files
  - indentit 6y ago
    > I've been working on a trigram-based search engine with support for exactly this (via github.com/trishume/syntect) over the past several months and plan on open-sourcing it soon. Awesome, I look forward to that - you'll post a "show HN" for it, I hope? :) Does syntect's lack of support for the newest sublime-syntax features cause you any problems? > You might also like https://comby.dev https://comby.dev - it is aware of code structure without parsing files Thanks for sharing, will check it out!
    - slimsag 6y ago
      > Awesome, I look forward to that - you'll post a "show HN" for it, I hope? :) Does syntect's lack of support for the newest sublime-syntax features cause you any problems? Show HN -> Yep :) Lack of support for newest sublime-syntax features: less than you would expect, but a little for sure. Most of the syntax definitions in the wild today don't really use the newer features, at Sourcegraph I wrote a little Rust HTTP server wrapping syntect[1] and we use it for all our syntax highlighting for the past several years, I would say it works on like 95% of code files even if you include a lot of additional syntaxes that are untested with Syntect[2]. That said, it does barf hard on some specific files - either taking a really long time to do the work or getting completely stuck in a busy waiting loop for some reason. That said, it's still the 2nd best syntax highlighter out there (second only to Sublime itself.) One of my hopes for this side project is that I'll be able to contribute more time upstream with e.g. a more extensive test suite for Syntect against a much larger number of syntax definitions from the wild instead of just Sublime's built-in ones. [1] https://github.com/sourcegraph/syntect_server https://github.com/sourcegraph/syntect_server [2] https://github.com/slimsag/Packages/#license https://github.com/slimsag/Packages/#license
porpoise 6y ago
In some cases the problem can be somewhat mitigated if the search is incremental (and fast), as it basically allows you to refine your regex in real time and help train the correct intuition. For example, you can type out "foo" at first which will, say, display 7 results, then when you add "()" you see the list shrunk by one, so you immediately know the regex missed one it's supposed to capture.

by j1elo 6y ago

indentit 6y ago

Agreed, this is why I like the idea of searching parsed files, for example using scope selectors against code tokenized with a sublime-syntax grammar. Unfortunately, it means it needs to be parsed/indexed first, which is slower than a plain regex search. I wonder if the AST built by Tree-Sitter could also help with this type of search - does anyone know of any existing solutions for this?

masklinn 6y ago
> does anyone know of any existing solutions for this? https://semgrep.dev https://semgrep.dev, though it's mostly an analysis tool it can be used as a search tool. IIRC it's not super fast, but for the cases where there is no way to really contort a regex into something suitable (the regex has way too many false positives and / or negatives) it works rather well.
slimsag 6y ago
> I like the idea of searching parsed files, for example using scope selectors against code tokenized with a sublime-syntax grammar. I've been working on a trigram-based search engine with support for exactly this (via github.com/trishume/syntect) over the past several months and plan on open-sourcing it soon. Cool to see someone else with this idea! You might also like https://comby.dev https://comby.dev - it is aware of code structure without parsing files
- indentit 6y ago
  > I've been working on a trigram-based search engine with support for exactly this (via github.com/trishume/syntect) over the past several months and plan on open-sourcing it soon. Awesome, I look forward to that - you'll post a "show HN" for it, I hope? :) Does syntect's lack of support for the newest sublime-syntax features cause you any problems? > You might also like https://comby.dev https://comby.dev - it is aware of code structure without parsing files Thanks for sharing, will check it out!
  - slimsag 6y ago
    > Awesome, I look forward to that - you'll post a "show HN" for it, I hope? :) Does syntect's lack of support for the newest sublime-syntax features cause you any problems? Show HN -> Yep :) Lack of support for newest sublime-syntax features: less than you would expect, but a little for sure. Most of the syntax definitions in the wild today don't really use the newer features, at Sourcegraph I wrote a little Rust HTTP server wrapping syntect[1] and we use it for all our syntax highlighting for the past several years, I would say it works on like 95% of code files even if you include a lot of additional syntaxes that are untested with Syntect[2]. That said, it does barf hard on some specific files - either taking a really long time to do the work or getting completely stuck in a busy waiting loop for some reason. That said, it's still the 2nd best syntax highlighter out there (second only to Sublime itself.) One of my hopes for this side project is that I'll be able to contribute more time upstream with e.g. a more extensive test suite for Syntect against a much larger number of syntax definitions from the wild instead of just Sublime's built-in ones. [1] https://github.com/sourcegraph/syntect_server https://github.com/sourcegraph/syntect_server [2] https://github.com/slimsag/Packages/#license https://github.com/slimsag/Packages/#license

porpoise 6y ago

In some cases the problem can be somewhat mitigated if the search is incremental (and fast), as it basically allows you to refine your regex in real time and help train the correct intuition. For example, you can type out "foo" at first which will, say, display 7 results, then when you add "()" you see the list shrunk by one, so you immediately know the regex missed one it's supposed to capture.