3 ms·

What regex engines do people use for this task generally? Most regex engines I'm aware of don't support stream/incremental searching.

zogrodea 3y ago
This comment by someone at GitHub mentioned that Atom used PCRE’s partial match functionality, but I have no idea about what is used most commonly. https://news.ycombinator.com/item?id=15386155 https://news.ycombinator.com/item?id=15386155
vlovich123 3y ago
I was just going from the linked issue where I thought I read that Go’s engine supports incremental search. Maybe I misread? While I have you, how close is the Rust incremental search support? It sounded like regex-automata might make it possible but hard to easily get a sense of high level progress from a GitHub issue.
- burntsushi 3y ago
  Go has this: https://pkg.go.dev/regexp#Regexp.MatchReader https://pkg.go.dev/regexp#Regexp.MatchReader --- That just tells you whether an io.Reader contains a match anywhere or not. I don't think it tells you anything else, like the position of the match. > While I have you, how close is the Rust incremental search support? It sounded like regex-automata might make it possible but hard to easily get a sense of high level progress from a GitHub issue. I'm not working on it. The current status is that other people are working on trying to write it themselves on top of regex-automata in a way that works for their specific use case. That was my high level strategy: release a separately versioned library with the regex internals[1] exposed so that others can experiment and build on top of it. The regex-automata crate exposes the low level DFA (and lazy DFA) transition function. So you can pretty much do some kind of stream searching out of the box today (certainly at least what Go provides): https://docs.rs/regex-automata/latest/regex_automata/#build-a-full-dfa-and-walk-it-manually https://docs.rs/regex-automata/latest/regex_automata/#build-... Bottom line here is that if you're looking to implement stream searching in Rust, then you should be able to go out and build something to do it today with regex-automata. But you aren't going to get a streamlined experience. (And whether you ever will or not remains unclear to me.) [1]: https://blog.burntsushi.net/regex-internals/ https://blog.burntsushi.net/regex-internals/
  - LukeShu 3y ago
    > Go ... I don't think it tells you anything else, like the position of the match. The regexp.Regexp.FindReaderIndex and .FindReaderSubmatchIndex methods tell you the position of the match.
    - burntsushi 3y ago
      Despite having studied Go's regex engine in depth, I have somehow missed those methods. To the point that when you responded, I assumed they were added recently. I checked to be sure, and indeed not. They've been there since Go 1.0. Thanks for the correction!

zogrodea 3y ago

This comment by someone at GitHub mentioned that Atom used PCRE’s partial match functionality, but I have no idea about what is used most commonly. https://news.ycombinator.com/item?id=15386155 https://news.ycombinator.com/item?id=15386155

vlovich123 3y ago

I was just going from the linked issue where I thought I read that Go’s engine supports incremental search. Maybe I misread? While I have you, how close is the Rust incremental search support? It sounded like regex-automata might make it possible but hard to easily get a sense of high level progress from a GitHub issue.

burntsushi 3y ago
Go has this: https://pkg.go.dev/regexp#Regexp.MatchReader https://pkg.go.dev/regexp#Regexp.MatchReader --- That just tells you whether an io.Reader contains a match anywhere or not. I don't think it tells you anything else, like the position of the match. > While I have you, how close is the Rust incremental search support? It sounded like regex-automata might make it possible but hard to easily get a sense of high level progress from a GitHub issue. I'm not working on it. The current status is that other people are working on trying to write it themselves on top of regex-automata in a way that works for their specific use case. That was my high level strategy: release a separately versioned library with the regex internals[1] exposed so that others can experiment and build on top of it. The regex-automata crate exposes the low level DFA (and lazy DFA) transition function. So you can pretty much do some kind of stream searching out of the box today (certainly at least what Go provides): https://docs.rs/regex-automata/latest/regex_automata/#build-a-full-dfa-and-walk-it-manually https://docs.rs/regex-automata/latest/regex_automata/#build-... Bottom line here is that if you're looking to implement stream searching in Rust, then you should be able to go out and build something to do it today with regex-automata. But you aren't going to get a streamlined experience. (And whether you ever will or not remains unclear to me.) [1]: https://blog.burntsushi.net/regex-internals/ https://blog.burntsushi.net/regex-internals/
- LukeShu 3y ago
  > Go ... I don't think it tells you anything else, like the position of the match. The regexp.Regexp.FindReaderIndex and .FindReaderSubmatchIndex methods tell you the position of the match.
  - burntsushi 3y ago
    Despite having studied Go's regex engine in depth, I have somehow missed those methods. To the point that when you responded, I assumed they were added recently. I checked to be sure, and indeed not. They've been there since Go 1.0. Thanks for the correction!