3 ms·

I've written this code (in C++) for an employer. RE2 scaled fine to hundreds of thousands of regexes. You'll want to use RE2::Set, which compiles multiple rege

I've written this code (in C++) for an employer. RE2 scaled fine to hundreds of thousands of regexes. You'll want to use RE2::Set, which compiles multiple regexes into a single DFA, and probably the "Filter" functionality (whose name I don't precisely remember and am too lazy to look up) which uses an Aho-Corasick tree to subset the potential matches. One thing you'll have to watch out for is RE2's maximum DFA size; if compilation of your RE2::Set fails, just split your set of regexes in half and compile again. You could probably do some fun optimizations by grouping the regexes which depend on the same literals into their own sets, but I never needed to.

burntsushi 5y ago
This is basically what ripgrep will do for you automatically. (ripgrep uses Rust's regex engine, which is a descendant of RE2.) But when you get up into hundreds of thousands of regexes, the NFA (and the resulting DFA) get really big. And things generally don't scale that well. Here's a good example: http://web.archive.org/web/20210302010420/https://01.org/hyperscan/blogs/jpviiret/2017/regex-set-scanning-hyperscan-and-re2set http://web.archive.org/web/20210302010420/https://01.org/hyp... The problem is that for a big enough NFA, you'll wind up spending most of your search doing powerset construction to build the DFA. > One thing you'll have to watch out for is RE2's maximum DFA size You can configure this in ripgrep with the --dfa-size-limit flag. (See also --regex-size-limit.)