Thanks for responding! > but considering that regex handling is a common task, it would be a missed opportunity not to consider AVX-512. Just to be really cle

Thanks for responding! > but considering that regex handling is a common task, it would be a missed opportunity not to consider AVX-512. Just to be really clear here, it's less about "not considering it" and more about "not prioritizing it." I don't have any telemetry on where ripgrep is running, but I would guess that 99% of its uses are on consumer hardware. And consumer hardware is heading in a direction that doesn't include AVX-512. That lowers the priority for using it, especially when AVX-2 is available nearly everywhere. And indeed, ripgrep uses AVX-2. > While ripgrep delegates compression to other tools, I'm curious about your approach to decompression if you really needed it. I'm interested in exploring ways in which people do use ripgrep with compression. Do you use transparent disk compression? I don't really have an approach. ripgrep's decompression support isn't really about performance or improving throughput. It's just about the user experience. It's about making it easier to search stuff without having to decompress it explicitly first. You have a `foo.txt.gz` on your system? Great, ripgrep can search it using the standard tools available to you. The separate question of making that faster (`gzip -d` is usually not the fastest way to decompress gzipped data for example) is something I'd like to explore in the future. The other separate question of how to integrate compression and searching such that the holistic result unlocks interesting use cases and potentially even makes the overall search faster is not a use case that ripgrep really cares about. The important thing with ripgrep is about being able to search existing compressed data using standard tools. What I'm trying to say here is that Sneller has a much more specialized and interesting target audience than ripgrep. :-) > Can you explain how you calculated that. In our calculation we accounted for the time needed to do the decompression. I'm looking at `Regex1` in your table. ripgrep has a reported timing of 3.899s and Sneller with one thread has a reported timing of 10.024s. > Could you give some of those questions? I tried to include those scattered around in my original comment. Otherwise I don't know what was on my mind when I wrote that. It was a while ago. > That is unfortunate, because in Go `\w` is the Perl word character class `[0-9A-Za-z_]` (https://pkg.go.dev/regexp/syntax https://pkg.go.dev/regexp/syntax) and not what you use, which I assume is `\p{L}`. The state of the DFA with `\p{L}` has 12 states and 1291 edges, compared to 12 states and 132 edges with `\w`. When would you call a DFA enormous or impractical? How are you getting 12 states for \pL? DFAs with Unicode aware \w are about a couple orders of magnitude bigger than DFAs with an ASCII-only \w: $ regex-cli debug dense dfa -p '\w' -q --minimize --start-kind unanchored parse time: 28.222µs translate time: 28.317µs compile nfa time: 915.99µs compile dfa time: 12.150283ms memory: 159296 pattern len: 1 start kind: Unanchored alphabet len: 113 stride: 128 has empty?: false is utf8?: true $ regex-cli debug dense dfa -p '(?-u)\w' -q --minimize --start-kind unanchored parse time: 35.948µs translate time: 32.843µs compile nfa time: 67.048µs compile dfa time: 102.022µs memory: 384 pattern len: 1 start kind: Unanchored alphabet len: 10 stride: 16 has empty?: false is utf8?: true That's with DFA minimization. Also, '\w' has 311 states while '(?-u)\w' has 5 states. I don't have a precise definition of enormous or impractical. Does it matter? I suppose one obvious one is when DFA construction time starts having a significant impact on total search times. > Additionally, the results are not the same: the number of matches is not equal to 7882. How could I make `\w` conform to other regex implementations in ripgrep? By following UTS#18: https://unicode.org/reports/tr18/#word https://unicode.org/reports/tr18/#word Most regex engines make \w be ASCII-only by default. But most also have a way to opt into Unicode-aware mode. RE2, Go's regexp and ECMAScript are popular regex engines that have no way to change the interpretation of \w. > Fair question how regex compilers handle nefarious regexes. Go does not handle NFA with more than 1000 states, and, as you observed, we added some more restrictions when processing the NFA. It can be an interesting academic exercise to find monstrous regexes, but we haven't encountered useful regexes that hit these limits. But I guess you know some... It's definitely not academic. People use regexes for lexers. People use big regexes to recognize certain things like email addresses and dates. Here's a real regex used in real software to identify dates in unstructured text for example: https://github.com/akoumjian/datefinder/blob/5376ece0a522c44762b1ab656fc80737b427ed16/datefinder/constants.py#L112-L124 https://github.com/akoumjian/datefinder/blob/5376ece0a522c44... Otherwise, as I hinted at above, the thing that can make regexes very large very quickly is when you mix Unicode classes with counted repetitions. It doesn't take a lot to make them "big."