4 ms·

Author of ripgrep here. Very cool! It looks like Rak has a lot of nice features. I will say that getting this thing built took some digging. Could you please

Author of ripgrep here. Very cool! It looks like Rak has a lot of nice features. I will say that getting this thing built took some digging. Could you please add install/build instructions to the README? As it is, I had to go sleuthing through your CI setup to figure it out. On Archlinux, here's what I did: $ yay -S rakudo-bin $ export PATH="/usr/share/perl6/site/bin:$PATH" $ git clone https://github.com/lizmat/App-Rak $ cd App-Rak $ zef install . $ export PATH="$HOME/.raku/bin:$PATH" I realize features and an expressive language is your main goal, but I wanted to see what it looked like from a perf perspective: $ git remote -v origin git@github.com:rust-lang/rust (fetch) origin git@github.com:rust-lang/rust (push) $ git rev-parse HEAD 699f4a637ef382ea0716c72f60d0c1c4cd79df41 $ time rg accelerated --files-with-matches compiler/rustc_target/src/spec/thumbv7em_none_eabi.rs src/llvm-project/lldb/source/Plugins/LanguageRuntime/ObjC/AppleObjCRuntime/AppleObjCTrampolineHandler.cpp src/llvm-project/llvm/docs/PDB/HashTable.rst real 0.126 user 0.544 sys 0.850 maxmem 22 MB faults 0 $ time rak accelerated --files-with-matches compiler/rustc_target/src/spec/thumbv7em_none_eabi.rs src/llvm-project/lldb/source/Plugins/LanguageRuntime/ObjC/AppleObjCRuntime/AppleObjCTrampolineHandler.cpp src/llvm-project/llvm/docs/PDB/HashTable.rst real 17.005 user 2:07.88 sys 8.866 maxmem 1565 MB faults 0 I'm not totally clear on what kind of filtering `rak` does by default, so I also ran a search on a single plain text file: $ cd /dev/shm $ curl -LO https://burntsushi.net/stuff/opensubtitles/2018/en/sixteenth.txt.gz $ gunzip sixteenth.txt.gz $ time rg 'Sherlock Holmes' sixteenth.txt -c 426 real 0.068 user 0.047 sys 0.020 maxmem 778 MB faults 0 $ time rg 'Sherlock Holmes' sixteenth.txt -c --no-mmap 426 real 0.089 user 0.026 sys 0.063 maxmem 14 MB faults 0 $ time rak 'Sherlock Holmes' sixteenth.txt --count-only 426 matches in 1 file real 48.236 user 48.010 sys 0.334 maxmem 215 MB faults 0 $ time grep 'Sherlock Holmes' sixteenth.txt -c 426 real 0.297 user 0.233 sys 0.063 maxmem 14 MB faults 0

makeworld 3y ago
Wow, thanks for doing the perf comparison.
pizzalife 3y ago
Just wanted to say thanks for rg! I use it every day.
lizmat 3y ago
Author of App::Rak here: > I'm not totally clear on what kind of filtering `rak` does by default By default, rak search all of the files that look like text, by reading the first 2K (if I remember correctly) and using some algorithm to determine "binaryness". > I realize features and an expressive language is your main goal Indeed. > I wanted to see what it looked like from a perf perspective The performance times you've given here, coincide with my findings. Which for the "sixteenth.txt" surprises me: a bare loop in Raku reading that file line-by-line finishes in 8 seconds for me. Which would be one order of magnitude less than the search result. Still not in `rg`'s ball park, but speed wasn't a big concern for me. But 8 -> 48 seconds is a pretty big difference, so I guess it's back to the drawing board for me on that one :-)
librasteve 3y ago
@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too (sidebar - Larry took to opportunity with perl6/raku to properly reinvent regexes for unicode - there is quite lot to unpack in raku regexes: graphemes management, unicode props as char classes, named regexes, etc.) soo - I suggest this proposal to anyone looking for a cool rust/unicode project ... extend ripgrep from PCRE/ECMA262 regex 1.0 to embrace the "21st century" regex syntax of raku some initial reading is here: - https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b - https://docs.raku.org/language/regexes https://docs.raku.org/language/regexes - https://github.com/edumentab/p6-ecma262regex https://github.com/edumentab/p6-ecma262regex
- burntsushi 3y ago
  To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata. ripgrep does have fairly complete UTS#18 Level 1 support already.
  - lizmat 3y ago
    > likely need to write an entirely new regex engine Indeed. Also, it should be noted that the Raku Regex engine is not a state machine as such. The regex "slang" in Raku, is basically just another way to write code. Code that uses some state machine primitives underneath, for sure, but still code. A grammar in Raku is nothing other than a class in which the methods are codegenned using the regex syntax. But a class nonetheless, which can also have attributes and "proper" methods. This gives the Raku regex engine the flexibility needed to be able to parse Raku source code, and generate executable bytecode from that.
  - librasteve 3y ago
    forgive me for assumption that rg does PCRE(2)/ECMA, we had a bit of a chat on raku and UTS#18 here ... https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_unicode_regular_expressions_raku/ https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_un... ... raku does not attmept to implement that explicitly but (by virtue of being an HLL and having built ins such as Set operators) comes quite close to it at all levels.

makeworld 3y ago

Wow, thanks for doing the perf comparison.

pizzalife 3y ago

Just wanted to say thanks for rg! I use it every day.

lizmat 3y ago

Author of App::Rak here: > I'm not totally clear on what kind of filtering `rak` does by default By default, rak search all of the files that look like text, by reading the first 2K (if I remember correctly) and using some algorithm to determine "binaryness". > I realize features and an expressive language is your main goal Indeed. > I wanted to see what it looked like from a perf perspective The performance times you've given here, coincide with my findings. Which for the "sixteenth.txt" surprises me: a bare loop in Raku reading that file line-by-line finishes in 8 seconds for me. Which would be one order of magnitude less than the search result. Still not in `rg`'s ball park, but speed wasn't a big concern for me. But 8 -> 48 seconds is a pretty big difference, so I guess it's back to the drawing board for me on that one :-)

librasteve 3y ago

@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too (sidebar - Larry took to opportunity with perl6/raku to properly reinvent regexes for unicode - there is quite lot to unpack in raku regexes: graphemes management, unicode props as char classes, named regexes, etc.) soo - I suggest this proposal to anyone looking for a cool rust/unicode project ... extend ripgrep from PCRE/ECMA262 regex 1.0 to embrace the "21st century" regex syntax of raku some initial reading is here: - https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b - https://docs.raku.org/language/regexes https://docs.raku.org/language/regexes - https://github.com/edumentab/p6-ecma262regex https://github.com/edumentab/p6-ecma262regex

burntsushi 3y ago
To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata. ripgrep does have fairly complete UTS#18 Level 1 support already.
- lizmat 3y ago
  > likely need to write an entirely new regex engine Indeed. Also, it should be noted that the Raku Regex engine is not a state machine as such. The regex "slang" in Raku, is basically just another way to write code. Code that uses some state machine primitives underneath, for sure, but still code. A grammar in Raku is nothing other than a class in which the methods are codegenned using the regex syntax. But a class nonetheless, which can also have attributes and "proper" methods. This gives the Raku regex engine the flexibility needed to be able to parse Raku source code, and generate executable bytecode from that.
- librasteve 3y ago
  forgive me for assumption that rg does PCRE(2)/ECMA, we had a bit of a chat on raku and UTS#18 here ... https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_unicode_regular_expressions_raku/ https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_un... ... raku does not attmept to implement that explicitly but (by virtue of being an HLL and having built ins such as Set operators) comes quite close to it at all levels.