4 ms·
Show HN: ngrep – grep plus word embeddings (Rust)
I got curious about a simple question: regular expressions are purely syntactic, but what happens if you add just a little bit of semantics?
To answer, I ended up building ngrep: a grep-like tool that extends regular expressions with a new operator ~(token) that matches a word by meaning using word2vec-style embeddings (FastText, GloVe, Wikipedia2Vec).
A simple demo: "~(big)+ \b~(animal;0.35)+\b" over Moby-Dick can find many ways used to refer to a large animal, surfacing "great whale", "enormous creature", "huge elephant" and so on. Pipe it through sort | uniq -c and the winner is, unsurprisingly, "great whale" :)
Built in Rust on top of the awesome fancy-regex, and ~() composes with all standard operators (negative lookahead, quantifiers, etc.). Currently a PoC with many missing optimizations (e.g: no caching, no compilation to standard regex, etc.), obviously without the guarantees of plain regex and subject to the limits of w2v-style embeddings...but thought it was worth sharing!
- shablulman 7mo ago[dead]
- jruohonen 7mo agoJust a remark about the name; there is already a well-known ngrep. https://linux.die.net/man/8/ngrep https://linux.die.net/man/8/ngrep.
- xnan 7mo agoOops, didn't know, thanks! I just added a section on how to compile and try it without a system installation to avoid any name clash :)
- spranab 7mo ago[dead]