3 ms·
Show HN: Language agnostic full-text search using lempel-ziv factoring
- graderjs 6y agoThis is not ready for anything yet i think. It's still in the research phase. Why? I want to support any language, even code, without configuration. LZ is used as a way to stem and tokenize because it finds repeat factors, which we assume are useful. So far, i feel the hypothesis that this works is supported, and I'm working to get both recall and precision between .7 to .8, currently they are at, 0.65 both after a lot of work. Three way i evaluate it is not perfect, but i feel it's an approximation that's useful enough. My ultimate use case is to bolt it in to my web archiving project 22120, to give people everywhere the full text search they've been asking for. I think i can make it good enough. I think if you come expecting some amazing results, you will probably be disappointed. But i like the ones and think it's clever to use LZ as a language agnostic way to get tokens to index on. Apart from that, everything else is at a very basic level, and there is no support for any type of sophisticated queries right now, such as, Boolean, exact phrase, facet and so on, but some of these should be possible. I've done half a blog post draft about this, and one final thing is, the most interesting part of this so far for me had been thinking about how to do snippets to show in search results efficiently.