4 ms·

We recently chose a Solr/Lucene solution for our game search at Big Fish Games (launches tomorrow!). I cannot imagine writing from scratch many features that c

We recently chose a Solr/Lucene solution for our game search at Big Fish Games (launches tomorrow!). I cannot imagine writing from scratch many features that come with Solr/Lucene: spelling correction, word stemming (walk == walking), stop words (don't return every hit for "the"), sane defaults for tokenizing (splitting up sentences into indexable and searchable chunks, like words), and uhh soon Fuzzy Query matching. It's certainly easy to use a very limited subset of Lucene's capabilities to come up with a very intuitive user-searchable index of data. If you wrote something from scratch that truly works better than an out of the box Solr server, let's just say I'd be surprised.

stonemetal 15y ago
If you can't imagine writing a spelling corrector from scratch you might want to take a minute and broaden your horizons(I know it was magic pixie dust for me before I read it). http://norvig.com/spell-correct.html http://norvig.com/spell-correct.html Stemming could be accomplished just as easily, put in a word get back a list of stems. Just a dictionary look up that could be precomputed.
- awj 15y ago
  I thought his comment was more along the lines of "I cannot imagine writing industrial-strength versions of all of these things Lucene gives me." Yes, you can write a spellchecker in 21 lines of code, that doesn't necessarily mean it will be fast enough to be a component in website search or that it will be any kind of a pleasure to query or maintain the word corpus. I can put together toy versions of many things Lucene provides pretty easily in my own time. Building useful, dependable versions of most anything takes a nontrivial amount of time and effort, so it's smart to restrict my usage of toys to understanding the concepts.
timr 15y ago
I don't want to sound like a troll, but Lucene's built-in spellchecker is a toy. If you're doing much more than trying to just plonk in a search engine on your website, you'll probably find it insufficient. As far as stemming goes, AFAIK, Lucene didn't invent anything. It's a lot like crypto, in that nearly everyone uses public implementations of of stemmers for different languages: http://snowball.tartarus.org/ http://snowball.tartarus.org/ The previous two points both apply to stop words (you'll almost certainly want to customize your own list, and there are good, curated lists out there for the most common languages), and Lucene's tokenizer options aren't really that smart -- if you want to get into chunking, phrase identification, etc., you're going to have to write it yourself. But then you're going to be working within the Lucene world, tied to their index format, mucking around in the depths of unfamiliar code, etc. That's a big trade-off. Basically, Lucene is great if you're working on a website and you need a drop-in solution for search, but if your core user experience is search, it has a lot of deficiencies.

stonemetal 15y ago

If you can't imagine writing a spelling corrector from scratch you might want to take a minute and broaden your horizons(I know it was magic pixie dust for me before I read it). http://norvig.com/spell-correct.html http://norvig.com/spell-correct.html Stemming could be accomplished just as easily, put in a word get back a list of stems. Just a dictionary look up that could be precomputed.

awj 15y ago
I thought his comment was more along the lines of "I cannot imagine writing industrial-strength versions of all of these things Lucene gives me." Yes, you can write a spellchecker in 21 lines of code, that doesn't necessarily mean it will be fast enough to be a component in website search or that it will be any kind of a pleasure to query or maintain the word corpus. I can put together toy versions of many things Lucene provides pretty easily in my own time. Building useful, dependable versions of most anything takes a nontrivial amount of time and effort, so it's smart to restrict my usage of toys to understanding the concepts.

timr 15y ago

I don't want to sound like a troll, but Lucene's built-in spellchecker is a toy. If you're doing much more than trying to just plonk in a search engine on your website, you'll probably find it insufficient. As far as stemming goes, AFAIK, Lucene didn't invent anything. It's a lot like crypto, in that nearly everyone uses public implementations of of stemmers for different languages: http://snowball.tartarus.org/ http://snowball.tartarus.org/ The previous two points both apply to stop words (you'll almost certainly want to customize your own list, and there are good, curated lists out there for the most common languages), and Lucene's tokenizer options aren't really that smart -- if you want to get into chunking, phrase identification, etc., you're going to have to write it yourself. But then you're going to be working within the Lucene world, tied to their index format, mucking around in the depths of unfamiliar code, etc. That's a big trade-off. Basically, Lucene is great if you're working on a website and you need a drop-in solution for search, but if your core user experience is search, it has a lot of deficiencies.