4 ms·

Postgres, elasticsearch and SOLR all seem pretty heavyweight to me. What's the lightweight approach to create an inverted index nowadays with some basic stemmi

by hendry 12y ago

Postgres, elasticsearch and SOLR all seem pretty heavyweight to me. What's the lightweight approach to create an inverted index nowadays with some basic stemming?

latch 12y ago
redis sets or sorted sets with a npl library (natural for node, treat for ruby...). I'm sure there are packages that do the storage and retrieval for you, but it's the sort of lightweight thing you can do quickly and tweak from there.
mborch 12y ago
http://xapian.org/ http://xapian.org/
rjbwork 12y ago
You can always just roll your own with lucene or lucene.NET, which is what ES/SOLR use underneath.
- collyw 12y ago
  I never use Lucene, but I got the impression fro others that it was difficult to use, and that Elasticsearch gives it a relatively usable layer on top.
  - rpedela 12y ago
    It isn't that hard to use in my opinion, but it is only a library. So you need to implement many things to turn it into a server which Solr and Elasticsearch have already done.
mrweasel 12y ago
It might fall into "heavyweight" for you, but Sphinx Search is pretty easy to setup and get started with. For Go there's also Bleve (http://www.blevesearch.com/ http://www.blevesearch.com/), but I haven't tried it yet.
frik 12y ago
* Sphinx search (free, open source): http://sphinxsearch.com http://sphinxsearch.com , http://en.wikipedia.org/wiki/Sphinx_(search_engine) http://en.wikipedia.org/wiki/Sphinx_(search_engine) It's the lightweight & fast alternative, similar to what Nginx is to Apache and IIS. * SQLite FTS 4 (free, open source): http://www.sqlite.org/fts3.html http://www.sqlite.org/fts3.html Lightweight & embedded solution (suitable for small to medium sized database and single process).
framos 12y ago
For Python it doesn't get easier than Whoosh (https://pypi.python.org/pypi/Whoosh https://pypi.python.org/pypi/Whoosh). Both indexing and searching are done in Python, no need for external services or processes. Features are limited and performance won't be the best, but sometimes ease of use and maintenance are all you need.
lobster_johnson 12y ago
Xapian is excellent. It's an embedded C++ library, and there are bindings for most languages. It's well-designed but fairly basic. Since it's a library, you'll have to write the server part yourself. (It has a server called Omega, but I've never used it.) Indexing is slower than Lucene, though, but it's not too bad. The only significant problem I found with it the last time I was using it (which was 5+ years ago) was that writers occasionally would cause queries to abort, and you had to restart them, a cycle which could lead to long query times if you had lots of write churn.
leo_santagada 12y ago
https://pythonhosted.org/Whoosh/intro.html https://pythonhosted.org/Whoosh/intro.html

by hendry 12y ago

Postgres, elasticsearch and SOLR all seem pretty heavyweight to me. What's the lightweight approach to create an inverted index nowadays with some basic stemming?

latch 12y ago

redis sets or sorted sets with a npl library (natural for node, treat for ruby...). I'm sure there are packages that do the storage and retrieval for you, but it's the sort of lightweight thing you can do quickly and tweak from there.

mborch 12y ago

http://xapian.org/ http://xapian.org/

rjbwork 12y ago

You can always just roll your own with lucene or lucene.NET, which is what ES/SOLR use underneath.

collyw 12y ago
I never use Lucene, but I got the impression fro others that it was difficult to use, and that Elasticsearch gives it a relatively usable layer on top.
- rpedela 12y ago
  It isn't that hard to use in my opinion, but it is only a library. So you need to implement many things to turn it into a server which Solr and Elasticsearch have already done.

mrweasel 12y ago

It might fall into "heavyweight" for you, but Sphinx Search is pretty easy to setup and get started with. For Go there's also Bleve (http://www.blevesearch.com/ http://www.blevesearch.com/), but I haven't tried it yet.

frik 12y ago

* Sphinx search (free, open source): http://sphinxsearch.com http://sphinxsearch.com , http://en.wikipedia.org/wiki/Sphinx_(search_engine) http://en.wikipedia.org/wiki/Sphinx_(search_engine) It's the lightweight & fast alternative, similar to what Nginx is to Apache and IIS. * SQLite FTS 4 (free, open source): http://www.sqlite.org/fts3.html http://www.sqlite.org/fts3.html Lightweight & embedded solution (suitable for small to medium sized database and single process).

framos 12y ago

For Python it doesn't get easier than Whoosh (https://pypi.python.org/pypi/Whoosh https://pypi.python.org/pypi/Whoosh). Both indexing and searching are done in Python, no need for external services or processes. Features are limited and performance won't be the best, but sometimes ease of use and maintenance are all you need.

lobster_johnson 12y ago

Xapian is excellent. It's an embedded C++ library, and there are bindings for most languages. It's well-designed but fairly basic. Since it's a library, you'll have to write the server part yourself. (It has a server called Omega, but I've never used it.) Indexing is slower than Lucene, though, but it's not too bad. The only significant problem I found with it the last time I was using it (which was 5+ years ago) was that writers occasionally would cause queries to abort, and you had to restart them, a cycle which could lead to long query times if you had lots of write churn.

leo_santagada 12y ago

https://pythonhosted.org/Whoosh/intro.html https://pythonhosted.org/Whoosh/intro.html