3 ms·

Did you try looking at Solr or Elasticsearch? Inverted index (Lucene) is what they are based on. You even said "search" in your own article....

greglindahl 11y ago
Or Xapian.
thedarkproject 11y ago
Would recommend looking into ElasticSearch too. Recently had my first hands-on experiences and I am pretty amazed.
- lqdc13 11y ago
  Not sure about 2GB limit though. It doesn't work well at low RAM. Probably better to just use your own suffix tree or some trie for all the stuff you want to search for and use that. I bet you can get it down to something like 100MB.
  - loopbit 11y ago
    Depending on the index size (and type of queries, and schema, and expected performance...), 2GB might be more than enough. Solr does a very good work in caching results, query filters and so on in memory. Apart from that, it's disk based. Of course, performance won't be stellar if it has to go to disk to find every result, but even with that you should be able to get sub-second results with some tweaking.
- endymi0n 11y ago
  He didn't really define what he was going to "search" for, so yeah, when meaning full text search, Elastic is probably the way to go. When going for exact lookups, Aerospike will actually make use of his SSDs, when going for slice & dice on wide rows, BigQuery or Redshift will fit his bill - unfortunately he didn't say much about his exact use case. But inventing your own data store is probably one of the worst ideas ever except for gaining knowledge / as a rite of passage.
  - deleted 11y ago
    [deleted]
  - isoos 11y ago
    ElasticSearch (and Solr and Lucene) can be used to do index (and lookup) exact expressions, e.g. string keys/enums, or numbers. It can work with the combination of text search and exact search terms. I've had great results of using these for general purpose search and database lookups.
jgord 11y ago
.. I should give a fair and proper comparison to Lucene / Solr / Elasticsearch .. and indeed Sphinx - but haven't used these much. I guess I assumed the entry-point is a 2 to 4 node cluster ? But this may be my own wrong assumptions.. they may be really ram efficient. I confess I have a somewhat irrational dislike of what I call big-java .. I really wanted to like Cassandra+Spark for instance, but the install is just such a heavy download of code, I just feel an aversion to installing that. I have worked with Java, but not recently.. there may be incredibly efficient and small codebases that belie the big-java stereotype. Iv'e been enjoying the node.js / npm ecosystem myself - but I know smart people who love java-land. We can tolerate each others rants and still share a beer :]
- acdha 11y ago
  ElasticSearch is usually less work to install or manage than the average node app – it completely belies the Java stereotypes. The Lucene-based search engines are also very well optimized. I'd tend to think of a 30M row dataset as a single machine size unless you need failover or the records are quite large.
  - Can_Not 11y ago
    > ElasticSearch is usually less work to install or manage than the average node app – it completely belies the Java stereotypes. Wow, that is definitely the opposite of true.

greglindahl 11y ago

Or Xapian.

thedarkproject 11y ago

Would recommend looking into ElasticSearch too. Recently had my first hands-on experiences and I am pretty amazed.

lqdc13 11y ago
Not sure about 2GB limit though. It doesn't work well at low RAM. Probably better to just use your own suffix tree or some trie for all the stuff you want to search for and use that. I bet you can get it down to something like 100MB.
- loopbit 11y ago
  Depending on the index size (and type of queries, and schema, and expected performance...), 2GB might be more than enough. Solr does a very good work in caching results, query filters and so on in memory. Apart from that, it's disk based. Of course, performance won't be stellar if it has to go to disk to find every result, but even with that you should be able to get sub-second results with some tweaking.
endymi0n 11y ago
He didn't really define what he was going to "search" for, so yeah, when meaning full text search, Elastic is probably the way to go. When going for exact lookups, Aerospike will actually make use of his SSDs, when going for slice & dice on wide rows, BigQuery or Redshift will fit his bill - unfortunately he didn't say much about his exact use case. But inventing your own data store is probably one of the worst ideas ever except for gaining knowledge / as a rite of passage.
- deleted 11y ago
  [deleted]
- isoos 11y ago
  ElasticSearch (and Solr and Lucene) can be used to do index (and lookup) exact expressions, e.g. string keys/enums, or numbers. It can work with the combination of text search and exact search terms. I've had great results of using these for general purpose search and database lookups.

jgord 11y ago

.. I should give a fair and proper comparison to Lucene / Solr / Elasticsearch .. and indeed Sphinx - but haven't used these much. I guess I assumed the entry-point is a 2 to 4 node cluster ? But this may be my own wrong assumptions.. they may be really ram efficient. I confess I have a somewhat irrational dislike of what I call big-java .. I really wanted to like Cassandra+Spark for instance, but the install is just such a heavy download of code, I just feel an aversion to installing that. I have worked with Java, but not recently.. there may be incredibly efficient and small codebases that belie the big-java stereotype. Iv'e been enjoying the node.js / npm ecosystem myself - but I know smart people who love java-land. We can tolerate each others rants and still share a beer :]

acdha 11y ago
ElasticSearch is usually less work to install or manage than the average node app – it completely belies the Java stereotypes. The Lucene-based search engines are also very well optimized. I'd tend to think of a 30M row dataset as a single machine size unless you need failover or the records are quite large.
- Can_Not 11y ago
  > ElasticSearch is usually less work to install or manage than the average node app – it completely belies the Java stereotypes. Wow, that is definitely the opposite of true.