4 ms·

I'm getting the impression that recreating functionality from Lucene on top of RDBMS abstractions is certainly possible, but requires a lot of effort from devs

I'm getting the impression that recreating functionality from Lucene on top of RDBMS abstractions is certainly possible, but requires a lot of effort from devs and you still need to understand stuff like tokenization, stemming, inverted indexes etc. if you want to use it. At this point one may as well just learn Solr or Elasticsearch. And these are still better at searching at the low level, although of course you sacrifice consistency guarantees. This is a legitimately different problem arena, not NoSQL for the sake of it.

mistrial9 7y ago
> recreating functionality from Lucene on top of RDBMS strange mix of assumptions there, since searching text strings is Job One of computer search since 'forever' .. if Lucene has done some (search) things well, there is no reason that others do not have a chance to also do those things well. In fact, encourage it, foster it, engage with the process that brings new software capacities ..
- stereolambda 7y ago
  Sure, I was meaning all the structures and algorithmic stuff we do to limit as much as possible straight up scanning strings. I came to appreciate much the work accumulated, layer by layer, in Lucene. Of course you can redo it elsewhere and it's a worthy programming enterprise but often not directly related to what you want to accomplish with your application.
galacticdessert 7y ago
I see your point, but I also see why RDBMS are appealing. We just built a documentation portal for our customers, and we initially used ElasticSearch for as-you-type full text search. I ended up redoing the implementation with SQLite (using fts5) and a simple express api later on top of it. The resource consumption is 3 orders of magnitude lower, the speed is the same and the quality of the results comparable. Elastic was crashing on us due to excessive heap memory consumption, even if 2gb of ram was assigned to it and we had two nodes. I thought the resource consumption was crazy for a light search use case, and found out that SQLite can do the same with 20 mbs of ram.

mistrial9 7y ago

> recreating functionality from Lucene on top of RDBMS strange mix of assumptions there, since searching text strings is Job One of computer search since 'forever' .. if Lucene has done some (search) things well, there is no reason that others do not have a chance to also do those things well. In fact, encourage it, foster it, engage with the process that brings new software capacities ..

stereolambda 7y ago
Sure, I was meaning all the structures and algorithmic stuff we do to limit as much as possible straight up scanning strings. I came to appreciate much the work accumulated, layer by layer, in Lucene. Of course you can redo it elsewhere and it's a worthy programming enterprise but often not directly related to what you want to accomplish with your application.

galacticdessert 7y ago

I see your point, but I also see why RDBMS are appealing. We just built a documentation portal for our customers, and we initially used ElasticSearch for as-you-type full text search. I ended up redoing the implementation with SQLite (using fts5) and a simple express api later on top of it. The resource consumption is 3 orders of magnitude lower, the speed is the same and the quality of the results comparable. Elastic was crashing on us due to excessive heap memory consumption, even if 2gb of ram was assigned to it and we had two nodes. I thought the resource consumption was crazy for a light search use case, and found out that SQLite can do the same with 20 mbs of ram.