6 ms·

Neat! I hope this goes far, it'd be great to have a faster/lighterweight Elastcsearch. Something similar I'm really hoping to see is Tantivy in a Postgres exte

by nathcd 8y ago

Neat! I hope this goes far, it'd be great to have a faster/lighterweight Elastcsearch. Something similar I'm really hoping to see is Tantivy in a Postgres extension, so I can stop playing the game of trying to keep my search engine and database in sync. Seeing pg-extend-rs (https://github.com/bluejekyll/pg-extend-rs https://github.com/bluejekyll/pg-extend-rs) on HN the other week got me thinking about it again. Does anyone know whether this is feasible or if anyone is working on something in this vein?

aldoushuxley001 8y ago
I know you can go pretty far just within postgresql these days without any extensions, even for production grade search. For my projects thats all I've been doing, in order to avoid the overhead of ensuring ES or Solr are synced with db.
- GordonS 8y ago
  Just released a system that does the same - Postgres FTS can get you a long way. I've used Elasticsearch before, and it's a lot of complexity to add, and isn't always necessary.
profquail 8y ago
Out of curiosity —- have you looked at using Postgresql’s full text search functionality to implement your search engine (e.g. [1])? If so, what do you get out of the combination of Postgres + Elasticsearch that you chose it over just the Postgres full text search? [1] http://rachbelaid.com/postgres-full-text-search-is-good-enough/ http://rachbelaid.com/postgres-full-text-search-is-good-enou...
- Ralfp 8y ago
  Major problem with Postgres full-text search that those articles don't dwell into too much is that unless your documents are in one of the "chosen languages", you are more likely to find support for your language in search engine (like ElasticSearch) than get it on PostgreSQL. You can convert existing dictionaries available to format Postgres understand, but this is annoying pain point if you happen to be an open source project like CMS or communication platform.
  - hashhar 8y ago
    I don't get the hype about elasticsearch at all. Elasticsearch is more suited to searching logs. It doesn't have powerful sort functions, doesn't allow you to use multiple sort parameters etc. Apache Solr is more suited to search. Lots of document filters, query filters, the index itself is highly configurable and the ability to sort on multiple parameters is great. LTR is also something too good to miss out on.
    - rpedela 8y ago
      Can you explain what you mean by "multiple sort parameters" because it looks like you can to me in ES [1]. There is a well maintained LTR plugin for ES. Honestly Solr and ES are more similar than different. There are a few things Solr has which ES doesn't and the reverse is true too. 1. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
      - hashhar 8y ago
        Thanks for letting me know about Elasticsearch LTR.
    - bryanrasmussen 8y ago
      Both solr and elasticsearch use lucene to index.
      - hashhar 8y ago
        ES doesn't expose as many knobs last I checked.
    - k9s9 8y ago
      The difference is in the size of your Sales team. Lot of open source projects would benefit, if they had dedicated sales teams contacting firms day in, day out talking about their features. People who run IT departments in the Enterprise world or even small firms lacking resources to keep up, just pick tools and make software decisions based on who reaches out to them. Invest in a sales team and you penetrate markets that don't spend time monitoring developments in the tech world which is really the majority of all orgs. Elastic has done that quite well and is reaping the rewards.
- burntsushi 8y ago
  Last time I checked, PostgreSQL doesn't support any kind of tfidf ranking scheme because it doesn't track corpus frequencies of terms in the index. This impacts how well relevance ranking works for some workloads.
lykr0n 8y ago
ZoomDB (https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb) offers something like this.
- 33degrees 8y ago
  Didn’t know about this project, very cool.
  - zombodb 8y ago
    Thanks! It’s been fun to develop it and we love when people discover and use it
FridgeSeal 8y ago
There’s a super neat plugin for Postgres called “ZomboDB” that hooks up ElasticSearch to Postgres, so when you make full text queries, it gets performed on ES. https://www.zombodb.com/ https://www.zombodb.com/ I had a brief play with it for a project at work and it was super straightforward to get running.
jedisct1 8y ago
PGroonga https://pgroonga.github.io/ https://pgroonga.github.io/

by nathcd 8y ago

aldoushuxley001 8y ago

I know you can go pretty far just within postgresql these days without any extensions, even for production grade search. For my projects thats all I've been doing, in order to avoid the overhead of ensuring ES or Solr are synced with db.

GordonS 8y ago
Just released a system that does the same - Postgres FTS can get you a long way. I've used Elasticsearch before, and it's a lot of complexity to add, and isn't always necessary.

profquail 8y ago

Out of curiosity —- have you looked at using Postgresql’s full text search functionality to implement your search engine (e.g. [1])? If so, what do you get out of the combination of Postgres + Elasticsearch that you chose it over just the Postgres full text search? [1] http://rachbelaid.com/postgres-full-text-search-is-good-enough/ http://rachbelaid.com/postgres-full-text-search-is-good-enou...

Ralfp 8y ago
Major problem with Postgres full-text search that those articles don't dwell into too much is that unless your documents are in one of the "chosen languages", you are more likely to find support for your language in search engine (like ElasticSearch) than get it on PostgreSQL. You can convert existing dictionaries available to format Postgres understand, but this is annoying pain point if you happen to be an open source project like CMS or communication platform.
- hashhar 8y ago
  I don't get the hype about elasticsearch at all. Elasticsearch is more suited to searching logs. It doesn't have powerful sort functions, doesn't allow you to use multiple sort parameters etc. Apache Solr is more suited to search. Lots of document filters, query filters, the index itself is highly configurable and the ability to sort on multiple parameters is great. LTR is also something too good to miss out on.
  - rpedela 8y ago
    Can you explain what you mean by "multiple sort parameters" because it looks like you can to me in ES [1]. There is a well maintained LTR plugin for ES. Honestly Solr and ES are more similar than different. There are a few things Solr has which ES doesn't and the reverse is true too. 1. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
    - hashhar 8y ago
      Thanks for letting me know about Elasticsearch LTR.
  - bryanrasmussen 8y ago
    Both solr and elasticsearch use lucene to index.
    - hashhar 8y ago
      ES doesn't expose as many knobs last I checked.
  - k9s9 8y ago
    The difference is in the size of your Sales team. Lot of open source projects would benefit, if they had dedicated sales teams contacting firms day in, day out talking about their features. People who run IT departments in the Enterprise world or even small firms lacking resources to keep up, just pick tools and make software decisions based on who reaches out to them. Invest in a sales team and you penetrate markets that don't spend time monitoring developments in the tech world which is really the majority of all orgs. Elastic has done that quite well and is reaping the rewards.
burntsushi 8y ago
Last time I checked, PostgreSQL doesn't support any kind of tfidf ranking scheme because it doesn't track corpus frequencies of terms in the index. This impacts how well relevance ranking works for some workloads.

lykr0n 8y ago

ZoomDB (https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb) offers something like this.

33degrees 8y ago
Didn’t know about this project, very cool.
- zombodb 8y ago
  Thanks! It’s been fun to develop it and we love when people discover and use it

FridgeSeal 8y ago

There’s a super neat plugin for Postgres called “ZomboDB” that hooks up ElasticSearch to Postgres, so when you make full text queries, it gets performed on ES. https://www.zombodb.com/ https://www.zombodb.com/ I had a brief play with it for a project at work and it was super straightforward to get running.

jedisct1 8y ago

PGroonga https://pgroonga.github.io/ https://pgroonga.github.io/