5 ms·
why do companies use databases like elastic search when they could just use inverted indexes in postgres? Performance, features, and scalability
by brigadier132 3y ago
why do companies use databases like elastic search when they could just use inverted indexes in postgres?
Performance, features, and scalability
- simonw 3y agoI use Elasticsearch along with PostgreSQL, but I don't think of Elasticsearch as a database. It's an external index that I denormalize some of my data into.
- tyre 3y agoES is a good parallel to a vector database. run a search against ES => hydrate records from postgres run a search against a vector database => hydrate LLM context from documents Neither is a primary canonical datastore. Postgres is for ES and S3 is for documents.
- riku_iki 3y agoso, its just happened that there is no good implementation of full text search in postgres, but fundamentally nothing prevents from implementing such search, except horizontal scalability if it is needed.
- johnthescott 3y agothe pg extension called "rum" is very performant on millions of pdfs. https://github.com/postgrespro/rum https://github.com/postgrespro/rum
- riku_iki 3y agomy case is tens of billions of short docs, and default search is fine except index creation is single threaded, so it takes multiple days to build index, and github page says rum is even slower..
- aforwardslash 3y agoI love Postgres, but you're comparing apples and oranges. You don't have the same level of query sophistication neither the same indexing quality you get from eg. elasticsearch. As a quick example, the other day I got a bug in PostgreSQL because "searching" does not perform diacritics conversion. Also, stuff like stop words, similarity search and whatnot either required dedicated indexes or are downright "round hole square peg" problems.
- mkarrmann 3y agoI believe they're being rhetorical, and agree that there are good reasons to use elasticsearch over postgres.