4 ms·
Ask HN: Full Text Search
Hey HN,
I need full-text search for a large amount of text data (500gb++).
I've looked at Algolia, Meilisearch, Typesense, Elastic. From what I can tell, they all require a server which can keep everything in memory.
MySQL full text search is probably sufficient, so maybe I'm asking about best practices to store a MySQL database at terabyte scale.
Thoughts on how to scale a database and do faceted full-text search without it costing a ton of $$?
- throwup238 2y agoHave you seen tantivy-cli [1]? It builds and uses an on disk index, which you can store separately (i.e. data on NAS, index locally). [1] https://github.com/quickwit-oss/tantivy-cli https://github.com/quickwit-oss/tantivy-cli
- iambateman 2y agoOooh I ended up using quickwit's main service, instead of just tantivy. That way I can get distributed indexes on S3 for cheap. Thanks for pointing the way!
- qwm 2y agoWhat are your needs? I quite dislike MySQL's fulltext search as it's much less configurable than say, ElasticSearch. I would recommend something like ElasticSearch if you need flexibility down the line. I've used it happily for a few years now as a search engine for a content site I work on.
- snikolaev 2y agoCheck out Manticore Search for your use case. It's open-source, cost-effective, and doesn't require keeping everything in memory. Key points: - Columnar Storage: Efficiently handles large datasets on disk, ideal for terabyte-scale data. It's not enabled by default but can be set up easily with "CREATE TABLE ... ENGINE='columnar'". - Faceted Search: Probably easier than anywhere else with just "FACET <field name>" added to your "SELECT" query. - MySQL Protocol and SQL Support: If you’re familiar with SQL and MySQL, it's easier to get started compared to other search engines.
- iambateman 2y agoThanks for your recommendation. I ended up going with Quickwit, since it lets me store data on S3.