5 ms·

There's a million paths, but here's one I like. Use ElasticSearch. It will scale more than Postgres. Three hosted options are AWS, Elastic, Bonsai. I founde

by fizx 3y ago

There's a million paths, but here's one I like. Use ElasticSearch. It will scale more than Postgres. Three hosted options are AWS, Elastic, Bonsai. I founded Bonsai and retired (so am partial), but they will provide the best human support for you, and you won't have to worry about java Xmx. Your goal with ES is to use the Regex PatternAnalyzer to split the code into reasonable exact code-shaped tokens (not english words). Here's a rough GPT4 explanation with sample config that I'd head towards: https://chat.openai.com/share/e4d08586-b7ef-48f2-9de1-7f82ea3c1f14 https://chat.openai.com/share/e4d08586-b7ef-48f2-9de1-7f82ea...

bytefish 3y ago
GitLab is also using ElasticSearch, so one could recreate the ElasticSearch Indices they came up with. [1] They also share some of the challenges, they faced along the way. It also discusses interesting challenges, like implementing the authorization model. [2], [3] When GitHub removed its most useful Search feature, which is sorting results by date, I wrote a small “Search Engine” with ElasticSearch to selectively index Microsoft repositories. It works good enough for my needs. [4] [1] https://gitlab.com/gitlab-org/gitlab/-/blob/7bbbc00bd871aeb67660a1c42ff30c65ba6ed47a/ee/lib/elastic/latest/config.rb https://gitlab.com/gitlab-org/gitlab/-/blob/7bbbc00bd871aeb6... [2] https://about.gitlab.com/blog/2019/07/16/elasticsearch-update/ https://about.gitlab.com/blog/2019/07/16/elasticsearch-updat... [3] https://about.gitlab.com/blog/2020/04/28/elasticsearch-update/ https://about.gitlab.com/blog/2020/04/28/elasticsearch-updat... [4] https://github.com/bytefish/ElasticsearchCodeSearch https://github.com/bytefish/ElasticsearchCodeSearch
philippemnoel 2y ago
Elasticsearch is good, and it does scale, but it is much more cumbersome and expensive to scale and operate than Postgres. If you use the managed service, you'll pay for the operational pain in the form of higher pricing. The Postgres movement is strong and extensions like ParadeDB https://github.com/paradedb/paradedb https://github.com/paradedb/paradedb are designed specifically to solve this pain point (Disclaimer: I work for ParadeDB)

by fizx 3y ago

bytefish 3y ago

GitLab is also using ElasticSearch, so one could recreate the ElasticSearch Indices they came up with. [1] They also share some of the challenges, they faced along the way. It also discusses interesting challenges, like implementing the authorization model. [2], [3] When GitHub removed its most useful Search feature, which is sorting results by date, I wrote a small “Search Engine” with ElasticSearch to selectively index Microsoft repositories. It works good enough for my needs. [4] [1] https://gitlab.com/gitlab-org/gitlab/-/blob/7bbbc00bd871aeb67660a1c42ff30c65ba6ed47a/ee/lib/elastic/latest/config.rb https://gitlab.com/gitlab-org/gitlab/-/blob/7bbbc00bd871aeb6... [2] https://about.gitlab.com/blog/2019/07/16/elasticsearch-update/ https://about.gitlab.com/blog/2019/07/16/elasticsearch-updat... [3] https://about.gitlab.com/blog/2020/04/28/elasticsearch-update/ https://about.gitlab.com/blog/2020/04/28/elasticsearch-updat... [4] https://github.com/bytefish/ElasticsearchCodeSearch https://github.com/bytefish/ElasticsearchCodeSearch

philippemnoel 2y ago

Elasticsearch is good, and it does scale, but it is much more cumbersome and expensive to scale and operate than Postgres. If you use the managed service, you'll pay for the operational pain in the form of higher pricing. The Postgres movement is strong and extensions like ParadeDB https://github.com/paradedb/paradedb https://github.com/paradedb/paradedb are designed specifically to solve this pain point (Disclaimer: I work for ParadeDB)