3 ms·
I actually built a search engine back in 2018 using postgresql https://austingwalters.com/fast-full-text-search-in-postgresql/ https://austingwalters.com/fast-
by lettergram 5y ago
I actually built a search engine back in 2018 using postgresql
https://austingwalters.com/fast-full-text-search-in-postgresql/ https://austingwalters.com/fast-full-text-search-in-postgres...
Worked quite well and still use it daily. Basically doing weighted searches on vectors is slower than my approach, but definitely good enough.
Currently, I can search around 50m HN & Reddit comments in 200ms on the postgresql running on my machine.
- rattray 5y agoNice – looks like the ~same approach recommended here of adding a generated `tsvector` column with a GIN index and querying it with `col @@ @@ to_tsquery('english', query)`.
- lettergram 5y agoYeah my internal approach was creating custom vectors which are quicker to search.
- vincnetas 5y agoOfftopic, but currious what are your use cases when searching all HN and reddit comments? Im at the beggining of this path, just crawled HN, but what to do with this, still a bit cloudy.
- lettergram 5y agoI built a search engine that quantified the expertise of authors of comments. Then I created what I called “expert rank” that allowed me to build a really good search engine. Super good if you’re at a company or something https://twitter.com/austingwalters/status/1041894765439201281?s=21 https://twitter.com/austingwalters/status/104189476543920128...
- turbocon 5y agoWow, that lead me down quite a rabbit hole, impressive work.
- busymom0 5y agoIf you are just interested in searching HN and don't need full data dumps, the Algolia search for HN is quite good and fast: https://hn.algolia.com https://hn.algolia.com They also have a free API. If you need data dumps, maybe look into Google BigQuery.