3 ms·
Typesense looks easy to use. But with 3x memory needed for the indexes, the server costs probably aren't going to be bootstrap'able. Especially for a "small"
by aantix 5y ago
Typesense looks easy to use.
But with 3x memory needed for the indexes, the server costs probably aren't going to be bootstrap'able.
Especially for a "small" crawl of a billion web pages, event at just 10k per page.
- marginalia_nu 5y agoEh, I run a 100M-index off consumer hardware in my living room. Very doable if you avoid bloated off the shelf solutions.
- aantix 5y agoWhat search software do you run? What sort of memory and space do you have on the single server? What's the average document size that you index? Genuinely curious on how doable a modern search engine is on modern hardware.
- marginalia_nu 5y agoI rum all custom software, I feel most off the shelf solutions aren't very resource effective. The server has 128 Gb RAM and the index currently fits on a single 1 Tb SSD + an Optane drive of 480 Gb. I find the average document to clock in at 7 Kb, in terms of raw HTML. In the index that's, dunno, probably less than 1 KB/doc.