4 ms·
Interesting, I wonder what they used for the search database, since that enormous amount of text can't fit in RAM, it would have to be partitioned & sharded in
by tmikaeld 5y ago
Interesting, I wonder what they used for the search database, since that enormous amount of text can't fit in RAM, it would have to be partitioned & sharded in something like Scylla DB
- gilbetron 5y agoTheir job postings mention Solr: > Experience with Solr or similar search technologies. So I'm guessing that's most of it. Which makes sense given Full Text Search is the point of Solr/Lucene!
- speedgoose 5y agoYou can rent from various cloud providers many VMs with 2TB of ram each. AWS also offers some high memory machines with up to 24TB of ram in some locations.
- tmikaeld 5y agoSure you can do that, but that's a hefty price per user.
- winrid 5y agoIIRC they have a pretty standard PostGres setup. I'd bet they just setup another PostGres shard replicated just for search, using an extension for the index. Doesn't require the index or working set to be in RAM.
- philliphaydon 5y agoWhy do some people write Postgres as PostGres?
- __alexs 5y agoBecause it's the database that came after Ingres.
- stavros 5y agoWhich isn't spelles InGres, is it?
- philliphaydon 5y agohttps://en.wikipedia.org/wiki/Ingres_(database) https://en.wikipedia.org/wiki/Ingres_(database) https://en.wikipedia.org/wiki/PostgreSQL https://en.wikipedia.org/wiki/PostgreSQL Ingres and Postgres/PostgreSQL. I still don't know /why/ some people say PostGres. I've seen it a few times, on HN and elsewhere, but its wrong isn't it?
- winrid 5y agoWhy do some people call Redis "read-is"? The world may never know...
- datalopers 5y agoinverted indexes do not require that much overhead
- lovelearning 5y agoAt one point, their search used LucidWorks Fusion [1], a commercial product based around Solr (that uses Lucene indexes under the hood) but also integrates a vector database and the like for semantic search. The linked wiki page still has Lucene-style queries. [1]: https://lucidworks.com/customers/reddit/ https://lucidworks.com/customers/reddit/
- tmikaeld 5y agoThat's kind of my point, Solr and Lucene put the index in memory.
- liveoneggs 5y agonot the entire index, only what is needed for each query.
- omegalulw 5y agoCan't the index be shared across a cluster? Possibly with multiple replicas for a share to allow for servers to easily go up and down?
- ddorian43 5y agocan't use scylladb as a search engine
- inciampati 5y agoI doubt that a succinct full text index (like an FM-index) of this data would require more than a modest server to keep in memory. Why aren't these used in this context?
- tmikaeld 5y agoBut we're talking petabytes of text comments and an index of that would be a lot larger. How do you access that data fast enough to enable search?
- inciampati 5y agoSuccinct full text indexes can be substantially smaller than the source text. It depends on the zero order entropy of the text. If things are highly repetitive, a very small index might be feasible. Usually lookup times are linearly proportional to query size, with logarithmic factors in database size.
- tmikaeld 5y agoI've yet to see such a system (in production) except for Sonic, but sonic doesn't allow for full-text search only search on a key-by-key basis.