4 ms·
I'm one of the authors of the paper, nice to see it on HN. I remember when we first experimented with this, the compression improvement compared to our previou
by ot 2y ago
I'm one of the authors of the paper, nice to see it on HN.
I remember when we first experimented with this, the compression improvement compared to our previous heuristic was massive, but the algorithm took a day to run on a double-digit-node Giraph cluster for a single index shard. I was very skeptical we'd ever be able to use it in production, given that we had to run it on thousands of index shards every few days.
Eventually we reimplemented it in C++, optimized all the data structures to make them fit in memory, and we were able to run it in a couple of hours on a single (beefy) machine. Over the years it has been optimized further.
The algorithm has been reproduced externally with an open-source implementation [1], which AFAIR was pretty good when I looked at it.
[1] https://culpepper.io/publications/mm+19-ecir.pdf https://culpepper.io/publications/mm+19-ecir.pdf
- godelski 2y agoI actually think this is a great reminder to how much things can be optimized. Especially if we are talking about research code. All too often I see people point to python code and talk about it being slow as if someone couldn't rewrite it in C for massive improvements. llama.cpp is a great example of this and there's lots of similarly low hanging fruit in ML