4 ms·
Curious what embedding compression technique they're using since it allows for dynamic dimensionality reduction. Or maybe they trained a different projection fo
by pizza 3y ago
Curious what embedding compression technique they're using since it allows for dynamic dimensionality reduction. Or maybe they trained a different projection for each conceivable dim argument?
- isoprophlex 3y agoJust PCA the thing on a couple billion embeddings, store those coefficients (only ~3000 floats) and in the API, report the n components with the highest variance/eigenvalue?
- byefruit 3y agoProbably something similar to https://arxiv.org/abs/2310.07707 https://arxiv.org/abs/2310.07707
- teaearlgraycold 3y agoHonestly the answers to these questions are usually the most obvious thing. It could just be some basic dimensionality reduction technique precomputed for each input/output size combination. That’s still just a few thousand matrices. I’m sure they can handle the training and distribution of that set.
- danielcampos93 3y agoProbably as simple as training the smaller model to approximate the larger model. Well studied and done via tinylm and minilm.