4 ms·
I think that it's a poorly named reference to this paper [1] that they mention in the blogpost. If I had to give it another more descriptive name, I would proba
by andy12_ 1y ago
I think that it's a poorly named reference to this paper [1] that they mention in the blogpost. If I had to give it another more descriptive name, I would probably name it "Per-Layer Embedding Dimensionality"
[1] https://arxiv.org/pdf/2310.07707 https://arxiv.org/pdf/2310.07707
- yorwba 1y agoThe MatFormer is clearly called out as a different aspect of the model design. PLE is much more likely to be a reference to the Per-Layer Embeddings paper that will be published in the future once it doesn't give away any secret sauce anymore.
- andy12_ 1y agoI thought the same, but Per-Layer Embeddings as a name doesn't make sense in any context, and MatFormer does exactly what the blogpost says PLE does. I just think it's more probable that the blogpost was written by several authors and that noone bothered to check the final result.