4 ms·
How might LLMs store facts [video]
- cl3misch 2y agoThe mentioned "Johnson–Lindenstrauss lemma" is especially interesting: it essentially states that for N>3, the number M of almost perpendicular vectors you can fit into an N-dimensional space is M>>N. This makes it possible to embed a lot of facts into the encoding vectors of an LLM.