4 ms·
Cool, but isn't this encoding a potentially very long thinking process into a fixed embedding? Intuitively should not work as well.
by antirez 2y ago
Cool, but isn't this encoding a potentially very long thinking process into a fixed embedding? Intuitively should not work as well.
- pishpash 2y agoThat's already the case with visible text. There's an embedding inside the model as it spits out the next token.
- antirez 2y agoSure, but you have multiple thoughts tokens in the context the model sees to process the next token.
- pona-a 2y agoBut don't words have a fixed size embedding? Causal models create a sequence of attended word embeddings. It's only about 300 dimensions per one word, so it seems counterintuitive you can compress an entire reasoning chain into such a small vector.