3 ms·
I understand this part. They learn how to continue text. But what rules are they discovering - how do they become so good at text continuation?
by iamflimflam1 4y ago
I understand this part. They learn how to continue text.
But what rules are they discovering - how do they become so good at text continuation?
- t-vi 4y agoDo they need to learn rules though, or could they memorize enough to learn the probabilities of the most likely continuation? To my mind the blurry-jpeg-metaphor[1] is very much spot-on. While it is speculation, it would seem to me personally that LLMs de-facto doing a (lossy) memorization and neural networks in general being able to do well in "local consistency" seems a way to think about them that is consistent with most of the model behaviour we observe. 1. https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
- rnosov 4y agoMy understanding that the breakthrough was the attention mechanism where attention layers learn where important words are.
- jstx1 4y ago> But what rules are they discovering The point of ML is that you don't think about the rules explicitly, you let the algorithm figure them out from the data. > how do they become so good at text continuation? Good underlying algorithm (transformers do better than other algorithms we've tried) + lots of data. I guess this isn't very satisfying but you're looking for some super deep and profound answer and there is none.