4 ms·

It seems to explain them well because you already know how they work. To most people, that diagram would not explain very much.

by raylad 3y ago

KirillPanov 3y ago
No, I didn't! Before I came across that image (and the paper Linearized Transformers are Secretly Fast Weight Programmers) I very much did not understand transformers. I had spent at least 20 hours with Attention Is All You Need, and probably another dozen hours with https://e2eml.school/transformers.html https://e2eml.school/transformers.html and wasn't getting much of anywhere. I have no ML/AI background; just a typical undergraduate-CS-level familiarity with neural networks -- the basic stuff that hasn't changed since the 1990s. I do have some experience with linear algebra, but that isn't the hard part of any of this. Frankly, most people who publish in this field go out of their way to obfuscate the key insights. Mediocre physicists (but not the truly brilliant ones) do the same thing. It's very annoying.
- xcv123 3y ago
  So the diagram makes sense after you spent at least 32 hours studying LLMs.
  - ggm 3y ago
    There's this sentence about monoids...
    - deleted 3y ago
      [deleted]
- ben_w 3y ago
  Different things click for different people; for me, the image you linked in the initial post of this thread… it makes as little sense to me as the Time Cube image or the widely mocked Pepsi rebrand document from 2008. "That it's objectively true" isn't enough of an advantage, sadly. > Frankly, most people who publish in this field go out of their way to obfuscate the key insights. Mediocre physicists (but not the truly brilliant ones) do the same thing. It's very annoying. I certainly sympathise, but I think the overlap here is "maths is hard to communicate", and the conflict between rigorously explaining vs. keeping everything simple enough to follow. Grant Sanderson did a talk on this, though I appreciate an hour may be a bit too long: https://youtu.be/UOuxo6SA8Uc?si=e4dojDIXACdI44sb https://youtu.be/UOuxo6SA8Uc?si=e4dojDIXACdI44sb

It seems to explain them well because you already know how they work. To most people, that diagram would not explain very much.

by raylad 3y ago

It seems to explain them well because you already know how they work. To most people, that diagram would not explain very much.

KirillPanov 3y ago

No, I didn't! Before I came across that image (and the paper Linearized Transformers are Secretly Fast Weight Programmers) I very much did not understand transformers. I had spent at least 20 hours with Attention Is All You Need, and probably another dozen hours with https://e2eml.school/transformers.html https://e2eml.school/transformers.html and wasn't getting much of anywhere. I have no ML/AI background; just a typical undergraduate-CS-level familiarity with neural networks -- the basic stuff that hasn't changed since the 1990s. I do have some experience with linear algebra, but that isn't the hard part of any of this. Frankly, most people who publish in this field go out of their way to obfuscate the key insights. Mediocre physicists (but not the truly brilliant ones) do the same thing. It's very annoying.

xcv123 3y ago
So the diagram makes sense after you spent at least 32 hours studying LLMs.
- ggm 3y ago
  There's this sentence about monoids...
  - deleted 3y ago
    [deleted]
ben_w 3y ago
Different things click for different people; for me, the image you linked in the initial post of this thread… it makes as little sense to me as the Time Cube image or the widely mocked Pepsi rebrand document from 2008. "That it's objectively true" isn't enough of an advantage, sadly. > Frankly, most people who publish in this field go out of their way to obfuscate the key insights. Mediocre physicists (but not the truly brilliant ones) do the same thing. It's very annoying. I certainly sympathise, but I think the overlap here is "maths is hard to communicate", and the conflict between rigorously explaining vs. keeping everything simple enough to follow. Grant Sanderson did a talk on this, though I appreciate an hour may be a bit too long: https://youtu.be/UOuxo6SA8Uc?si=e4dojDIXACdI44sb https://youtu.be/UOuxo6SA8Uc?si=e4dojDIXACdI44sb