3 ms·
Think of each token (roughly each word) as having a specific signal or tone. Each token after the first has a mix of the tones that came before. The transform
by dartos 3y ago
Think of each token (roughly each word) as having a specific signal or tone.
Each token after the first has a mix of the tones that came before.
The transformer can guess what the tone of the next token is.
The tones and how new tones are mixed from older tones are learned through training.
Each tone is mathematically dependent on previous tones.
It creates a dependency chain (more like a map, I think) in this way.
“Feels like a conversation” is a rough metric to understand, but I think that feel mostly comes from how chatgpt is presented from a UX perspective.
Personally I don’t like using words like “practice” or phrases like “20 years of knowledge” because they’re fuzzy and don’t really reflect what’s going on under the hood.
Imo they make things harder to understand
- hammock 3y agoYeah conversation was my word for “better” but I didn’t mean it any deeper than that. “20 years of knowledge “ I just mean the map is bigger and more filled in I think we are saying along the same lines
- dartos 3y agoI’d recommend reading the paper “Attention is all you need” as it lays a lot of the foundational knowledge about transformers. I’m no math guru, so I had to read the paper like 5 or 6 times to wrap my head around it. I had to stop trying to understand how the math worked exactly and just accepted that it did, then it started to make sense. Now going back I can actually understand some of why the math works.