4 ms·

I read somewhere today that a key benefit of transformers is they are effective at creating and using really long chains of dependencies. In my mind I interpre

I read somewhere today that a key benefit of transformers is they are effective at creating and using really long chains of dependencies. In my mind I interpreted that as “chatgpt feels like a conversation because the transformer model is emulating like 10, 20, 30 years of language practice /knowledge (not necessarily intelligence, but patterns and knowledge) with every query” meaning it goes way deeper than any neural net that came before it. Is that more or less accurate in my very layman’s understanding?

dartos 3y ago
Think of each token (roughly each word) as having a specific signal or tone. Each token after the first has a mix of the tones that came before. The transformer can guess what the tone of the next token is. The tones and how new tones are mixed from older tones are learned through training. Each tone is mathematically dependent on previous tones. It creates a dependency chain (more like a map, I think) in this way. “Feels like a conversation” is a rough metric to understand, but I think that feel mostly comes from how chatgpt is presented from a UX perspective. Personally I don’t like using words like “practice” or phrases like “20 years of knowledge” because they’re fuzzy and don’t really reflect what’s going on under the hood. Imo they make things harder to understand
- hammock 3y ago
  Yeah conversation was my word for “better” but I didn’t mean it any deeper than that. “20 years of knowledge “ I just mean the map is bigger and more filled in I think we are saying along the same lines
  - dartos 3y ago
    I’d recommend reading the paper “Attention is all you need” as it lays a lot of the foundational knowledge about transformers. I’m no math guru, so I had to read the paper like 5 or 6 times to wrap my head around it. I had to stop trying to understand how the math worked exactly and just accepted that it did, then it started to make sense. Now going back I can actually understand some of why the math works.

dartos 3y ago

Think of each token (roughly each word) as having a specific signal or tone. Each token after the first has a mix of the tones that came before. The transformer can guess what the tone of the next token is. The tones and how new tones are mixed from older tones are learned through training. Each tone is mathematically dependent on previous tones. It creates a dependency chain (more like a map, I think) in this way. “Feels like a conversation” is a rough metric to understand, but I think that feel mostly comes from how chatgpt is presented from a UX perspective. Personally I don’t like using words like “practice” or phrases like “20 years of knowledge” because they’re fuzzy and don’t really reflect what’s going on under the hood. Imo they make things harder to understand

hammock 3y ago
Yeah conversation was my word for “better” but I didn’t mean it any deeper than that. “20 years of knowledge “ I just mean the map is bigger and more filled in I think we are saying along the same lines
- dartos 3y ago
  I’d recommend reading the paper “Attention is all you need” as it lays a lot of the foundational knowledge about transformers. I’m no math guru, so I had to read the paper like 5 or 6 times to wrap my head around it. I had to stop trying to understand how the math worked exactly and just accepted that it did, then it started to make sense. Now going back I can actually understand some of why the math works.