4 ms·
I like this, but think there is some crucial motivation missing in steps 10.1-10.3 regarding what query/key weights are and why they're needed.
by causal 3y ago
I like this, but think there is some crucial motivation missing in steps 10.1-10.3 regarding what query/key weights are and why they're needed.
- ThouYS 3y agoyes, same issue in all transformer tutorials
- causal 3y agoThe 2b1b video was the first to make it click for me
- hotdogscout 3y agoYou mean 3b1b (three blue one brown)?
- causal 3y agoAh that's right, miscounted the blues
- lordswork 3y agoI suspect this is because most people (including people writing these tutorials) don't have a strong grasp on this piece as well.
- vikiomega9 3y agothis post made sense to me https://teltam.github.io/posts/soft-dictionary-keys.html https://teltam.github.io/posts/soft-dictionary-keys.html It helps to think of kqv as a form of look up.
- abhgh 3y agoThey are like "continuous" databases. See slides 4-5 here [1] - this is from a talk I had given a while ago. [1] https://drive.google.com/file/d/12uHo9QIfS-jBpVTs3lmQ3BEpxhDSiO8G/view?usp=drive_link https://drive.google.com/file/d/12uHo9QIfS-jBpVTs3lmQ3BEpxhD...