4 ms·
I found this description of the GPT-3 transformer architecture useful: https://dugas.ch/artificial_curiosity/GPT_architecture.html https://dugas.ch/artificial_c
by bryan0 4y ago
I found this description of the GPT-3 transformer architecture useful: https://dugas.ch/artificial_curiosity/GPT_architecture.html https://dugas.ch/artificial_curiosity/GPT_architecture.html
Not eli5 but close enough.
- petesergeant 4y agoSo it really is just a fancy Markov chain?
- dotancohen 4y agoNo.
- tarvaina 4y agoA Markov model is a stochastic process with a discrete visible state where the next state depends only on the current state. GPT’s state is neither discrete nor visible so it is not a Markov model.
- miltondts 4y agoWhat I don't understand is where is the memory? How does GPT-3 or ChatGPT remember so much information with just that architecture? It would seem that the maximum it could remember is 2048 words. EDIT: Maybe it's 2048 x 96? Still seems low for what it can do.
- mjburgess 4y ago300bn weights, at 4bytes/weight is 1.2TB
- Epa095 4y agoYes, but how does it remember the stuff you told it earlier in the conversation? Those 1.2TB is the trained model, and I assume that those weights are not changed by the conversation?
- bryan0 4y agoI believe that the previous input, from earlier in the conversation, is always prepended to the new input.