4 ms·
Can someone explain, is this a gigantic char-rnn, picking the next most likely character and then feeding it back into the system?
by keepquestioning 4y ago
Can someone explain, is this a gigantic char-rnn, picking the next most likely character and then feeding it back into the system?
- eks391 4y agoI haven't worked on GPT, but if I had to guess, it certainly is not a char-rnn. There are more sofisticated versions of char-rnn that use words chunks that work better (think of prefixes, etc), as well as some that actually use entire words. Theres one that isn't an rnn, that off the top of my head I cant recall what type of nn it is, but it is a list of all words with weights of intrinsic meaning (obv not manually made), combined with the traditional rnn use so grammar stays functional. I would suspect that GPT is a super huge one of these.
- keepquestioning 4y agoHow does GPT know how to answer questions
- eks391 4y agoBeats me. I looked on wikipedia to see if GPTs framework is described without luck. It did say the parameters file (comparable to a csv, very lightweight) is nearly a terabyte, which is absolutely insane. Although I want to guess that it is more than an rnn because of how good GPT is, it could be successful solely to its sheer size