4 ms·
1. Partially that just moves where we must search for implicitly included symmetries. Word embedding most famously create structure that allows to define shift
by niemandhier 3y ago
1. Partially that just moves where we must search for implicitly included symmetries. Word embedding most famously create structure that allows to define shift operations in the embedding space “doctor + female = nurse” might be the most infamous. By evolution only those word embedding so that yield good results are used. Again just because we did not put the structure there, but discovered it by trial and error does not mean that the structure is not key to the success.
2. Gpt 4 was trained on 13T tokens, all books ever written would be about 6.5 T tokens by Fermi calculation.
- famouswaffles 3y ago1. Sorry but that's still far less bias than ConvNets for vision. And far less bias than previous methods for language. The less bias the better. It's simple. 2. Assuming the rumor is fact, GPT was trained for multiple epochs and books are a small percent of what trains LLMs lol.