3 ms·
The essence of the bitter lesson is that the less inductive bias we try to bake in, the better performing they tend to be. Transformers are preferred to ConvNe
by famouswaffles 3y ago
The essence of the bitter lesson is that the less inductive bias we try to bake in, the better performing they tend to be.
Transformers are preferred to ConvNets these days in Computer Vision despite the latter having all sorts of vision based inductive biases.
GPT-4 has not been trained on all text lol
- niemandhier 3y ago1. Partially that just moves where we must search for implicitly included symmetries. Word embedding most famously create structure that allows to define shift operations in the embedding space “doctor + female = nurse” might be the most infamous. By evolution only those word embedding so that yield good results are used. Again just because we did not put the structure there, but discovered it by trial and error does not mean that the structure is not key to the success. 2. Gpt 4 was trained on 13T tokens, all books ever written would be about 6.5 T tokens by Fermi calculation.
- famouswaffles 3y ago1. Sorry but that's still far less bias than ConvNets for vision. And far less bias than previous methods for language. The less bias the better. It's simple. 2. Assuming the rumor is fact, GPT was trained for multiple epochs and books are a small percent of what trains LLMs lol.