3 ms·
I can't believe anyone considers a single number, which would work about equally well if it were 10% higher or lower, to be a trade secret.
by whatshisface 4y ago
I can't believe anyone considers a single number, which would work about equally well if it were 10% higher or lower, to be a trade secret.
- typest 4y agoThis is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics. Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.
- whatshisface 4y agoThat ten percent is not loss it is parameter count.
- pixl97 4y agoIt's about causing your competition to waste millions of dollars in compute time and power doing something unproductive. There is not a huge pile of excess TPUs laying around for people to use. Any strategic advantage can quickly compound and put you well ahead of others.