4 ms·
AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits
by minimaxir 6y ago
AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits of GPT-2/Transformer architectures is that they are capable of scaling indefinitely, but they still have diminishing returns)
- Voloskaya 6y agoI completly agree, it's a diminishing return but a return nonetheles, so to have a "world class GPT" model with 10% of the data, and 10% (optimistically) of the compute that OpenAI used, you will need some very very impressive innovation elsewhere.