4 ms·
It’s been two more years for the past three years. What’s going to happen in those two years? Are we going to find 10 more internets to train on?
by bopbopbop7 1y ago
It’s been two more years for the past three years. What’s going to happen in those two years? Are we going to find 10 more internets to train on?
- Zacharias030 1y agotwo years ago it didn’t do anything that made me want to use it during work. The data wall idea of needing 10 more internets is like the peak oil theory of the 2000s, imho. I don’t think 10 more internets are required, and all those tasks like proper security and i18n, none of these require creativity of any kind. These are ideal candidates for LLMs to solve quite soon. But to talk about the data wall more: Current SOTA LLMs are trained on circa 30-50T tokens. First, that is not a full internet‘s worth; it is estimated that Meta owns more like 200T tokens of user data etc. Second, the work in optimizing for data efficiency is just beginning.
- bopbopbop7 1y agoThe current training data is not the full internet worth because most of the internet is garbage data. It is removed for a reason. But sure, those 200T tokens of Facebook user data is the answer to smarter LLMs.