5 ms·
Who is commenting on this and what will they say??! 1-2 trillion tokens of training for gptx class models It would take 20 years for a human to read that many
by breatheoften 4y ago
Who is commenting on this and what will they say??!
1-2 trillion tokens of training for gptx class models
It would take 20 years for a human to read that many tokens at 8 hrs of reading per day (although i'm pretty unclear on whether that's unique token sequences or randomly selected token pairs -- feels like a big difference between the two!!)
We are all here now -- who thinks we are anywhere of note together?
Personally I think that models that do not experience incremental change remain an entirely separate class of intelligence from those which evolve continuously -- but I am open to being persuaded otherwise (certainly I never thought making gpt2 bigger would be as impressive as gpt4) ...
- meindnoch 4y ago>1-2 trillion tokens of training for gptx class models >It would take 20 years for a human to read that many tokens at 8 hrs of reading per day Normal reading is ~200 words per minute. That's 12,000 words per hour. That's 288,000 words per day (reading for 24 hours straight). That's 105,120,000 words per year. That's 2,102,400,000 words per 20 years. A trillion is 1,000,000,000,000.
- tonightstoast 4y agoPlus almost no one is reading academic papers at 200 words per minute. So for the level of material it could mean that number is halved per 20 years.
- lostmsu 4y agoVideo from a pair of eyes is many terabytes per year though. If you compress it with best codecs. Raw video would be closer to 1PB per year. I think people overestimate the data efficiency of the biological neural networks w.r.t. transformer models.
- actionfromafar 4y agoThe power effiency is pretty brutal though, in favour of bio-gelpacks.
- lostmsu 4y agoHow can you be sure about power efficiency, if you don't know how data efficiency compares? It is low power. But so is a phone running llama.cpp
- actionfromafar 4y agoYou must be pretty clever to write that, and you are human right? Your data effiency and power effiency seems top notch to me.
- lostmsu 4y agoMaybe I am, but how many are out there who aren't? And how many smartphones can run alpaca?
- Thiez 4y agoPeople with bad eyesight are not (to the best of my knowledge) less intelligent than those with perfect vision. We could probably supply video at a much lower resolution without significantly affecting the outcome.
- lostmsu 4y agoTrue, but it's not like sound and/or touch are low bandwidth channels either.
- GordonS 4y agoAnd, as humans, we don't only get "tokens" from what we read, but from our entire sensorium.