3 ms·
1) common crawl is >100TB so obviously contains more than 20trn tokens + Ilya has said many times in interviews that there is still way more data for training u
by tempusalaria 3y ago
1) common crawl is >100TB so obviously contains more than 20trn tokens + Ilya has said many times in interviews that there is still way more data for training usage >10x
2) GPT-4 is way slower so this point is irrelevant
3) OpenAI have a 10000 A100 training farm that they are expanding to 2500. They are spending >$1mln on compute per day. They have just raised $10bln. They can afford to pay for inference
- CaptainNegative 3y ago> OpenAI have a 10000 A100 training farm that they are expanding to 2500. Does the first number have an extra zero or is the second number missing one?
- tempusalaria 3y agoSecond number is missing a zero sorry. Should be 10000 and 25000
- deleted 3y ago[deleted]