3 ms·

I don't think it's possible to develop a frontier model without mass scraping. The economics simply don't add up. You need at least 10 trillion tokens to make a

by xena 2y ago

I don't think it's possible to develop a frontier model without mass scraping. The economics simply don't add up. You need at least 10 trillion tokens to make an 8 billion parameter model. 10 trillion tokens is something like 40 terabytes. You simply can't get 40 terabytes of text without mass scraping.

mdp2021 2y ago
> You need at least 10 trillion tokens to make an 8 billion parameter model Are you sure it is not just very inefficient?
- GTP 2y ago
  I'm not an AI expert, but it seems to me that the common consensus is that current LLMs are quite inefficient and that there's room for improvement.