3 ms·

Cutting edge models are quite expensive to train, so it'd be quite possible to restrict them (unfortunately).

Topfi 3y ago
You are absolutely right for the moment, but the question is whether this will always remain the case. I am honestly doubtful and fairly certain this will be more a question of when than if. Seeing as both Nvidia and Apple continue making consumer hardware that does well in inferencing, I fail to see why training sufficiently capable models locally may not become more common place, especially as there are already murmurs in the field that overly large models (100B+) may not be necessary for many use cases. Add to that the fact that institutions (including research institutes[0][1] or universities outside US or EU jurisdictions) are likely to continue undertaking the task of training larger in size models, making the fruits of their labor freely accessible, I just do not see a way for centralized enforcement of specific LLM training rules. [0] https://huggingface.co/tiiuae/falcon-40b-instruct https://huggingface.co/tiiuae/falcon-40b-instruct [1] https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
- dkarras 3y ago
  Yeah, we will find ways of training these things faster, and we will have faster hardware. We will have better, more parameter efficient models. This will inevitably be something we can do locally. You can already trade time for money. Like, right now you can buy hardware for less than what a house costs, train using the hardware for a year and you basically have something as capable as GPT3.5 - this is already possible and needs to be done only once (and you can still sell the hardware if you want) and a single dedicated person is capable of doing it by themselves. It will only get faster, and cheaper from here. Same goes for inference, still expensive to run but will get cheaper.
stellaathena 3y ago
They’re expensive for you, a random person on the internet, but 5-10M USD is really not prohibitively expensive for a government or large company.

You are absolutely right for the moment, but the question is whether this will always remain the case. I am honestly doubtful and fairly certain this will be more a question of when than if. Seeing as both Nvidia and Apple continue making consumer hardware that does well in inferencing, I fail to see why training sufficiently capable models locally may not become more common place, especially as there are already murmurs in the field that overly large models (100B+) may not be necessary for many use cases. Add to that the fact that institutions (including research institutes[0][1] or universities outside US or EU jurisdictions) are likely to continue undertaking the task of training larger in size models, making the fruits of their labor freely accessible, I just do not see a way for centralized enforcement of specific LLM training rules. [0] https://huggingface.co/tiiuae/falcon-40b-instruct https://huggingface.co/tiiuae/falcon-40b-instruct [1] https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm https://www.databricks.com/blog/2023/04/12/dolly-first-open-...

dkarras 3y ago
Yeah, we will find ways of training these things faster, and we will have faster hardware. We will have better, more parameter efficient models. This will inevitably be something we can do locally. You can already trade time for money. Like, right now you can buy hardware for less than what a house costs, train using the hardware for a year and you basically have something as capable as GPT3.5 - this is already possible and needs to be done only once (and you can still sell the hardware if you want) and a single dedicated person is capable of doing it by themselves. It will only get faster, and cheaper from here. Same goes for inference, still expensive to run but will get cheaper.

stellaathena 3y ago

They’re expensive for you, a random person on the internet, but 5-10M USD is really not prohibitively expensive for a government or large company.