3 ms·

There has been some effort to quantify "AI scaling laws": ie how much performance increases as we scale up the resources involved. See section 1.2 of https://ar

by kmod 5y ago

There has been some effort to quantify "AI scaling laws": ie how much performance increases as we scale up the resources involved. See section 1.2 of https://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/2001.08361.pdf My main takeaway from that paper is that a 2x increase in training cost improves performance by 5% (100x by 42%). I only skimmed the paper though. To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two. I talked to someone working on model scaling and they see the same numbers and draw a very different conclusion: my interpretation of their argument is that they view scaling money as easy versus finding new fundamental technical advances.

ShamelessC 5y ago
> my interpretation of their argument is that they view scaling money as easy versus finding new fundamental technical advances. This paper only discussed transformers. Given the current pace of research, it's not a given that transformers won't be replaced by something else that has better scaling laws. And indeed, this paper tells you the amount of compute and data you need for a transformer. But Moore's law isn't doing great lately. For the purposes of research,you may be able to train trillion-parameter models - but you will likely not run such a model on your phone. In order to merit the large cost of both training and running predictions (which don't even fit the entire model in a single GPU for larger models) - models will need to become more parameter efficient than the vanilla transformer. Otherwise it's just too expensive.
baryphonic 5y ago
I hadn't seen that paper before, but it is excellent. Thank you for sharing! > To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two. I totally agree with this assessment, and note the absence of a GPT-4 release.

by kmod 5y ago

ShamelessC 5y ago

> my interpretation of their argument is that they view scaling money as easy versus finding new fundamental technical advances. This paper only discussed transformers. Given the current pace of research, it's not a given that transformers won't be replaced by something else that has better scaling laws. And indeed, this paper tells you the amount of compute and data you need for a transformer. But Moore's law isn't doing great lately. For the purposes of research,you may be able to train trillion-parameter models - but you will likely not run such a model on your phone. In order to merit the large cost of both training and running predictions (which don't even fit the entire model in a single GPU for larger models) - models will need to become more parameter efficient than the vanilla transformer. Otherwise it's just too expensive.

baryphonic 5y ago

I hadn't seen that paper before, but it is excellent. Thank you for sharing! > To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two. I totally agree with this assessment, and note the absence of a GPT-4 release.