3 ms·
To show the cost of extra parameters is exponentional, they made a graph with price linear and parameter count logarithmic and then fit an exponential? It seem
by cma 4y ago
To show the cost of extra parameters is exponentional, they made a graph with price linear and parameter count logarithmic and then fit an exponential? It seems very misleading:
https://proofinprogress.com/assets/images/cost-llms.png https://proofinprogress.com/assets/images/cost-llms.png
With linear or polynomial increase per parameter it would also look exponential with that graph setup.
- timdaub 4y agomhh yeah you might have a point. I didn‘t do it on purpose though. I‘ll go through this again
- cma 4y agoMatrix multiply scales polynomially so don't be thrown off by that. You'd also expect some increases in cost where things move from single GPU to multi-GPU to networked machines. I'm still not sure if that pushes things to exponential or would just step up cost at each transition.