3 ms·
Maybe don't think of it as curves or functions. Just go through an LLM and think about all the things that could be improved. It's a long list once you get into
by danielmarkbruce 2y ago
Maybe don't think of it as curves or functions. Just go through an LLM and think about all the things that could be improved. It's a long list once you get into the details. By and large...sota models are:
a) trained on crappy data, including questionable RLHF feedback.
b) trained with questionable embedding layers.
c) trained with questionable loss functions
d) trained with questionable optimizers
e) trained at questionable precision (somewhat related to d)
f) are very big which stops fast iteration around all the above.
It's kinda like semiconductors. You don't have to think of it as a curve - just ask people who are really close to them and they'll have a laundry list of stupid stuff which is currently done and will likely be improved upon over time.
- lxgr 2y agoI don't doubt that there's tons of work ahead to even just integrate current-day capabilities of LLMs into our society and economy. But when talking about future growth potential, I don't think you can get around making assumptions about the shape of the growth function.
- danielmarkbruce 2y agoYou can't write a nice article about it. But you can talk about it with a simple "things are going to get a loooot better". The problem with thinking about progress in functions is it suggests some underlying law exists, and there isn't one. Even if someone could point to a function and say "10x better" by 2030 - what does that even mean in the context of an LLM for example?