5 ms·
We've only had LLMs for seven years, and it's all basically all refinements of the transformer with minor additions. I think there are paths to substantial arc
by impossiblefork 1y ago
We've only had LLMs for seven years, and it's all basically all refinements of the transformer with minor additions.
I think there are paths to substantial architecture improvements-- things resembling edge transformers, things like adding extra tokens sort of like how that is done in vision transformers etc.
For this reason I think employment doomerism isn't very unreasonable. The architectures aren't here-- we simply haven't figured out how to substantially improve the transformer, but it's surely possible.
I myself made experiments recently where I got improved performance on some demanding toy problems. There must be hundreds of people working on this kind of thing, and even if there aren't the general research is building knowledge that will eventually guide people to design better architectures.
- iLoveOncall 1y agoI for one cannot wait to see how ChatGPT-17.3 with Prime Edge Transformers Optimum++ will take over my job and come kill me in my house by... generating slightly less inaccurate text?
- impossiblefork 1y agoSo, what if future models could actually program and do mathematics like the best professionals and experts? A world where you can ask this ChatGPT-17.3 to generate a version control tool with these or that properties, solving some list of problems with previous version control tools, and then it does so over the weekend at the cost of $1000 in GPU rent? Presumably at the point it will not be sensible to employ very many programmers. A couple of experts would determine requirements and then they would feed that into the machines and the required software, mathematics etc. would be produced, analyzed by a small group of experts and then used to produce requirements for the next version.
- iLoveOncall 1y agoSo, what if this never happens?
- impossiblefork 1y agoWell, then it doesn't, but why would you expect that to be the case? Why would transformers be so ideal that there is no major improvement left?
- iLoveOncall 1y agoWhy would you expect otherwise? You're the one with an extraordinary claim. I'm not saying that transformers are ideal or that there's no major improvement left. I'm saying that a major improvement over something that is barely a net positive is nowhere near the extinction level event you're making it seem to be.
- lostmsu 1y agoIt's not like the proof to the claim is not extraordinary. 4GB of weights demonstrates reasoning abilities unthinkable 4 years ago.
- iLoveOncall 1y agoNext token generation is not reasoning.
- lostmsu 1y agoI would argue that is more reasoning than you do in the specific instances of SotA LLMs. Because you are demonstrating a rather simple logical mistake by claiming "T is in NTG => T is not in R" when you have no information about relationship between NTG and R sets. But I wasn't even talking about reasoning. I specifically said "demonstration".
- palmotea 1y ago>> So, what if future models could actually program and do mathematics like the best professionals and experts? > So, what if this never happens? We can hope they fail, but the goal of these people are working towards is the replacement of humans for economically valuable intellectual work. We should take those goals seriously.