3 ms·
Thanks :) > But the question I would ask you is whether you are any barrier to simply scaling up the tokens on the existing technology? I don’t. I see us at th
by ben_w 3y ago
Thanks :)
> But the question I would ask you is whether you are any barrier to simply scaling up the tokens on the existing technology? I don’t. I see us at the beginning of a ladder where the only input is capacity just like when Intel was young.
My expectation is that we need algorithmic improvements rather than scaling; AI can read approximately all of the internet, but current models need to actually do so just to reach the level of intern or fresh graduate. While this makes them superhuman in the breadth of skills they can perform, they need something else to improve the maximum quality in any given skill — in some cases, we can already train them on synthetic data or self-play, e.g. chess, though I don't know how broad an impact that would have.
But I do expect such algorithmic improvements, so in effect we are in agreement, if not in the details of how.
When it comes to hardware improvements, I'm not sure how that particular landscape will change over the next decade. Transistors are close enough to atomic scale they can't go on much longer, and Dennard scaling has long since stopped, but that doesn't mean nobody cares or that nobody is working on the energy efficiency. And if — just if, it isn't necessarily true — if human level intelligence needs a network with as many free parameters as there are synapses in a human brain, we're around 3-4 orders of magnitude away from that at present.
> I mean to invoke Smihula's theory of waves in this argument, since I know you’ll understand that.
Thanks, I was unfamiliar with it: https://en.wikipedia.org/wiki/Smihula_waves https://en.wikipedia.org/wiki/Smihula_waves
> Having worked a lot with large scale networks, agents and task specialized networked AI systems get me excited. My brain considers it a blue ocean opportunity.
I think you're correct. The current zeitgeist is do-everything models, and the only blue ocean opportunities are found when you zig when everyone else is zagging, and vice-versa.
Although, be quick; if my cursory reading of Smihula's theory of waves was correct, you don't have much time before the current market reaches saturation, and moves on to the next thing.
- ukuina 3y agoSmihula's waves, when extrapolated, would indicate a 15-year cycle for the smartphone era (assumed to have "innovation saturated" 2007-2022), and an 8-year cycle for AI (starting 2023). That's a staggering amount of change in too short a period to adjust to.