4 ms·
exponential curves tend to feel linear early and obviously non-linear in hindsight. add to that extreme dependence on starting conditions and you get a perfect
by baq 2y ago
exponential curves tend to feel linear early and obviously non-linear in hindsight. add to that extreme dependence on starting conditions and you get a perfect mix of incompatibility what human psychology.
this is why it's all so scary. almost nobody believes it'll happen until it's basically already happened or can't be stopped.
- XorNot 2y agoAnd sigmoidal curves feel exponential at the start, then linear. I see little evidence we're headed towards an exponential regime: the cost and resource usage versus capability hasn't been acting that way.
- baq 2y agoOTOH we know there are multiple orders of magnitude of efficiency to gain: the brain does what it does at 20W. Forecasting where the inflection point is on the sigmoid is as hard as everything else.
- XorNot 2y agoWhile true, I'd argue that's worse for current progress: the overall trend has been to throw mammoth amounts of electricity at the problem via compute. In so much as something like DeepSeek proves there's gains to be made, the current growth in resource usage would speak against declaring it proves improvement soon. Like plug that issue into known hardware limitations due to physical limits (i.e. we're already up against the wall re: feature sizes) and I'm even more skeptical. If we were looking at a buy down in training resources which was accelerating, then I'd be a lot more interested.