3 ms·

AI seems to progress in bursts, and I think we're in the middle of one, but it may be naive to think the progress will continue at the same pace for another 4 y

by audunw 4y ago

AI seems to progress in bursts, and I think we're in the middle of one, but it may be naive to think the progress will continue at the same pace for another 4 years. When Big Blue beat Kasparov in Chess in 1997, I wonder if anyone would've guessed that it'd take almost 20 years for a computer to beat a master in Go. IBM Watson was launched in 2010 and had many of the same promises as GPT. It supposedly fell flat in many cases in the real world. I think GPT and other models of the same level can succeed commercially on the same tasks within the next 1-4 years, but that shows it can easily be a decade from some kind of demonstration to actual game changing applications.

simonster 4y ago
Given the field's record of AI winters, it would be naive to think progress will certainly continue, but given the amount of progress that has been made as well as how it's being made, it would also be naive to think it will certainly not. The advances that have come in the last few years have been driven first and foremost by compute and secondarily by methodology. The compute can continue to scale for another couple orders of magnitude. It's possible that we'll be bottlenecked by methodology; there are certain things that current networks are simply incapable of, like learning from instructions and incorporating that knowledge into their weights. That said, one of the amazing things about recent successes is that the precise methodology doesn't seem to matter so much. Diffusion is great, but autoregressive image generation models like Parti also generate nice images, albeit at a higher computational cost. RL from human feedback achieves impressive results, but chain of hindsight (supposedly) achieves similar results without RL. It's entirely plausible to me that the remaining challenges on the path to AGI can be solved by obvious ideas + engineering + scaling + data from the internet. We've also gotten to the point where AI systems can make substantial contributions to engineering more powerful AI systems, and maybe soon, to ideation. We haven't yet figured out how to extract all of the productivity gains from the systems we already have, and next-generation systems will provide larger productivity gains, even if they are just scaled up versions of current-generation systems.
MattRix 4y ago
Even if the technology froze at GPT-4 and never advanced, it would still be enough to change the world in all kinds of ways. The fact that the tech is still advancing as well is huge. Also now you’re seeing tons of solo devs, startups, and large corporations all coming up with new ways to use AI. This “burst” is not like the others you mentioned.
- TrapLord_Rhodo 4y ago
  Exactly this. This is a different 'Leap' than the ones before it. It's a leap with an API. Now hundreds of thousands of company's can fine tune it and train it on their specific business task. parroting your point, it will take years for the true fecundity of the technology in chat GPT 4 to be fully fleshed out.
deleted 4y ago
[deleted]
deeviant 4y ago
> AI seems to progress in bursts Historically yes. Today, no way. It's a sprint and it's not slowing down.
- scotty79 4y ago
  It's fueled by raw gpus and servers and pretty much nothing else. GPT is pretty much perceptron with some places hardcoded. Resources bound to run out at some point.

by audunw 4y ago

simonster 4y ago

Given the field's record of AI winters, it would be naive to think progress will certainly continue, but given the amount of progress that has been made as well as how it's being made, it would also be naive to think it will certainly not. The advances that have come in the last few years have been driven first and foremost by compute and secondarily by methodology. The compute can continue to scale for another couple orders of magnitude. It's possible that we'll be bottlenecked by methodology; there are certain things that current networks are simply incapable of, like learning from instructions and incorporating that knowledge into their weights. That said, one of the amazing things about recent successes is that the precise methodology doesn't seem to matter so much. Diffusion is great, but autoregressive image generation models like Parti also generate nice images, albeit at a higher computational cost. RL from human feedback achieves impressive results, but chain of hindsight (supposedly) achieves similar results without RL. It's entirely plausible to me that the remaining challenges on the path to AGI can be solved by obvious ideas + engineering + scaling + data from the internet. We've also gotten to the point where AI systems can make substantial contributions to engineering more powerful AI systems, and maybe soon, to ideation. We haven't yet figured out how to extract all of the productivity gains from the systems we already have, and next-generation systems will provide larger productivity gains, even if they are just scaled up versions of current-generation systems.

MattRix 4y ago

Even if the technology froze at GPT-4 and never advanced, it would still be enough to change the world in all kinds of ways. The fact that the tech is still advancing as well is huge. Also now you’re seeing tons of solo devs, startups, and large corporations all coming up with new ways to use AI. This “burst” is not like the others you mentioned.

TrapLord_Rhodo 4y ago
Exactly this. This is a different 'Leap' than the ones before it. It's a leap with an API. Now hundreds of thousands of company's can fine tune it and train it on their specific business task. parroting your point, it will take years for the true fecundity of the technology in chat GPT 4 to be fully fleshed out.

deleted 4y ago

[deleted]

deeviant 4y ago

> AI seems to progress in bursts Historically yes. Today, no way. It's a sprint and it's not slowing down.

scotty79 4y ago
It's fueled by raw gpus and servers and pretty much nothing else. GPT is pretty much perceptron with some places hardcoded. Resources bound to run out at some point.