4 ms·

We wont get to AGI if we dont get models with both larger context and dreaming (aka distilling important parts from their 'day long' context and adding those to

We wont get to AGI if we dont get models with both larger context and dreaming (aka distilling important parts from their 'day long' context and adding those to their neural net) with about the same effort/cost as inference. LLM models cannot do this and wont be able to, so if no one comes up with a better model, AGI cannot be reached, no matter what amounts are invested, so we will get an AI winter. So many smart minds are on this now, that if anyone had an idea how to 'learn during inference', someone would have released something. No one has a clue, so I am betting the downfall will come soon. Still, we got incredible progress from this AI boom, so it is not bad, just money slushing.

stingraycharles 1y ago
Not sure if there will be an AI winter, but the claims of AGI being very close are likely very optimistic. Maybe AGI requires an entirely new paradigm, maybe it requires quantum computing, or biological computing (our brains are able to work at just a fraction of the cost of current LLMs after all), regardless it’s much further out than certain industry leaders would like you to believe. But an AI winter is unlikely to come either, as it currently adds a lot of value in many places. But the profits coming out of it are unlikely to line up with current investments being poured in.
- anonzzzies 1y ago
  The previous winters still had a lot (for the time) of benefits and money coming from the discoveries before the winter, but there were just very few fundamental advancements to AI tech itself. So just using LLMs and optimising them etc but getting nothing new is still a winter to me.
- anonzzzies 1y ago
  Also these industry leaders believe that we can get there just optimising llms: this is a mistake for sure: the only general intelligence we know is human and humans add new facts to their neural nets daily while llms need weeks of expensive training and then have a knowledge cut off. It cannot happen like that.
  - stingraycharles 1y ago
    Yup, it’s as if you fine-tune your model constantly, every interaction triggering backpropagation. Current techniques absolutely are not economically viable to do that, but I can imagine that maybe OpenAI are running some small scale experiments like that. You’d basically need a dedicated model per user (or chat even) + continuous training infrastructure as a prerequisite for AGI, I don’t see that happening in the next decade or without a completely different paradigm, let alone being economically viable.
- Yoric 1y ago
  LLMs add a value that remains orders of magnitude lower than the investments. So I fully expect their valuation to decrease suddenly by end of 2027 – and to take the rest of AI with them into a new AI winter.

stingraycharles 1y ago

Not sure if there will be an AI winter, but the claims of AGI being very close are likely very optimistic. Maybe AGI requires an entirely new paradigm, maybe it requires quantum computing, or biological computing (our brains are able to work at just a fraction of the cost of current LLMs after all), regardless it’s much further out than certain industry leaders would like you to believe. But an AI winter is unlikely to come either, as it currently adds a lot of value in many places. But the profits coming out of it are unlikely to line up with current investments being poured in.

anonzzzies 1y ago
The previous winters still had a lot (for the time) of benefits and money coming from the discoveries before the winter, but there were just very few fundamental advancements to AI tech itself. So just using LLMs and optimising them etc but getting nothing new is still a winter to me.
anonzzzies 1y ago
Also these industry leaders believe that we can get there just optimising llms: this is a mistake for sure: the only general intelligence we know is human and humans add new facts to their neural nets daily while llms need weeks of expensive training and then have a knowledge cut off. It cannot happen like that.
- stingraycharles 1y ago
  Yup, it’s as if you fine-tune your model constantly, every interaction triggering backpropagation. Current techniques absolutely are not economically viable to do that, but I can imagine that maybe OpenAI are running some small scale experiments like that. You’d basically need a dedicated model per user (or chat even) + continuous training infrastructure as a prerequisite for AGI, I don’t see that happening in the next decade or without a completely different paradigm, let alone being economically viable.
Yoric 1y ago
LLMs add a value that remains orders of magnitude lower than the investments. So I fully expect their valuation to decrease suddenly by end of 2027 – and to take the rest of AI with them into a new AI winter.