9 ms·

Reading that thread it’s really interesting to me. I see how far we’ve come in a short couple of years. But I still can’t grasp how we’ll achieve AGI within any

Reading that thread it’s really interesting to me. I see how far we’ve come in a short couple of years. But I still can’t grasp how we’ll achieve AGI within any reasonable amount of time. It just seems like we’re missing some really critical… something… Idk. Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot.

jay-barronville 2y ago
When it comes to AI, as a rule, you should assume that whatever has been made public by a company like OpenAI is AT LEAST 6 months behind what they’ve accomplished internally. At least. So yes, the insiders very likely know a thing or two that the rest of us don’t.
- vineyardmike 2y ago
  I understand this argument, but I can't help but feel we're all kidding ourselves assuming that their engineers are really living in the future. The most obvious reason is costs - if it costs many millions to train foundation models, they don't have a ton of experiments sitting around on a shelf waiting to be used. They may only get 1 shot at the base-model training. Sure productization isn't instant, but no one is throwing out that investment or delaying it longer than necessary. I cannot fathom that you can train an LLM at like 1% size/tokens/parameters to experiment on hyper parameters, architecture, etc and have a strong idea on end-performance or marketability. Additionally, I've been part of many product launches - both hyped up big-news-events and unheard of flops. Every time, I'd say that 25-50% of the product is built/polished in the mad rush between press event and launch day. For an ML Model, this might be different, but again see above point. Sure products may be planned month/years out, but OpenAI didn't even know LLMs were going to be this big a deal in May 2022. They had GPT-2 and GPT-3 and thought they were fun toys at that time, and had an idea for a cool tech demo. I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside.
  - HarHarVeryFunny 2y ago
    > I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside. I agree, and they are also living in a group-think bubble of AI/AGI hype. I don't think you'd be too welcome at OpenAI as a developer if you didn't believe they are on the path to AGI.
- ein0p 2y ago
  If they had anything close to AGI, they’d just have it improve itself. Externally this would manifest as layoffs.
  - int_19h 2y ago
    This really doesn't follow. True AGI would be general, but it doesn't necessarily mean that it's smarter than people; especially the kind of people who work as top researchers for OpenAI.
    - ein0p 2y ago
      I don’t see why it wouldn’t be superhuman if there’s any intelligence at all. It already is superhuman at memory and paying attention, image recognition, languages, etc. Add cognition to that and humans basically become pets. Trouble is nobody has a foggiest clue on how to add cognition to any of this.
      - int_19h 2y ago
        It is definitely not superhuman or even above average when it comes to creative problem solving, which is the relevant thing here. This is seemingly something that scales with model size, but if so, any gains here are going to be gradual, not sudden.
        ein0p 2y ago
        I’m actually not so sure they will be gradual. It’ll be like with LLMs themselves where we went from shit to gold in the span of a month when GPT 3.5 came out.
        int_19h 2y ago
        Much of what GPT 3.5 could do was already there with GPT 3. The biggest change was actually the public awareness.
- solidasparagus 2y ago
  But you also have to remember that the pursuit of AGI is a vital story behind things like fundraising, hiring, influencing politicians, being able to leave and raise large amounts of money for your next endeavor, etc. If you've been working on AI, you've seen everything go up and to the right for a while - who really benefits from pointing out that a slowdown is occurring? Who is incentivized to talk about how the benefits from scaling are slowing down or the publicly available internet-scale corpuses are running out? Not anyone who trains models and needs compute, I can tell you that much. And not anyone who has a financial interest in these companies either.
- HarHarVeryFunny 2y ago
  Sure, they know what they are about to release next, and what they plan to work on after that, but they are not clairvoyants and don't know how their plans are going to pan out. What we're going to see over next year seems mostly pretty obvious - a lot of productization (tool use, history, etc), and a lot of efforts with multimodality, synthetic data, and post-training to add knowledge, reduce brittleness, and increase benchmark scores. None of which will do much to advance core intelligence. The major short-term unknown seems to be how these companies will be attempting to improve planning/reasoning, and how successful that will be. OpenAI's Schulman just talked about post-training RL over longer (multi-reasoning steps) time horizons, and another approach is external tree-of-thoughts type scaffolding. These both seem more about maximizing what you can get out of the base model rather than fundamentally extending it's capabilities.
deleted 2y ago
[deleted]
candiddevmike 2y ago
Personally, I think catastrophic global warming and climate change will happen before we get AGI, possibly in part due to the pursuit of AGI. But as the saying goes, yes the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders.
- xpe 2y ago
  Want to share your model? Or is this more like a hunch?
  - fartfeatures 2y ago
    Sounds like standard doomer crap tbh. I'm not sure which is more dangerous at this point - climate change denialism (it isn't happening) or climate change doomerism (we can't stop it, might as well give up)
    - devjab 2y ago
      I’m not sure where you found your information to somehow form that ludicrous last strawman… Climate change is real, you can’t deny it, you can’t debate it. Simply look at the data. What you can debate is the cause… Again a sort of pointless debate if you look at the science. Not even climate change deniers as you call them are necessary saying that we shouldn’t do anything about it. Even big oil is looking into ways to lessen the CO2 in the atmosphere through various means. That being said, the GP you’re talking about made no such statement whatsoever.
      - fartfeatures 2y ago
        Of course climate change is real but of course we can do something about it. My point is denialism and defeatism lead to the same end point. Attack that statement directly if you want to change my mind.
        data_maan 2y ago
otabdeveloper4 2y ago
> But I still can’t grasp how we’ll achieve AGI within any reasonable amount of time. That's easy, we just need to make meatspace people stupider. Seems to be working great so far.
raverbashing 2y ago
> Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot. Honestly? I'm not too worried We've seen how the google employee that was "seeing a conscience" (in what was basically GPT-2 lol) was a nothing burger We've seen other people in "AI Safety" overplay their importance and hype their CV more than actually do any relevant work. (Usually also playing the diversity card) So, no, AI safety is important but I see it attracting the least helpful and resourceful people to the area.
- llamaimperative 2y ago
  I think when you’re jumping to arguments that resolve to “Ilya Sutskever wasn’t doing important work… might’ve played the diversity card,” it’s time to reassess your mental model and inspect it closely for motivated reasoning.
  - raverbashing 2y ago
    Ilya's case is different. He thought the engineers would win in a dispute with Sam at board level. That has proven to be a mistake
    - llamaimperative 2y ago
      And Jan Leike, one of the progenitors of RLHF? What about Geoffrey Hinton? Stuart Russell? Dario Amodei? Also exceptions to your model?
      - raverbashing 2y ago
        https://x.com/ylecun/status/1791850158344249803 https://x.com/ylecun/status/1791850158344249803
        llamaimperative 2y ago
        Another person’s interpretation of another person’s interpretation of another person’s interpretation of Jan’s actions doesn’t even answer the question I asked as it pertains to Jan, never mind the other model violations I listed. I’m pretty sure if Jan came to believe safety research wasn’t needed he would’ve just said that. Instead he said the actual opposite of that. Why don’t you just answer the question? It’s a question about how these datapoints fit into your model.
killerstorm 2y ago
I have a theory why people end up with wildly different estimates... Given the model is probabilistic and does many things in parallel, its output can be understood as a mixture, e.g. 30% trash, 60% rehashed training material, 10% reasoning. People probe model in different ways, they see different results, and they make different conclusions. E.g. somebody who assumes AI should have impeccable logic will find "trash" content (e.g. incorrectly retrieved memory) and will declare that the whole AI thing is overhyped bullshit. Other people might call model a "stochastic parrot" as they recognize it basically just interpolates between parts of the training material. Finally, people who want to probe reasoning capabilities might find it among the trash. E.g. people found that LLMs can evaluate non-trivial Python code as long as it sends intermediate results to output: https://x.com/GrantSlatton/status/1600388425651453953 https://x.com/GrantSlatton/status/1600388425651453953 I interpret "feel the AGI" (Ilya Sutskever slogan, now repeated by Jan Leike) as a focus on these capabilities, rather than on mistakes it makes. E.g. if we go from 0.1% reasoning to 1% reasoning it's a 10x gain in capabilities, while to an outsider it might look like "it's 99% trash". In any case, I'd rather trust intuition of people like Ilya Sutskever and Jan Leike. They aren't trying to sell something, and overhyping the tech is not in their interest. Regarding "missing something really critical", it's obvious that human learning is much more efficient than NN learning. So there's some algorithm people are missing. But is it really required for AGI? And regarding "It cannot reason" - I've seen LLMs doing rather complex stuff which is almost certainly not in the training set, what is it if not reasoning? It's hard to take "it cannot reason" seriously from people
seankurtz 2y ago
Everyone involved in building these things has to have some amount of hubris. Its going to come smashing down on them. What's going unsaid in all of this is just how swiftly the tide has turned against this tech industry attempt to save itself from a downtrend. The whole industry at this point is acting like the tobacco industry back when they first started getting in hot water. No doubt the prophecies about imminent AGI will one day look to our descendents exactly like filters on cigarettes. A weak attempt to prevent imminent regulation and reduced profitability as governments force an out of control industry to deal with the externalities involved in the creation of their products. If it wasn't abundantly clear...I agree with you that AGI is infinitely far away. Its the damage that's going to be caused by sociopaths (Sam Altman at the top of the list) in attempting to justify the real things they want (money) in their march towards that impossible goal that concerns me.
- freehorse 2y ago
  It becoming more and more clear that for "Open"AI the whole "AI-safety/alignment" thing has been a PR-stunt to attract workers, cover the actual current issues with AI (eg stealing data, use for producing cheap junk, hallucinations and societal impact), and build rapport in the AI scene and politics. Now that they have reached a real product and have a strong position in AI development, they could not care less about these things. Those who -naively- believed in the "existential risk" PR stunt and were working on that are now discarded.
iknownthing 2y ago
This may sound harsh but I think some of these researchers have a sort of god complex. Something like "I am so brilliant and what I have created is so powerful that we MUST think about all the horrible things that my brilliant creation can do". Meanwhile what they have created is just a very impressive next token predictor.
- dmd 2y ago
  "Meanwhile what they have created is just a very impressive speeder-up of a lump of lead." "Meanwhile what they have created is just a very impressive hot water bottle that turns a crank." "Meanwhile what they have created is just a very impressive rock where neutrons hit other neutrons." The point isn't how it works, the point is what it does.
  - iknownthing 2y ago
    which is what?
    - CamperBob2 2y ago
      Whatever it is, over the last couple of years it got a lot smarter. Did you?
      - iknownthing 2y ago
        Excellent point CamperBob2
escapecharacter 2y ago
People’s bar for the “I” part is widely varying, many of whom set the bar at “can it make stuff up while appearing confident” Nobody defines what they’re trying to do as “useful AI” since that’s a much more weasily target, isn’t it?

jay-barronville 2y ago

When it comes to AI, as a rule, you should assume that whatever has been made public by a company like OpenAI is AT LEAST 6 months behind what they’ve accomplished internally. At least. So yes, the insiders very likely know a thing or two that the rest of us don’t.

vineyardmike 2y ago
I understand this argument, but I can't help but feel we're all kidding ourselves assuming that their engineers are really living in the future. The most obvious reason is costs - if it costs many millions to train foundation models, they don't have a ton of experiments sitting around on a shelf waiting to be used. They may only get 1 shot at the base-model training. Sure productization isn't instant, but no one is throwing out that investment or delaying it longer than necessary. I cannot fathom that you can train an LLM at like 1% size/tokens/parameters to experiment on hyper parameters, architecture, etc and have a strong idea on end-performance or marketability. Additionally, I've been part of many product launches - both hyped up big-news-events and unheard of flops. Every time, I'd say that 25-50% of the product is built/polished in the mad rush between press event and launch day. For an ML Model, this might be different, but again see above point. Sure products may be planned month/years out, but OpenAI didn't even know LLMs were going to be this big a deal in May 2022. They had GPT-2 and GPT-3 and thought they were fun toys at that time, and had an idea for a cool tech demo. I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside.
- HarHarVeryFunny 2y ago
  > I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside. I agree, and they are also living in a group-think bubble of AI/AGI hype. I don't think you'd be too welcome at OpenAI as a developer if you didn't believe they are on the path to AGI.
ein0p 2y ago
If they had anything close to AGI, they’d just have it improve itself. Externally this would manifest as layoffs.
- int_19h 2y ago
  This really doesn't follow. True AGI would be general, but it doesn't necessarily mean that it's smarter than people; especially the kind of people who work as top researchers for OpenAI.
  - ein0p 2y ago
    I don’t see why it wouldn’t be superhuman if there’s any intelligence at all. It already is superhuman at memory and paying attention, image recognition, languages, etc. Add cognition to that and humans basically become pets. Trouble is nobody has a foggiest clue on how to add cognition to any of this.
    - int_19h 2y ago
      It is definitely not superhuman or even above average when it comes to creative problem solving, which is the relevant thing here. This is seemingly something that scales with model size, but if so, any gains here are going to be gradual, not sudden.
      - ein0p 2y ago
        I’m actually not so sure they will be gradual. It’ll be like with LLMs themselves where we went from shit to gold in the span of a month when GPT 3.5 came out.
        int_19h 2y ago
        Much of what GPT 3.5 could do was already there with GPT 3. The biggest change was actually the public awareness.
solidasparagus 2y ago
But you also have to remember that the pursuit of AGI is a vital story behind things like fundraising, hiring, influencing politicians, being able to leave and raise large amounts of money for your next endeavor, etc. If you've been working on AI, you've seen everything go up and to the right for a while - who really benefits from pointing out that a slowdown is occurring? Who is incentivized to talk about how the benefits from scaling are slowing down or the publicly available internet-scale corpuses are running out? Not anyone who trains models and needs compute, I can tell you that much. And not anyone who has a financial interest in these companies either.
HarHarVeryFunny 2y ago
Sure, they know what they are about to release next, and what they plan to work on after that, but they are not clairvoyants and don't know how their plans are going to pan out. What we're going to see over next year seems mostly pretty obvious - a lot of productization (tool use, history, etc), and a lot of efforts with multimodality, synthetic data, and post-training to add knowledge, reduce brittleness, and increase benchmark scores. None of which will do much to advance core intelligence. The major short-term unknown seems to be how these companies will be attempting to improve planning/reasoning, and how successful that will be. OpenAI's Schulman just talked about post-training RL over longer (multi-reasoning steps) time horizons, and another approach is external tree-of-thoughts type scaffolding. These both seem more about maximizing what you can get out of the base model rather than fundamentally extending it's capabilities.

deleted 2y ago

[deleted]

candiddevmike 2y ago

Personally, I think catastrophic global warming and climate change will happen before we get AGI, possibly in part due to the pursuit of AGI. But as the saying goes, yes the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders.

xpe 2y ago
Want to share your model? Or is this more like a hunch?
- fartfeatures 2y ago
  Sounds like standard doomer crap tbh. I'm not sure which is more dangerous at this point - climate change denialism (it isn't happening) or climate change doomerism (we can't stop it, might as well give up)
  - devjab 2y ago
    I’m not sure where you found your information to somehow form that ludicrous last strawman… Climate change is real, you can’t deny it, you can’t debate it. Simply look at the data. What you can debate is the cause… Again a sort of pointless debate if you look at the science. Not even climate change deniers as you call them are necessary saying that we shouldn’t do anything about it. Even big oil is looking into ways to lessen the CO2 in the atmosphere through various means. That being said, the GP you’re talking about made no such statement whatsoever.
    - fartfeatures 2y ago
      Of course climate change is real but of course we can do something about it. My point is denialism and defeatism lead to the same end point. Attack that statement directly if you want to change my mind.
      - data_maan 2y ago

otabdeveloper4 2y ago

> But I still can’t grasp how we’ll achieve AGI within any reasonable amount of time. That's easy, we just need to make meatspace people stupider. Seems to be working great so far.

raverbashing 2y ago

> Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot. Honestly? I'm not too worried We've seen how the google employee that was "seeing a conscience" (in what was basically GPT-2 lol) was a nothing burger We've seen other people in "AI Safety" overplay their importance and hype their CV more than actually do any relevant work. (Usually also playing the diversity card) So, no, AI safety is important but I see it attracting the least helpful and resourceful people to the area.

llamaimperative 2y ago
I think when you’re jumping to arguments that resolve to “Ilya Sutskever wasn’t doing important work… might’ve played the diversity card,” it’s time to reassess your mental model and inspect it closely for motivated reasoning.
- raverbashing 2y ago
  Ilya's case is different. He thought the engineers would win in a dispute with Sam at board level. That has proven to be a mistake
  - llamaimperative 2y ago
    And Jan Leike, one of the progenitors of RLHF? What about Geoffrey Hinton? Stuart Russell? Dario Amodei? Also exceptions to your model?
    - raverbashing 2y ago
      https://x.com/ylecun/status/1791850158344249803 https://x.com/ylecun/status/1791850158344249803
      - llamaimperative 2y ago
        Another person’s interpretation of another person’s interpretation of another person’s interpretation of Jan’s actions doesn’t even answer the question I asked as it pertains to Jan, never mind the other model violations I listed. I’m pretty sure if Jan came to believe safety research wasn’t needed he would’ve just said that. Instead he said the actual opposite of that. Why don’t you just answer the question? It’s a question about how these datapoints fit into your model.

killerstorm 2y ago

I have a theory why people end up with wildly different estimates... Given the model is probabilistic and does many things in parallel, its output can be understood as a mixture, e.g. 30% trash, 60% rehashed training material, 10% reasoning. People probe model in different ways, they see different results, and they make different conclusions. E.g. somebody who assumes AI should have impeccable logic will find "trash" content (e.g. incorrectly retrieved memory) and will declare that the whole AI thing is overhyped bullshit. Other people might call model a "stochastic parrot" as they recognize it basically just interpolates between parts of the training material. Finally, people who want to probe reasoning capabilities might find it among the trash. E.g. people found that LLMs can evaluate non-trivial Python code as long as it sends intermediate results to output: https://x.com/GrantSlatton/status/1600388425651453953 https://x.com/GrantSlatton/status/1600388425651453953 I interpret "feel the AGI" (Ilya Sutskever slogan, now repeated by Jan Leike) as a focus on these capabilities, rather than on mistakes it makes. E.g. if we go from 0.1% reasoning to 1% reasoning it's a 10x gain in capabilities, while to an outsider it might look like "it's 99% trash". In any case, I'd rather trust intuition of people like Ilya Sutskever and Jan Leike. They aren't trying to sell something, and overhyping the tech is not in their interest. Regarding "missing something really critical", it's obvious that human learning is much more efficient than NN learning. So there's some algorithm people are missing. But is it really required for AGI? And regarding "It cannot reason" - I've seen LLMs doing rather complex stuff which is almost certainly not in the training set, what is it if not reasoning? It's hard to take "it cannot reason" seriously from people

seankurtz 2y ago

Everyone involved in building these things has to have some amount of hubris. Its going to come smashing down on them. What's going unsaid in all of this is just how swiftly the tide has turned against this tech industry attempt to save itself from a downtrend. The whole industry at this point is acting like the tobacco industry back when they first started getting in hot water. No doubt the prophecies about imminent AGI will one day look to our descendents exactly like filters on cigarettes. A weak attempt to prevent imminent regulation and reduced profitability as governments force an out of control industry to deal with the externalities involved in the creation of their products. If it wasn't abundantly clear...I agree with you that AGI is infinitely far away. Its the damage that's going to be caused by sociopaths (Sam Altman at the top of the list) in attempting to justify the real things they want (money) in their march towards that impossible goal that concerns me.

freehorse 2y ago
It becoming more and more clear that for "Open"AI the whole "AI-safety/alignment" thing has been a PR-stunt to attract workers, cover the actual current issues with AI (eg stealing data, use for producing cheap junk, hallucinations and societal impact), and build rapport in the AI scene and politics. Now that they have reached a real product and have a strong position in AI development, they could not care less about these things. Those who -naively- believed in the "existential risk" PR stunt and were working on that are now discarded.

iknownthing 2y ago

This may sound harsh but I think some of these researchers have a sort of god complex. Something like "I am so brilliant and what I have created is so powerful that we MUST think about all the horrible things that my brilliant creation can do". Meanwhile what they have created is just a very impressive next token predictor.

dmd 2y ago
"Meanwhile what they have created is just a very impressive speeder-up of a lump of lead." "Meanwhile what they have created is just a very impressive hot water bottle that turns a crank." "Meanwhile what they have created is just a very impressive rock where neutrons hit other neutrons." The point isn't how it works, the point is what it does.
- iknownthing 2y ago
  which is what?
  - CamperBob2 2y ago
    Whatever it is, over the last couple of years it got a lot smarter. Did you?
    - iknownthing 2y ago
      Excellent point CamperBob2

escapecharacter 2y ago

People’s bar for the “I” part is widely varying, many of whom set the bar at “can it make stuff up while appearing confident” Nobody defines what they’re trying to do as “useful AI” since that’s a much more weasily target, isn’t it?