6 ms·

It's interesting that in order for his pitch to work (if you invest in OpenAI, you will get up to 100x returns), assuming they do build AGI, it still requires t

It's interesting that in order for his pitch to work (if you invest in OpenAI, you will get up to 100x returns), assuming they do build AGI, it still requires that their AGI acquires a very stable, virtually guaranteed advantage of large magnitude. This very strongly requires that they cannot share anything they discover whatsoever. Especially since they apparently plan on using it to make strategic investments to beat the market by a huge margin. That would mean they obtain information (about the economy, world affairs, technology, the future, etc.) not possessed by anyone else, or that information would be reflected in the market already. Any information leakage, whether regarding their AI or whatever it learns about the world, would compromise that advantage. In other words, what Altman says about "we can't only let one group of investors have that" can't be true, or at least not sincere. The more investors who have access to it, the more its returns get distrubuted across society more evenly (which would be a good thing, obviously), but lowers the incentive for initial investments. They will want to keep it contained within a small group of investors for as long as possible.

ackbar03 7y ago
If openai ends up becoming some sort of hedge fund I'd be very very disappointed...
idlewords 7y ago
The implication is that Altman belongs to the 'hard takeoff' sect of AI religion that believes a feedback cycle of recursive self-improvement kicks in, so that the first AI to surpass human levels of intelligence is also the last.
- tristanm 7y ago
  You think so? I think if he thought hard takeoff was a real risk, he would be devoted to actually making their research "open", in the original sense of the word when Musk was in charge. According to the hard takeoff theory, it is more likely to occur due to "hardware overhangs", where big organizations accrue lots of compute infrastructure over time, but AI progress occurs in sudden jumps. Then all that infrastructure is just sitting there waiting to be eaten up by a hungry AI. These sudden jumps are more likely to occur if AI research is generally undertaken in a secretive manner, with leaks or espionage resulting in staggered spread of knowledge. This is at least my understanding.
  - idlewords 7y ago
    I'm not up on all the epicycles in hard takeoff theory, so I'd be happy to hear more from you or other people who have followed it into more fantastical territory.
deleted 7y ago
[deleted]
Leary 7y ago
Or AGI would provide so much benefit that it does not need to be exclusive to OpenAI for the company to accrue significant economic returns.
- tristanm 7y ago
  Yes. This is in the scenario where the benefits are widely distributed. But in that case the value of the entire planet goes up; You would not necessarily need to make an investment in OpenAI in order to reap benefits. Unless you think that scenario is more likely to happen with OpenAI and not via the usual academic research + shared industry research (which I don't see why that would be the case).
  - NhanH 7y ago
    Something along this line: Google increases the entire value of the whole internet by a lot, but you also benefit more by investing in it in the early day.
    - tristanm 7y ago
      In the normal case of one innovation in the free market, there is a punctuated moment of growth followed by a plateau. The owners of the innovation get the biggest share of the rewards, but after the plateau the benefits become more distributed. They would need to keep innovating if they want to gain further. In the case of AGI, there is no plateau after the initial acceleration. With recursive self improvement, there is just an exponential explosion of growth. Unless this event was initiated such that distributed rewards were specifically intended to occur, the feedback loop would prevent external pressures from incentivizing any kind of distribution of rewards.
    - eli_gottlieb 7y ago
      So you're saying that OpenAI is producing a public good in which the whole of society has a stake?
not_a_moth 7y ago
Yeah, there's a big assumptions about the nature of an AGI breakthrough, mainly that it will be a snowball of run away value. Why assume this? Is it because we think AlphaGo/Zero can produce human-like cognition? Why wouldn't it be a long, incremental process of X thousand small breakthroughs, over say a decade, where the result is something like average human-level intelligence; maybe the most important invention of our time, but not super intelligence, and not "run away". (Then after another X years (or decades) you might figure out super intelligence, if regulations haven't intervened by then) If the trajectory is incremental as described, it seems untenable that OpenAI could keep some major monopolistic advantage on AGI, without being completely un-open/sealed off for decade(s).
- rytill 7y ago
  Going from 0 human-intelligence level pieces of software to 1 is the hardest part. Once you have 1, you can duplicate it as much as you want given resources. It can also be pointed inward to improve its own effectiveness. Actually, there are a lot of good arguments for logistic growth. The only ones for linear or sublinear I’ve heard are not strong and mostly take as an implicit assumption “those alarmists and their exponential growth! They probably didn’t even consider that it could be a slower, more incremental growth” instead of actual fully-fledged arguments. There’s also a meta-argument that I have yet to hear reproduced in anti-alarmist sentiments. Which case demands more attention, if it does happen? If there’s a 5% chance of the growth being exponential, how much attention should we devote to that case, where the impact is much higher than linear or sublinear growth. This is such a big deal - it’s like Pascal’s wager but with a real occurrence that I believe most would admit has at least a small chance of happening. Apologies for any brashness coming across. I’m still figuring out how to communicate effectively about a thing I feel a lot of emotions when thinking about.
  - not_a_moth 7y ago
    I didn't say linear, but devil's advocate, isn't that roughly how humans learn? We start out as "pre-intelligent" little creatures who, slowly, methodically, with help of others, develop aptitudes and learn about the world. Learning in fact continues in this manner your entire life should you continue... slow, incremental progress requiring teachers, peers, trials/ errors, crises, 1/3 of your life being unconscious in sleep, etc., in the absence of which no learning at all may happen ... and the bot may potentially have greater computation constraints than humans under current technologies, as the brain is far more efficient than any computer today. I'm not convinced that each of the arcs, elementary intelligence --> average intelligence --> super intelligence, wouldn't be painstaking and roughly linear.

ackbar03 7y ago

If openai ends up becoming some sort of hedge fund I'd be very very disappointed...

idlewords 7y ago

The implication is that Altman belongs to the 'hard takeoff' sect of AI religion that believes a feedback cycle of recursive self-improvement kicks in, so that the first AI to surpass human levels of intelligence is also the last.

tristanm 7y ago
You think so? I think if he thought hard takeoff was a real risk, he would be devoted to actually making their research "open", in the original sense of the word when Musk was in charge. According to the hard takeoff theory, it is more likely to occur due to "hardware overhangs", where big organizations accrue lots of compute infrastructure over time, but AI progress occurs in sudden jumps. Then all that infrastructure is just sitting there waiting to be eaten up by a hungry AI. These sudden jumps are more likely to occur if AI research is generally undertaken in a secretive manner, with leaks or espionage resulting in staggered spread of knowledge. This is at least my understanding.
- idlewords 7y ago
  I'm not up on all the epicycles in hard takeoff theory, so I'd be happy to hear more from you or other people who have followed it into more fantastical territory.

deleted 7y ago

[deleted]

Leary 7y ago

Or AGI would provide so much benefit that it does not need to be exclusive to OpenAI for the company to accrue significant economic returns.

tristanm 7y ago
Yes. This is in the scenario where the benefits are widely distributed. But in that case the value of the entire planet goes up; You would not necessarily need to make an investment in OpenAI in order to reap benefits. Unless you think that scenario is more likely to happen with OpenAI and not via the usual academic research + shared industry research (which I don't see why that would be the case).
- NhanH 7y ago
  Something along this line: Google increases the entire value of the whole internet by a lot, but you also benefit more by investing in it in the early day.
  - tristanm 7y ago
    In the normal case of one innovation in the free market, there is a punctuated moment of growth followed by a plateau. The owners of the innovation get the biggest share of the rewards, but after the plateau the benefits become more distributed. They would need to keep innovating if they want to gain further. In the case of AGI, there is no plateau after the initial acceleration. With recursive self improvement, there is just an exponential explosion of growth. Unless this event was initiated such that distributed rewards were specifically intended to occur, the feedback loop would prevent external pressures from incentivizing any kind of distribution of rewards.
  - eli_gottlieb 7y ago
    So you're saying that OpenAI is producing a public good in which the whole of society has a stake?

not_a_moth 7y ago

Yeah, there's a big assumptions about the nature of an AGI breakthrough, mainly that it will be a snowball of run away value. Why assume this? Is it because we think AlphaGo/Zero can produce human-like cognition? Why wouldn't it be a long, incremental process of X thousand small breakthroughs, over say a decade, where the result is something like average human-level intelligence; maybe the most important invention of our time, but not super intelligence, and not "run away". (Then after another X years (or decades) you might figure out super intelligence, if regulations haven't intervened by then) If the trajectory is incremental as described, it seems untenable that OpenAI could keep some major monopolistic advantage on AGI, without being completely un-open/sealed off for decade(s).

rytill 7y ago
Going from 0 human-intelligence level pieces of software to 1 is the hardest part. Once you have 1, you can duplicate it as much as you want given resources. It can also be pointed inward to improve its own effectiveness. Actually, there are a lot of good arguments for logistic growth. The only ones for linear or sublinear I’ve heard are not strong and mostly take as an implicit assumption “those alarmists and their exponential growth! They probably didn’t even consider that it could be a slower, more incremental growth” instead of actual fully-fledged arguments. There’s also a meta-argument that I have yet to hear reproduced in anti-alarmist sentiments. Which case demands more attention, if it does happen? If there’s a 5% chance of the growth being exponential, how much attention should we devote to that case, where the impact is much higher than linear or sublinear growth. This is such a big deal - it’s like Pascal’s wager but with a real occurrence that I believe most would admit has at least a small chance of happening. Apologies for any brashness coming across. I’m still figuring out how to communicate effectively about a thing I feel a lot of emotions when thinking about.
- not_a_moth 7y ago
  I didn't say linear, but devil's advocate, isn't that roughly how humans learn? We start out as "pre-intelligent" little creatures who, slowly, methodically, with help of others, develop aptitudes and learn about the world. Learning in fact continues in this manner your entire life should you continue... slow, incremental progress requiring teachers, peers, trials/ errors, crises, 1/3 of your life being unconscious in sleep, etc., in the absence of which no learning at all may happen ... and the bot may potentially have greater computation constraints than humans under current technologies, as the brain is far more efficient than any computer today. I'm not convinced that each of the arcs, elementary intelligence --> average intelligence --> super intelligence, wouldn't be painstaking and roughly linear.