14 ms·

Governance of Superintelligence

93 points by davidbarker 3y ago

groby_b 3y ago
There is not a single shred of evidence we'll reach superintelligence any time soon. There's however a lot of evidence that regulation benefits incumbents. You can do the math from there.
- mitthrowaway2 3y ago
  What is the evidence you'd expect to see that you aren't seeing, and how many years in advance is too soon to make a plan?
  - groby_b 3y ago
    Any development trajectory that doesn't look very suspiciously like a logistics curve is a good starter. (Pretty much all LLMs I've seen show an exponential increase in resources for a less-than linear increase in capability lately) And if we want to make a plan, how about we don't ask the fox to guard the hen house?
- cubefox 3y ago
  AI progress keeps accelerating more and more --> "not a single shred of evidence"
  - groby_b 3y ago
    Does it? Does it really? Yes, we've seen huge improvements. We're still not close to human intelligence level. (No, test taking doesn't count). Worse, we've got pretty clear evidence that the training material increase for larger LLMs is basically reaching an upper bound. What we'd need to see: * A breakthrough that either decouples us from parameter count or allows parameter count increases with smaller training sets. * Any evidence that it's doing anything more than asymptotically crawling towards human-comparable evidence. The entire ZOMG SUPERAI faction suffers from the AI that somehow thinking more and faster is thinking better. It's not. There's no evidence pointing in that direction. We currently have ~8B human-level intelligences. They haven't managed to produce anything above human level intelligence. Where's the indication that emulation of their mode of thinking at scale will result in something breaking the threshold? If anything, machine intelligence is doing worse, because any slight increase in capacity is paid for in large amounts of "hallucination".
    - cubefox 3y ago
      That's like arguing 10 years ago that ChatGPT was impossible in the next ten years because AlexNet could only recognize objects in photos with some medium reliability, and there was no path to scale CNNs to something like ChatGPT. The mistake of course was assuming we were stuck with CNNs. And we will probably also not keep using LLMs. We already know there are more effective architectures, as animals implement one of them.
      - groby_b 3y ago
        None of the architectures we know imply in any way giant leaps in intelligence. That's the main sticking point. Thinking more does not equate thinking better. Or even thinking well. As for "animals implement them", it's worth noting that we mostly qualify for an award in impressive lack of understanding in that area. Even with exponential improvements, that is not going to change within the next five years. The "but we just don't know" argument is useless. That also applies to aliens landing on this planet next week and capturing the government. Theoretically possible, but not a pressing concern. Should we think about what AI regulations look like? Yes. Should we enact regulations on something that doesn't even really exist, without deeply understanding it, at the behest of the party that stands to gain financially from it? Fuck no.
immichaelwang 3y ago
Is there any downside whatsoever for OpenAI/Sam to be the one proposing/leading the calls for regulation? Cynics will say they are trying to pull the ladder up from underneath them, so this is massively beneficial for them. What's the downside (if any)? Getting a small subset of the community mad doesn't seem like a lot of downside.
- A4ET8a8uTh0 3y ago
  << What's the downside (if any)? Compliance is still a burden. Even if you write the law, eventually the bureaucracy that was willed into existence will start living its own life, imposing new restrictions and basically making your life miserable. Still, the profits from keeping the field restricted to a small circle of largely non-competitors helps to offset that.
- adastra22 3y ago
  Then they’d be engineering their own eventual demise. Any regulatory capture regime ends up stalling progress and bloating incumbents until eventually a nimble competitor is able to circumvent regulators and steal their lunch money.
  - namaria 3y ago
    Humans are temporary beings, we only need temporary wealth.
    - adastra22 3y ago
      If that were true, Sam would never have founded OpenAI. He already had generational wealth.
- mhb 3y ago
  Whether there's a downside is moot since no one knows how to do any sort of regulation effectively. That being said, I don't know why you think that only a small community will see this as self-serving. It's not subtle even though it may be unavoidable.
ablyveiled 3y ago
Security is impossible. Therefore, "AI alignment" is impossible. I'm not sure that even AGI is possible, per Bostrom's Self-Sampling Assumption.
- jw1224 3y ago
  > I'm not sure that even AGI is possible, per Bostrom's Self-Sampling Assumption. Can you explain more? I’ve found a definition for SSA, but unsure how it applies to AGI…
  - ablyveiled 3y ago
    The self-sampling assumption essentially says that you should reason as though your self-sampling is the likely outcome in consciousness-space-time. It's the anthropic principle applied to the distribution/shape of intelligence. Since your self-sample is of a human-shaped mind, human-shaped minds are the most likely outcome. So the probability of AGI is unlikely.
generalizations 3y ago
- Corporate-owned proprietary AIs? Check. - Monitoring of power consumption for illegal computer usage? Check. - Superintelligent AIs under tight supervision? Check. - Bootlegged neural nets passed around on torrenting networks? Check. - Poverty and homelessness running rampant? Check. Folks, we're officially living in a cyberpunk dystopia.
- EGreg 3y ago
  Yup. I have been warning about this for years. And things are going to get much more weird soon. “But Web3 sucks” Hold. The beer.
  - __loam 3y ago
    Web3 sucks regardless of how this shakes out.
    - EGreg 3y ago
      Web3 is positive but close to zero sum game for society, say. While AI can be massively negative sum game for the world.
      - __loam 3y ago
        Web3 is provably negative sum actually.
- schaefer 3y ago
  > - Poverty and homelessness running rampant? Check. Compared to what, exactly? because over the last 50 years, there have been dramatic improvements[1]. [1]: https://www.brookings.edu/research/the-evolution-of-global-poverty-1990-2030/ https://www.brookings.edu/research/the-evolution-of-global-p... It's true - there's room to do better. So, so much better. But discarding the progress of the last 50 years is so unbelievably counter-productive.
  - 3y ago
runsWphotons 3y ago
What is an example of an i dependent AI agent carrying out as much productive activity as a large company that is plausible?
- j45 3y ago
  Lots. Most enterprises processes and algorithms are big mosts around small processes really well understood which are tied together and neurotically checked or sufficiently slowed down to ask “are we sure?” enough times before implementing a change.
- cubefox 3y ago
  An AI that can build robots which build robots.
  - runsWphotons 3y ago
    Is there a prototype of this?
BeenAGoodUser 3y ago
I think it is counterproductive to limit the rate of growth of AI since some countries — typically the ones who are less concerned about ethics — will not follow those rules and could gain a considerable edge over the rest of the world this way.
- generic92034 3y ago
  Or they are the first countries falling to the superintelligent AIs they themselves have developed... :o
  - Jackson__ 3y ago
    ... with the rest of the world following shortly.
- standardUser 3y ago
  Historically, technologically advanced nations have worked together to develop some limited governance over sensitive technologies. It's an imperfect system, but tends to be better than a free-for-all.
  - cubefox 3y ago
    Counterexample: Atom bombs, ICMBs, spy satellites.
    - standardUser 3y ago
      You literally just named the actual examples.
      - cubefox 3y ago
        MAD is the opposite of cooperation.
        3y ago
abeppu 3y ago
> Second, we are likely to eventually need something like an IAEA for superintelligence efforts > Third, we need the technical capability to make a superintelligence safe. This is an open research question that we and others are putting a lot of effort into. I think it's important to remember that the IAEA was not proposed until 1954 and didn't get approved by the UN until 1957. By that time we had already gotten past the first fission reaction, fission weapon testing and uses, nuclear submarine and grid-connected power generation. We were trying to control something we already knew how to build, and about which we already understood a bunch of specific risks. And this is for a technology which (a) requires access to rare materials and (b) there's a pretty definite line that shows when fission is happening, even briefly. It's a different order of challenge to try to regulate the development of something we _don't_ yet understand how to build, which might be possible to build with computers which are everywhere, and where we don't have a clear mechanism to detect or identify the thing we're trying to control, let alone to decide whether or not it's "safe".
Animats 3y ago
After Putin, Xi, Trump, and Boris Johnson, we should give the AIs a chance.
sidcool 3y ago
What would superintelligence look like? What is the worst case scenario?
causi 3y ago
I wonder if AI will ever scale down to personal hardware, or if every step from here forward is going to rely on an internet connection forever. I don't like the idea of never having real control.
- Animats 3y ago
  > I wonder if AI will ever scale down to personal hardware Happened last week. Download here.[1] https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all
jonathankoren 3y ago
Interesting that they’re telling us to focus on dubious sci-fi movie tropes instead of actual dangers and risks documented today. But yeah, Skynet is who we need to watch out for, not biased AIs making decisions today.
- cubefox 3y ago
  Biased AI today is extremely unimportant compared to AI posing an existential risk in the future. It's like saying: "Who cares about the hypothetical effects climate change allegedly has in the far future? Let's focus on the effects that the local highway has on our frog population today."
  - jonathankoren 3y ago
    I’ll do one better. The existential risk of an AI running amok and enslaving or exterminating all of humanity is based on series of low probability events that we can’t even put error bars on, let alone understand what would be needed to even achieve such a feat. HOWEVER, the existential risk posed by the sun running out of hydrogen and destroying all life on planet Earth has known a probability (1.0), with known error bars around the best estimation of when that will happen. Furthermore, the problems involved with moving masses through space are understood quite well. As such, this is merely an engineering problem, rather than a problem of inadequate theories and philosophies. Therefore, it is undeniably logical that we must immediately redirect all industrial and scientific output to building a giant engine to move the Earth to another star with a longer lifespan, and any argument otherwise is an intentional ploy to make humanity go extinct. Any sacrifices made by the current and near future generations is a worthy price to pay for untold sextillions of humanity and its evolution all descendants that would be condemned to certain death by starvation if we did not immediately start building the Earth Engine. This is the only logical, mathematical provable, and morally correct answer.
    - cubefox 3y ago
      Series of low probability events? If a superintelligent AI has goals that differ from ours, then too bad for us. As Altman & Co say, the alignment problem is unsolved.
sylware 3y ago
Why would a big tech company be stoupid enough to build a massive and super fast neural net (at least on the human brain scale) with real time complex outputs/inputs and which would be trained (interacted) continuously for years? You know there is a big chance "it" could become self-aware, don't you? Oh! And "shutting down"/"resetting" such self-aware being would be murder. In theory, humanity would build such neural net if it was facing a certain extinction event and that "close" in time which human science/maths had no answer to. It might give us beyond human science/maths which could save humanity: Humanity: "Super AI? How could we save ourself from our soon to happen doomsday?" Super AI: "42"
tsunamifury 3y ago
I'm sorry, but Sam and OpenAI are massively overhyping the scary AI apocalypse vision of the future to get regulation in order to cover the "We store content from the planet and now want the exclusive rights to monetize it." I've been in Silicon Valley for many years, and this takes the cake of the most cynical, in your face, billionaires should steal sheep from their poor neighbors level of run away greed and sociopathy I have ever witnessed.
- j45 3y ago
  It’s Super Intelligent AI for me, but not for thee. It’s super intelligent AI for the few and not the many. Would demanding safety suddenly make a little more sense if a super intelligent GPT5 already exists? If someone has it they want to keep that advantage.
r_lee 3y ago
Isn't Sam Altman also developing a crypto project which collects your iris data? I mean, I don't understand what is up with their obsession with this AI Governance, it seems like it is indeed about retaining control, so that only the big guys can [attempt] to control any larger models. Same with him proposing such "AI Licenses". What a time to be alive...
- deleted 3y ago
  [deleted]
amelius 3y ago
Can't we just ask the AI how we should manage the AI?
lsy 3y ago
Nothing makes me think of Altman as a grifter more than his trying to spook uneducated lawmakers with sci-fi notions like "superintelligence" for which there are no plausible mechanisms or natural analogues, and for which the solution is to lobby government build a moat around his business and limit his competitors. We do not even have a consensus around a working definition of "intelligence", let alone any evidence that it is a linear or unbounded phenomenon, and even if it were, there is no evidence ChatGPT is a route to even human-level intelligence. The sum total of research into this "field" is a series of long chains of philosophical leaps that rapidly escape any connection to reality, which is no basis for a wide-ranging government intervention.
- deleted 3y ago
  [deleted]
- adamisom 3y ago
  > We do not even have a consensus around a working definition of "intelligence" I’m sorry but if you really think this is relevant, you just aren’t paying attention. This is my number one giveaway sign that someone just doesn’t have any idea why lots of people are concerned about ai x-risk.
- peripitea 3y ago
  What do you mean there are no plausible mechanisms? This reads like one of those "it will take 1-10 million years for man to fly" kind of statements.
  - cubefox 3y ago
    Exactly! How can so many people at HN not see this?
- filoleg 3y ago
  Thanks for putting into words what I couldn't articulate well, but what has been bothering me for a while. Very concise and to the point, saved your comment. It is just tiring seeing all those fearmongering articles in the news media about omnipotent god-like AGI, based on nothing but self-aggrandizing statements from "experts" (with large followings that have nothing to do with the topic they are authoritatively talking about) like Sam Altman. It got to the point, where seeing those articles makes me question things over and over in my head, only to come to the same conclusion (as nothing material has changed to lead me to a different conclusion this time compared to previous times). Almost feels like I am being gaslighted. And it feels so strong, I cannot even blame people outside of tech falling for all of this (let alone people of the typical US senators' age and tech literacy). Which is the exact part that makes me feel the most uneasy - people who understand this game like Sam Altman does and who have no issues with capitalizing on that using this dishonest rhetoric (to the major detriment of the rest of the people). /rantover
j45 3y ago
It’s interesting to see openai rush out of the gate without this. Gpt4 existed for 6-8 months before it was released. It wouldn’t be a stretch to say gpt5 or enough of it exists today. Did this become an issue because of what exists being closed doors, or because there is suddenly a lot of competition and catch-up from the big 3? Safety in regulation is often a moat against competition. Concerns about superintendent aside, who picks the haves and have nots for Full access to this technology? Instead of computers becoming a threat, are people being divided to be against each other?
deleted 3y ago
[deleted]
endisneigh 3y ago
The government will not save you. If anything what we need is more governance of the government. So many problems caused by government foolishness - housing crisis, giving businesses far too much influence over legislation, etc
- hammyhavoc 3y ago
  How do you propose that one governs government?
0xbadcafebee 3y ago
> Given the picture as we see it now, it’s conceivable that within the next ten years, AI systems will exceed expert skill level in most domains, and carry out as much productive activity as one of today’s largest corporations. This is the most ridiculous thing I've ever heard claimed about AI. They finally have a crappy algorithm that can sound confident, even though over half its answers are complete bullshit. With this accomplishment, they now expect in 10 years it will be able to do any job, better than an expert. This is some major league swing-for-the-fences bullshit. And that's not the worst part. Let's say the fancy algorithm has real pixie dust that just magically gives better-than-expert answers for literally any question. That still leaves it to the human to ask the questions. How much do you want to bet a police force won't use AI by submitting a random picture of a young black male suspect and asking it "What is the likelihood this person has committed a crime, and what was the crime?" The AI just interprets the question and answers it, and the human just accepts the answer, even though the premise is ridiculous. We won't create a real intelligent AI any time soon. But even if we did, the AI being perfect or not isn't the problem. The problem is the stupid humans using it. You can't "design", regulate, govern, etc your way out of human stupidity.
- AnimalMuppet 3y ago
  More cynically, this could be a claim that half of the expert's answers are also BS. Does anyone feel confident in their ability to disprove that one?
  - deleted 3y ago
    [deleted]
- cubefox 3y ago
  September 30, 2012: AlexNet achieves a top-5 error of 15.3% in ImageNet. November 30, 2022: ChatGPT launches, two to three months after GPT-4 internal training has finished. That's quite a lot of progress in 10 years. But somehow you are convinced no similar groundbreaking stuff will happen in the next 10 years.
  - 3y ago
progbits 3y ago
As I said last week on the lobbying article [1], if you don't like how "Open"AI is trying to build a regulatory capture moat, cancel your subscription. Yes, the open models are worse, but are getting better. There will be plenty of high quality commercial alternatives. [1] https://news.ycombinator.com/item?id=35967864 https://news.ycombinator.com/item?id=35967864
- api 3y ago
  Is there any place one can donate toward the creation of bigger and better open source models? I'll send my $20/month there.
phonebucket 3y ago
Is this article actually trying to talk about superintelligence? Or is this article just trying to make GPT look like AGI by taking AGI for granted?
deleted 3y ago
[deleted]
habryka 3y ago
I care quite a bit about distinguishing whether OpenAI is doing the things it is doing for regulatory capture reasons, or whether they are genuinely interested in reducing the risks around AI. So here are some thoughts on this: 1. It does sure seem like this licensing regime would limit the entrance of new players into the AGI market quite a bit, which would probably increase OpenAIs profits a bunch. 2. However, it also makes sense from a safety perspective to reduce the number of players, since that enables more coordination and reduces race-dynamics 3. OpenAI itself would likely be hit pretty hard by these regulations, though if they are the ones driving the progress they might be able to get some privileged position in the law-making process that leaves them more free than competitors 4. OpenAI hasn't made any kind of costly signal or promise that shows they really care about this kind of regulation, though it's unclear what a good promise or costly signal would be. "We would join any nationalized AGI project" would be something, though I also think it's pretty plausibly a quite bad idea. 5. Altman doesn't have any equity in OpenAI, though my sense is he would still end up extremely powerful and somehow get a ton of resources if OpenAI ends up ahead in the AGI race (before the AGI causes catastrophe that is).
- csomar 3y ago
  > Altman doesn't have any equity in OpenAI, though my sense is he would still end up extremely powerful and somehow get a ton of resources if OpenAI ends up ahead in the AGI race I think this explains the current dynamics, right? If Altman had serious equity in OpenAI, then he'd be concerned about turning a profit or making the next GPT-x. But this is not the case and now he is playing the bureaucracy game as it is the one that will give him power and leverage.
joduplessis 3y ago
I'm not sure Altman/OpenAI know how transparent that ruse was. A bit disappointing witnessing a for-good org, turning into a for-profit one.
ddp26 3y ago
Setting aside value judgments on whether this is a good idea or not, it's odd how nonspecific this blog post is. Metaculus has some probabilities [1] of what kind of regulation might actually happen by ~2024-2026, e.g. requiring disclosure of human/non-human, restricting APIs, reporting on large training runs, etc. [1] https://www.metaculus.com/project/ai-policy/ https://www.metaculus.com/project/ai-policy/
ctoth 3y ago
> Tracking compute and energy usage could go a long way, and give us some hope this idea could actually be implementable. > the cost to build it decreases each year I wonder if the first quote passed out of their context window while they were writing the second part?
jacooper 3y ago
Fwiw, altman did request the U.S. government not put a legal burden on open source and Startup AI. https://twitter.com/exteriorpower/status/1659069336227819520 https://twitter.com/exteriorpower/status/1659069336227819520
- api 3y ago
  He's being disingenuous and hoping people are too stupid to see the implication of what he's proposing. He's saying it's fine if there's a kiddie pool, but that there should be a glass ceiling beyond which regulation kicks in. Where will that glass ceiling be? Well of course it'll be just behind where OpenAI and other market leaders are at any given time. The argument will be made that because we know current offerings are safe, the ceiling should be just below current offerings. So it becomes basically illegal to compete with market leaders, namely OpenAI/Microsoft and Google. Other huge corporations will be able to buy their way into the club but upstarts will find it nearly impossible. Regulation will also require that everyone doing anything non-trivial report it to regulatory bodies so existing market incumbents can become aware of it. They can then prepare a strategy to respond either by lobbying to block it with regulation or preparing a market response ahead of time using massively greater resources. No stealth mode AI startups or other private efforts that OpenAI et. al. don't know about would be allowed. The end result of this will be to push innovation in this area off shore and open source onto the dark web.
nborwankar 3y ago
The elephant in the room is enforcement. For example, how on earth are you going to control what a rogue state like North Korea might do with AGI supplied by say China for military use such as autonomous killer drones? Will the international AI regulatory agencies insist on having monitors tracking everything happening on every cloud computer in every country? And expect every country to comply? This is not like nuclear testing which can be monitored by satellites. It’s very interesting that enforcement is not mentioned in all the verbiage around regulation. It’s definitely not addressing a critical aspect ie monitoring hostile military use of AGI. Which makes it a poorly thought through argument that only regulates the tech in the US and perhaps EU. I don’t think the OpenAI folks are naive about this. So why is this issue being left unaddressed? It gives credence to the accusations that the push for regulation is self serving.
- hammyhavoc 3y ago
  Monitoring via satellite does not mean stopping a country from doing x does it? It just means knowledge of it. Autonomous killer drones are already a possibility without a hypothetical AGI. It really changes nothing. Proxy wars fought via drones is modernity. The thing that does change it the ability to spread propaganda online like never before.
  - api 3y ago
    Open source models like Alpaca are already more than good enough to flood social media with unfilterable spam and propaganda that's more than good enough to convince a lot of people. Look at stuff like Qanon. You don't need an argument that's very sophisticated, just a lot of it and the ability to promote it at scale via hordes of undetectable bots. Current generation free models are good enough to do all that. People are either going to learn to become skeptical or they're going to be manipulated at massive scale. We are already there and arguably have been for a while.
- cubefox 3y ago
  They apparently don't know either what to do about enforcing international coordination. So they don't mention the problem. The situation seems quite hopeless.
jimsimmons 3y ago
How did this post go from new → top of HN → buried in 3rd page in 1 hour?!
carlosbaraza 3y ago
Why is everyone so cynical about this? I use GPT-4 all the time and it blows my mind often. OpenAI has demonstrated the capabilities to develop advanced AI, why would it be so hard to forecast that these models would keep improving and potentially be risky? > ...or we could collectively agree (with the backing power of a new organization like the one suggested below) that the rate of growth in AI capability at the frontier is limited to a certain rate per year. Why would they want to limit the growth rate of their cutting edge to a fixed rate if this effort wasn't genuine? This would just give all the other providers time to catch up, which is an undesirable business decision.
ddtaylor 3y ago
> systems dramatically more capable than even AGI We have absolutely nothing near AGI.

93 points by davidbarker 3y ago

There is not a single shred of evidence we'll reach superintelligence any time soon. There's however a lot of evidence that regulation benefits incumbents. You can do the math from there.

mitthrowaway2 3y ago
What is the evidence you'd expect to see that you aren't seeing, and how many years in advance is too soon to make a plan?
- groby_b 3y ago
  Any development trajectory that doesn't look very suspiciously like a logistics curve is a good starter. (Pretty much all LLMs I've seen show an exponential increase in resources for a less-than linear increase in capability lately) And if we want to make a plan, how about we don't ask the fox to guard the hen house?
cubefox 3y ago
AI progress keeps accelerating more and more --> "not a single shred of evidence"
- groby_b 3y ago
  Does it? Does it really? Yes, we've seen huge improvements. We're still not close to human intelligence level. (No, test taking doesn't count). Worse, we've got pretty clear evidence that the training material increase for larger LLMs is basically reaching an upper bound. What we'd need to see: * A breakthrough that either decouples us from parameter count or allows parameter count increases with smaller training sets. * Any evidence that it's doing anything more than asymptotically crawling towards human-comparable evidence. The entire ZOMG SUPERAI faction suffers from the AI that somehow thinking more and faster is thinking better. It's not. There's no evidence pointing in that direction. We currently have ~8B human-level intelligences. They haven't managed to produce anything above human level intelligence. Where's the indication that emulation of their mode of thinking at scale will result in something breaking the threshold? If anything, machine intelligence is doing worse, because any slight increase in capacity is paid for in large amounts of "hallucination".
  - cubefox 3y ago
    That's like arguing 10 years ago that ChatGPT was impossible in the next ten years because AlexNet could only recognize objects in photos with some medium reliability, and there was no path to scale CNNs to something like ChatGPT. The mistake of course was assuming we were stuck with CNNs. And we will probably also not keep using LLMs. We already know there are more effective architectures, as animals implement one of them.
    - groby_b 3y ago
      None of the architectures we know imply in any way giant leaps in intelligence. That's the main sticking point. Thinking more does not equate thinking better. Or even thinking well. As for "animals implement them", it's worth noting that we mostly qualify for an award in impressive lack of understanding in that area. Even with exponential improvements, that is not going to change within the next five years. The "but we just don't know" argument is useless. That also applies to aliens landing on this planet next week and capturing the government. Theoretically possible, but not a pressing concern. Should we think about what AI regulations look like? Yes. Should we enact regulations on something that doesn't even really exist, without deeply understanding it, at the behest of the party that stands to gain financially from it? Fuck no.

immichaelwang 3y ago

Is there any downside whatsoever for OpenAI/Sam to be the one proposing/leading the calls for regulation? Cynics will say they are trying to pull the ladder up from underneath them, so this is massively beneficial for them. What's the downside (if any)? Getting a small subset of the community mad doesn't seem like a lot of downside.

A4ET8a8uTh0 3y ago
<< What's the downside (if any)? Compliance is still a burden. Even if you write the law, eventually the bureaucracy that was willed into existence will start living its own life, imposing new restrictions and basically making your life miserable. Still, the profits from keeping the field restricted to a small circle of largely non-competitors helps to offset that.
adastra22 3y ago
Then they’d be engineering their own eventual demise. Any regulatory capture regime ends up stalling progress and bloating incumbents until eventually a nimble competitor is able to circumvent regulators and steal their lunch money.
- namaria 3y ago
  Humans are temporary beings, we only need temporary wealth.
  - adastra22 3y ago
    If that were true, Sam would never have founded OpenAI. He already had generational wealth.
mhb 3y ago
Whether there's a downside is moot since no one knows how to do any sort of regulation effectively. That being said, I don't know why you think that only a small community will see this as self-serving. It's not subtle even though it may be unavoidable.

ablyveiled 3y ago

Security is impossible. Therefore, "AI alignment" is impossible. I'm not sure that even AGI is possible, per Bostrom's Self-Sampling Assumption.

jw1224 3y ago
> I'm not sure that even AGI is possible, per Bostrom's Self-Sampling Assumption. Can you explain more? I’ve found a definition for SSA, but unsure how it applies to AGI…
- ablyveiled 3y ago
  The self-sampling assumption essentially says that you should reason as though your self-sampling is the likely outcome in consciousness-space-time. It's the anthropic principle applied to the distribution/shape of intelligence. Since your self-sample is of a human-shaped mind, human-shaped minds are the most likely outcome. So the probability of AGI is unlikely.

generalizations 3y ago

- Corporate-owned proprietary AIs? Check. - Monitoring of power consumption for illegal computer usage? Check. - Superintelligent AIs under tight supervision? Check. - Bootlegged neural nets passed around on torrenting networks? Check. - Poverty and homelessness running rampant? Check. Folks, we're officially living in a cyberpunk dystopia.

EGreg 3y ago
Yup. I have been warning about this for years. And things are going to get much more weird soon. “But Web3 sucks” Hold. The beer.
- __loam 3y ago
  Web3 sucks regardless of how this shakes out.
  - EGreg 3y ago
    Web3 is positive but close to zero sum game for society, say. While AI can be massively negative sum game for the world.
    - __loam 3y ago
      Web3 is provably negative sum actually.
schaefer 3y ago
> - Poverty and homelessness running rampant? Check. Compared to what, exactly? because over the last 50 years, there have been dramatic improvements[1]. [1]: https://www.brookings.edu/research/the-evolution-of-global-poverty-1990-2030/ https://www.brookings.edu/research/the-evolution-of-global-p... It's true - there's room to do better. So, so much better. But discarding the progress of the last 50 years is so unbelievably counter-productive.
- 3y ago

runsWphotons 3y ago

What is an example of an i dependent AI agent carrying out as much productive activity as a large company that is plausible?

j45 3y ago
Lots. Most enterprises processes and algorithms are big mosts around small processes really well understood which are tied together and neurotically checked or sufficiently slowed down to ask “are we sure?” enough times before implementing a change.
cubefox 3y ago
An AI that can build robots which build robots.
- runsWphotons 3y ago
  Is there a prototype of this?

BeenAGoodUser 3y ago

I think it is counterproductive to limit the rate of growth of AI since some countries — typically the ones who are less concerned about ethics — will not follow those rules and could gain a considerable edge over the rest of the world this way.

generic92034 3y ago
Or they are the first countries falling to the superintelligent AIs they themselves have developed... :o
- Jackson__ 3y ago
  ... with the rest of the world following shortly.
standardUser 3y ago
Historically, technologically advanced nations have worked together to develop some limited governance over sensitive technologies. It's an imperfect system, but tends to be better than a free-for-all.
- cubefox 3y ago
  Counterexample: Atom bombs, ICMBs, spy satellites.
  - standardUser 3y ago
    You literally just named the actual examples.
    - cubefox 3y ago
      MAD is the opposite of cooperation.
      - 3y ago

abeppu 3y ago

> Second, we are likely to eventually need something like an IAEA for superintelligence efforts > Third, we need the technical capability to make a superintelligence safe. This is an open research question that we and others are putting a lot of effort into. I think it's important to remember that the IAEA was not proposed until 1954 and didn't get approved by the UN until 1957. By that time we had already gotten past the first fission reaction, fission weapon testing and uses, nuclear submarine and grid-connected power generation. We were trying to control something we already knew how to build, and about which we already understood a bunch of specific risks. And this is for a technology which (a) requires access to rare materials and (b) there's a pretty definite line that shows when fission is happening, even briefly. It's a different order of challenge to try to regulate the development of something we _don't_ yet understand how to build, which might be possible to build with computers which are everywhere, and where we don't have a clear mechanism to detect or identify the thing we're trying to control, let alone to decide whether or not it's "safe".

Animats 3y ago

After Putin, Xi, Trump, and Boris Johnson, we should give the AIs a chance.

sidcool 3y ago

What would superintelligence look like? What is the worst case scenario?

causi 3y ago

I wonder if AI will ever scale down to personal hardware, or if every step from here forward is going to rely on an internet connection forever. I don't like the idea of never having real control.

Animats 3y ago
> I wonder if AI will ever scale down to personal hardware Happened last week. Download here.[1] https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all

jonathankoren 3y ago

Interesting that they’re telling us to focus on dubious sci-fi movie tropes instead of actual dangers and risks documented today. But yeah, Skynet is who we need to watch out for, not biased AIs making decisions today.

cubefox 3y ago
Biased AI today is extremely unimportant compared to AI posing an existential risk in the future. It's like saying: "Who cares about the hypothetical effects climate change allegedly has in the far future? Let's focus on the effects that the local highway has on our frog population today."
- jonathankoren 3y ago
  I’ll do one better. The existential risk of an AI running amok and enslaving or exterminating all of humanity is based on series of low probability events that we can’t even put error bars on, let alone understand what would be needed to even achieve such a feat. HOWEVER, the existential risk posed by the sun running out of hydrogen and destroying all life on planet Earth has known a probability (1.0), with known error bars around the best estimation of when that will happen. Furthermore, the problems involved with moving masses through space are understood quite well. As such, this is merely an engineering problem, rather than a problem of inadequate theories and philosophies. Therefore, it is undeniably logical that we must immediately redirect all industrial and scientific output to building a giant engine to move the Earth to another star with a longer lifespan, and any argument otherwise is an intentional ploy to make humanity go extinct. Any sacrifices made by the current and near future generations is a worthy price to pay for untold sextillions of humanity and its evolution all descendants that would be condemned to certain death by starvation if we did not immediately start building the Earth Engine. This is the only logical, mathematical provable, and morally correct answer.
  - cubefox 3y ago
    Series of low probability events? If a superintelligent AI has goals that differ from ours, then too bad for us. As Altman & Co say, the alignment problem is unsolved.

sylware 3y ago

Why would a big tech company be stoupid enough to build a massive and super fast neural net (at least on the human brain scale) with real time complex outputs/inputs and which would be trained (interacted) continuously for years? You know there is a big chance "it" could become self-aware, don't you? Oh! And "shutting down"/"resetting" such self-aware being would be murder. In theory, humanity would build such neural net if it was facing a certain extinction event and that "close" in time which human science/maths had no answer to. It might give us beyond human science/maths which could save humanity: Humanity: "Super AI? How could we save ourself from our soon to happen doomsday?" Super AI: "42"

tsunamifury 3y ago

I'm sorry, but Sam and OpenAI are massively overhyping the scary AI apocalypse vision of the future to get regulation in order to cover the "We store content from the planet and now want the exclusive rights to monetize it." I've been in Silicon Valley for many years, and this takes the cake of the most cynical, in your face, billionaires should steal sheep from their poor neighbors level of run away greed and sociopathy I have ever witnessed.

j45 3y ago
It’s Super Intelligent AI for me, but not for thee. It’s super intelligent AI for the few and not the many. Would demanding safety suddenly make a little more sense if a super intelligent GPT5 already exists? If someone has it they want to keep that advantage.

r_lee 3y ago

Isn't Sam Altman also developing a crypto project which collects your iris data? I mean, I don't understand what is up with their obsession with this AI Governance, it seems like it is indeed about retaining control, so that only the big guys can [attempt] to control any larger models. Same with him proposing such "AI Licenses". What a time to be alive...

deleted 3y ago
[deleted]

amelius 3y ago

Can't we just ask the AI how we should manage the AI?

lsy 3y ago

Nothing makes me think of Altman as a grifter more than his trying to spook uneducated lawmakers with sci-fi notions like "superintelligence" for which there are no plausible mechanisms or natural analogues, and for which the solution is to lobby government build a moat around his business and limit his competitors. We do not even have a consensus around a working definition of "intelligence", let alone any evidence that it is a linear or unbounded phenomenon, and even if it were, there is no evidence ChatGPT is a route to even human-level intelligence. The sum total of research into this "field" is a series of long chains of philosophical leaps that rapidly escape any connection to reality, which is no basis for a wide-ranging government intervention.

deleted 3y ago
[deleted]
adamisom 3y ago
> We do not even have a consensus around a working definition of "intelligence" I’m sorry but if you really think this is relevant, you just aren’t paying attention. This is my number one giveaway sign that someone just doesn’t have any idea why lots of people are concerned about ai x-risk.
peripitea 3y ago
What do you mean there are no plausible mechanisms? This reads like one of those "it will take 1-10 million years for man to fly" kind of statements.
- cubefox 3y ago
  Exactly! How can so many people at HN not see this?
filoleg 3y ago
Thanks for putting into words what I couldn't articulate well, but what has been bothering me for a while. Very concise and to the point, saved your comment. It is just tiring seeing all those fearmongering articles in the news media about omnipotent god-like AGI, based on nothing but self-aggrandizing statements from "experts" (with large followings that have nothing to do with the topic they are authoritatively talking about) like Sam Altman. It got to the point, where seeing those articles makes me question things over and over in my head, only to come to the same conclusion (as nothing material has changed to lead me to a different conclusion this time compared to previous times). Almost feels like I am being gaslighted. And it feels so strong, I cannot even blame people outside of tech falling for all of this (let alone people of the typical US senators' age and tech literacy). Which is the exact part that makes me feel the most uneasy - people who understand this game like Sam Altman does and who have no issues with capitalizing on that using this dishonest rhetoric (to the major detriment of the rest of the people). /rantover

j45 3y ago

It’s interesting to see openai rush out of the gate without this. Gpt4 existed for 6-8 months before it was released. It wouldn’t be a stretch to say gpt5 or enough of it exists today. Did this become an issue because of what exists being closed doors, or because there is suddenly a lot of competition and catch-up from the big 3? Safety in regulation is often a moat against competition. Concerns about superintendent aside, who picks the haves and have nots for Full access to this technology? Instead of computers becoming a threat, are people being divided to be against each other?

deleted 3y ago

[deleted]

endisneigh 3y ago

The government will not save you. If anything what we need is more governance of the government. So many problems caused by government foolishness - housing crisis, giving businesses far too much influence over legislation, etc

hammyhavoc 3y ago
How do you propose that one governs government?

0xbadcafebee 3y ago

> Given the picture as we see it now, it’s conceivable that within the next ten years, AI systems will exceed expert skill level in most domains, and carry out as much productive activity as one of today’s largest corporations. This is the most ridiculous thing I've ever heard claimed about AI. They finally have a crappy algorithm that can sound confident, even though over half its answers are complete bullshit. With this accomplishment, they now expect in 10 years it will be able to do any job, better than an expert. This is some major league swing-for-the-fences bullshit. And that's not the worst part. Let's say the fancy algorithm has real pixie dust that just magically gives better-than-expert answers for literally any question. That still leaves it to the human to ask the questions. How much do you want to bet a police force won't use AI by submitting a random picture of a young black male suspect and asking it "What is the likelihood this person has committed a crime, and what was the crime?" The AI just interprets the question and answers it, and the human just accepts the answer, even though the premise is ridiculous. We won't create a real intelligent AI any time soon. But even if we did, the AI being perfect or not isn't the problem. The problem is the stupid humans using it. You can't "design", regulate, govern, etc your way out of human stupidity.

AnimalMuppet 3y ago
More cynically, this could be a claim that half of the expert's answers are also BS. Does anyone feel confident in their ability to disprove that one?
- deleted 3y ago
  [deleted]
cubefox 3y ago
September 30, 2012: AlexNet achieves a top-5 error of 15.3% in ImageNet. November 30, 2022: ChatGPT launches, two to three months after GPT-4 internal training has finished. That's quite a lot of progress in 10 years. But somehow you are convinced no similar groundbreaking stuff will happen in the next 10 years.
- 3y ago

progbits 3y ago

As I said last week on the lobbying article [1], if you don't like how "Open"AI is trying to build a regulatory capture moat, cancel your subscription. Yes, the open models are worse, but are getting better. There will be plenty of high quality commercial alternatives. [1] https://news.ycombinator.com/item?id=35967864 https://news.ycombinator.com/item?id=35967864

api 3y ago
Is there any place one can donate toward the creation of bigger and better open source models? I'll send my $20/month there.

phonebucket 3y ago

Is this article actually trying to talk about superintelligence? Or is this article just trying to make GPT look like AGI by taking AGI for granted?

deleted 3y ago

[deleted]

habryka 3y ago

I care quite a bit about distinguishing whether OpenAI is doing the things it is doing for regulatory capture reasons, or whether they are genuinely interested in reducing the risks around AI. So here are some thoughts on this: 1. It does sure seem like this licensing regime would limit the entrance of new players into the AGI market quite a bit, which would probably increase OpenAIs profits a bunch. 2. However, it also makes sense from a safety perspective to reduce the number of players, since that enables more coordination and reduces race-dynamics 3. OpenAI itself would likely be hit pretty hard by these regulations, though if they are the ones driving the progress they might be able to get some privileged position in the law-making process that leaves them more free than competitors 4. OpenAI hasn't made any kind of costly signal or promise that shows they really care about this kind of regulation, though it's unclear what a good promise or costly signal would be. "We would join any nationalized AGI project" would be something, though I also think it's pretty plausibly a quite bad idea. 5. Altman doesn't have any equity in OpenAI, though my sense is he would still end up extremely powerful and somehow get a ton of resources if OpenAI ends up ahead in the AGI race (before the AGI causes catastrophe that is).

csomar 3y ago
> Altman doesn't have any equity in OpenAI, though my sense is he would still end up extremely powerful and somehow get a ton of resources if OpenAI ends up ahead in the AGI race I think this explains the current dynamics, right? If Altman had serious equity in OpenAI, then he'd be concerned about turning a profit or making the next GPT-x. But this is not the case and now he is playing the bureaucracy game as it is the one that will give him power and leverage.

joduplessis 3y ago

I'm not sure Altman/OpenAI know how transparent that ruse was. A bit disappointing witnessing a for-good org, turning into a for-profit one.

ddp26 3y ago

Setting aside value judgments on whether this is a good idea or not, it's odd how nonspecific this blog post is. Metaculus has some probabilities [1] of what kind of regulation might actually happen by ~2024-2026, e.g. requiring disclosure of human/non-human, restricting APIs, reporting on large training runs, etc. [1] https://www.metaculus.com/project/ai-policy/ https://www.metaculus.com/project/ai-policy/

ctoth 3y ago

> Tracking compute and energy usage could go a long way, and give us some hope this idea could actually be implementable. > the cost to build it decreases each year I wonder if the first quote passed out of their context window while they were writing the second part?

jacooper 3y ago

Fwiw, altman did request the U.S. government not put a legal burden on open source and Startup AI. https://twitter.com/exteriorpower/status/1659069336227819520 https://twitter.com/exteriorpower/status/1659069336227819520

api 3y ago
He's being disingenuous and hoping people are too stupid to see the implication of what he's proposing. He's saying it's fine if there's a kiddie pool, but that there should be a glass ceiling beyond which regulation kicks in. Where will that glass ceiling be? Well of course it'll be just behind where OpenAI and other market leaders are at any given time. The argument will be made that because we know current offerings are safe, the ceiling should be just below current offerings. So it becomes basically illegal to compete with market leaders, namely OpenAI/Microsoft and Google. Other huge corporations will be able to buy their way into the club but upstarts will find it nearly impossible. Regulation will also require that everyone doing anything non-trivial report it to regulatory bodies so existing market incumbents can become aware of it. They can then prepare a strategy to respond either by lobbying to block it with regulation or preparing a market response ahead of time using massively greater resources. No stealth mode AI startups or other private efforts that OpenAI et. al. don't know about would be allowed. The end result of this will be to push innovation in this area off shore and open source onto the dark web.

nborwankar 3y ago

The elephant in the room is enforcement. For example, how on earth are you going to control what a rogue state like North Korea might do with AGI supplied by say China for military use such as autonomous killer drones? Will the international AI regulatory agencies insist on having monitors tracking everything happening on every cloud computer in every country? And expect every country to comply? This is not like nuclear testing which can be monitored by satellites. It’s very interesting that enforcement is not mentioned in all the verbiage around regulation. It’s definitely not addressing a critical aspect ie monitoring hostile military use of AGI. Which makes it a poorly thought through argument that only regulates the tech in the US and perhaps EU. I don’t think the OpenAI folks are naive about this. So why is this issue being left unaddressed? It gives credence to the accusations that the push for regulation is self serving.

hammyhavoc 3y ago
Monitoring via satellite does not mean stopping a country from doing x does it? It just means knowledge of it. Autonomous killer drones are already a possibility without a hypothetical AGI. It really changes nothing. Proxy wars fought via drones is modernity. The thing that does change it the ability to spread propaganda online like never before.
- api 3y ago
  Open source models like Alpaca are already more than good enough to flood social media with unfilterable spam and propaganda that's more than good enough to convince a lot of people. Look at stuff like Qanon. You don't need an argument that's very sophisticated, just a lot of it and the ability to promote it at scale via hordes of undetectable bots. Current generation free models are good enough to do all that. People are either going to learn to become skeptical or they're going to be manipulated at massive scale. We are already there and arguably have been for a while.
cubefox 3y ago
They apparently don't know either what to do about enforcing international coordination. So they don't mention the problem. The situation seems quite hopeless.

jimsimmons 3y ago

How did this post go from new → top of HN → buried in 3rd page in 1 hour?!

carlosbaraza 3y ago

Why is everyone so cynical about this? I use GPT-4 all the time and it blows my mind often. OpenAI has demonstrated the capabilities to develop advanced AI, why would it be so hard to forecast that these models would keep improving and potentially be risky? > ...or we could collectively agree (with the backing power of a new organization like the one suggested below) that the rate of growth in AI capability at the frontier is limited to a certain rate per year. Why would they want to limit the growth rate of their cutting edge to a fixed rate if this effort wasn't genuine? This would just give all the other providers time to catch up, which is an undesirable business decision.

ddtaylor 3y ago

> systems dramatically more capable than even AGI We have absolutely nothing near AGI.