6 ms·

> And one possibility might be that AIs could “improve themselves” to produce a single “apex intelligence” that would in a sense dominate everything else. But h

by mLuby 4y ago

> And one possibility might be that AIs could “improve themselves” to produce a single “apex intelligence” that would in a sense dominate everything else. But here we can see computational irreducibility as coming to the rescue. Because it implies that there can never be a “best at everything” computational system. It’s a core result of the emerging field of metabiology: that whatever “achievement” you specify, there’ll always be a computational system somewhere out there in the computational universe that will exceed it. That tigers are stronger than wolves doesn't prevent wolves from killing you. Wolfram makes that class of error repeatedly. Another example: what prevents a system of ethics from being both Godel-incomplete/inconsistent and at the same time encompassing everything in the realm of human experience? > For the first time in history, it’s become realistic to truly automate intellectual tasks. The leverage this provides is completely unprecedented. The essay also mentions how humans are still the best at manual labor, and how government can work with humans directing the AI as it runs things every more efficiently. It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy. The author is too used to thinking from first principles and hasn't studied enough human history to understand how leverage gets abused.

komali2 4y ago
> The author is too used to thinking from first principles and hasn't studied enough human history to understand how leverage gets abused. My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power. Right now power is primarily tied to capital. Access to GPT4 type technologies, or the ability to develop and use these technologies, is tied to same. More money, bigger servers, or, better likelihood to get access to GPT4 before anyone else, GPT5, whatever. Micro$oft can use as much as it wants right now. That capital is tied to power hasn't mattered in any revolution in the past because as soon as the "rabble" realizes this and dissolves capital, by every measure whatever that currency was instantly devalues as other more "true" measures of power are realized, such as rhetorical ability (convincing others to fight and die for your cause), raw strength (convincing others to not bother trying to fight back), or technological advantage and access "in the real" (being the one actually holding the gun). "Numbers" falls in there somewhere but is so tied into the other three measures it's hard to figure out how. Chomsky was onto it with "Manufacturing Consent." Rhetorical ability being one of the most dangerous weapons in a power struggle (used correctly it can direct both the strongest and the ones holding the guns, in greater numbers), leveraging that is the surest way to control the whole population. Before, you had to be Rupert Murdoch to do it, or buy him, which is expensive. Soon you'll just need to buy "Open"AI, or, reproduce their technology and their hardware. As soon as that's done, you can find out all sorts of cost effective methods to do what buying Rupert Murdoch used to do, and you can probably investigate ways to really cement this power. For example, setting up a fascist state, we've tried that a couple times and some people are trying that again, but every fascist state has failed eventually, so it's probably not a sustainable plan. What new, horrible method haven't we tried yet? One that lets us control people as effectively as we have through artificial scarcity (capitalism) but without the downsides of them eventually forming unions and finding ways to fight back within the system, or, reject the system entirely by going to form communes or whatever? Without the inherent contradictions that cause the system to cannibalize itself (depression of wages depressing buying power depressing profit leading to collapse). What creative legal structures can we develop to ensure that copyright can exist forever, but only for your friends? And how can we make sure that this power structure propagates indefinitely to our friends and family? Maybe this means a GPT5 recommending the construction of a Bene Gesserit style cult? Maybe it stumbles upon incredibly effective human-programming Memes, such as what Neal Stephenson warned about in Snowcrash and Fall / Reamde (can't remember which, whichever involved the collapse of the internet to distributed bots). My solution comes from Stephenson and Doctorow: distribute the technology freely. The sooner everyone has access to the incredible propagandic power of these technologies, the sooner we can develop counter measures. The task of the power-aggregator (fascist or whatever) will cease being "give me effective propaganda to easily control a population," it will be "give me propaganda that will bypass the population's GPT filter solutions." An arms race, but a more equal playing field.
- throwaway99111 4y ago
  >My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power. So, this is potentially a crackpot theory, but hear me out. I've noticed that a lot of the hype runners, posts claiming their businesses are optimizing this with chatgpt, or they've replaced most of their coding with chatgpt outputs have been generally accounts here that were created in 2021 or later. While before the last couple of months, you should generally assume good faith on HN and elsewhere, the fact is mass astroturfing now is very much possible on HN and elsewhere with chatGPT, and why wouldn't OpenAI want to hype their product and get as many users on board? They certainly could and would not be noticeable. You don't need nefarious political goals if you're just trying to sell something. You just need to create enough hype and illusion to grow your product. SV companies have done it before, and now it's automated, to a degree. One explanation (beyond basic confirmation bias) is that a lot of people joined the crypto wave and knew the orange site was the place where tech people hung out, and a lot of those types ditched crypto and are riding the AI wave, but I really cannot be sure anymore.
  - ChatGTP 4y ago
    I kind of feel you, everything seems really hyped, more than it should be. I mean I still don't even have any idea what ChatGPTs actual use case is besides look super impressive. It seems like it can do a bunch of stuff, sometimes really well, other times spectacularly bad. It's not an a true AGI, but it's kind of marketed like one. I think we're all completely baffled by it? I mean the thing can talk right? So we're all fired?
    - komali2 4y ago
      As per usual everyone else keeps coming up with the ideas that when I hear, I think, "damn, seems obvious, wish I had thought of that." Sometimes it's because it's a good idea, sometimes because it's an idea that will probably get funding, which makes it a good idea :P Examples: 1. Ingesting a shitload of unread emails after you come back from vacation, then telling you briefly what important things you missed 2. The same idea but for slack 3. Ingesting request-for-cost response emails from various suppliers and outputting the data in a machine-readable way so it can be easily ingested by another API 4. Generating "individual" lesson plans based on student needs (I've heard 5 different pitches around this, majority for language learning) 5. student tutor chatbot There's some other ideas that were too stupid (or more probably, too innovative for dumb old me to comprehend) to remember. At the very least it has people thinking and being imaginative, which I think is pretty cool. I like getting into political debates with the thing.
noobermin 4y ago
To be honest, neither has most people here, most people working at openai, or the hypemen around AI.
- XorNot 4y ago
  I'm pretty sure the people working at OpenAI have thought about it, and decided that it's better they get there first then some autocracy.
- still_grokking 4y ago
  I don't think so. Almost all people promoting this stuff do that for their very own advantage in the first place. They want to be the ones with the lever, because they know that this will become a very powerful lever.
Animats 4y ago
> And one possibility might be that AIs could “improve themselves” to produce a single “apex intelligence” that would in a sense dominate everything else. That's not what worries me. It's when AIs learn to distribute work to other AIs and humans. Once that starts to work, machines are going to be better at coordination than humans, simply because they have more I/O bandwidth.
- echelon 4y ago
  No more middle managers! But seriously, taking middle management self-promotion out of the equation could reduce some of the rotten malinvestment and political in-fighting that happens in large organizations.
  - still_grokking 4y ago
    To the AI all humans are "middle managers" though…
  - ta1243 4y ago
    Or it could result in the first distopia of "Manna", where humans are there for cleaning, but there's no middle management. https://marshallbrain.com/manna1 https://marshallbrain.com/manna1
- still_grokking 4y ago
  Yes, being able to perform orders of magnitude more APM than the opponent is a sure win, even when the particular actions aren't optimal.
- Faark 4y ago
  Imagine an AI spawning many parallel instances of itself, all feeding back into the same conscience. So many coordination problems just gone. All the resources humans spend fighting each other in some form or another... And entirely depending only on itself. No need for any external being. That could prove very deadly to us, since "we need each other" is the core pillar forming our society. A king needs his peasants. An AI? Could just boot up another instance...
ly3xqhl8g9 4y ago
15,687 words in that article and none of them are wisdom or metaphysics. As long as we won't have a change in our metaphysics (assigning value and deriving meaning from 'having a job' and so on), and we still proud ourselves in being "jobs creators" [1], it's all for nothing, i.e., more shareholder value. Because of our current metaphysics, instead of living in a world in which we just can't wait and go full throttle towards a future in which AIs take all the jobs, we fear and dread even the minor but very plausible day in which narrow AI will do the trivial task of driving autonomously all the vehicles on all the roads, because that would imply that 200+ millions of jobs will disappear as if they never were. [1] "Last May, Samsung outlined a plan to pour more than $350 billion into its businesses and create tens of thousands of new jobs through 2026", https://edition.cnn.com/2023/03/15/tech/korea-chips-investment-hnk-intl/index.html https://edition.cnn.com/2023/03/15/tech/korea-chips-investme...
jart 4y ago
> > For the first time in history, it’s become realistic to truly automate intellectual tasks. The leverage this provides is completely unprecedented. > > The essay also mentions how humans are still the best at manual labor, and how government can work with humans directing the AI as it runs things every more efficiently. It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy. Which wolves wielded leverage over the dog community, after humans determined that dogs were better suited to helping them achieve their goals?
slowmovintarget 4y ago
The worst part of this is we haven't produced intelligence. We've produced very good autocomplete. There's no motivating force governing what it spits out, or what inputs it takes in, short of the people prompting or training it. That means it can only do as it's told, and not what is meant by what it's told, and that class of errors, when executed with high efficiency and sophistication, makes all sorts of bad things happen.
- aatd86 4y ago
  Maybe we don't want to produce autonomous intelligence but only human-prompted intelligence. What is the dopaminergic equivalent in AI terms? What may motivate the AI (as a new kind of lifeform)? Perhaps that we want the AI to remain a mere extension of us... The motivation component coming from humans. Then the issue of responsability for the output is also solved. It's the human that prompted the AI.
  - 411111111111111 4y ago
    > Maybe we don't want to produce autonomous intelligence but only human-prompted intelligence. Why wouldn't we want these, if we could achieve that? I've only ever heard FUD arguments against it, and all of them would apply to your suggestion as well. Heck, I'd say true AI is safer then a highly effective autonomous bot that only does what it's told. You'd always be only a single command away from accidentally wiping out all organic life by ordering something like "get some iron"
    - aatd86 4y ago
      So you would run the risk to let it autonomously decide that humans are a threat to it and all humans must be eliminated or coerced into slavery à la Matrix ? The concern doesn't look very far-fetched to me. If the goal of any natural life-form/object is to maximize its energy level while having a stable structure (whether an an atom, a molecule, RNA, a cell, the body, your kinship, a company, a nation, humanity, a solar system etc...), then what would need to be encoded in those AI programs to make them autonomous? And then what decisions would they make as completely different, life forms with their own decision making? Would they empathize with us (which means they would have to have an idea of the human experience) or would they treat us as a lion treats an antelope, as we treat a cow, etc.?
ravenstine 4y ago
> It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy. Bingo. We don't know the answer to these questions yet, and to assume they are solved in a way that is generally equitable to humans is utterly insane.

by mLuby 4y ago

komali2 4y ago

> The author is too used to thinking from first principles and hasn't studied enough human history to understand how leverage gets abused. My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power. Right now power is primarily tied to capital. Access to GPT4 type technologies, or the ability to develop and use these technologies, is tied to same. More money, bigger servers, or, better likelihood to get access to GPT4 before anyone else, GPT5, whatever. Micro$oft can use as much as it wants right now. That capital is tied to power hasn't mattered in any revolution in the past because as soon as the "rabble" realizes this and dissolves capital, by every measure whatever that currency was instantly devalues as other more "true" measures of power are realized, such as rhetorical ability (convincing others to fight and die for your cause), raw strength (convincing others to not bother trying to fight back), or technological advantage and access "in the real" (being the one actually holding the gun). "Numbers" falls in there somewhere but is so tied into the other three measures it's hard to figure out how. Chomsky was onto it with "Manufacturing Consent." Rhetorical ability being one of the most dangerous weapons in a power struggle (used correctly it can direct both the strongest and the ones holding the guns, in greater numbers), leveraging that is the surest way to control the whole population. Before, you had to be Rupert Murdoch to do it, or buy him, which is expensive. Soon you'll just need to buy "Open"AI, or, reproduce their technology and their hardware. As soon as that's done, you can find out all sorts of cost effective methods to do what buying Rupert Murdoch used to do, and you can probably investigate ways to really cement this power. For example, setting up a fascist state, we've tried that a couple times and some people are trying that again, but every fascist state has failed eventually, so it's probably not a sustainable plan. What new, horrible method haven't we tried yet? One that lets us control people as effectively as we have through artificial scarcity (capitalism) but without the downsides of them eventually forming unions and finding ways to fight back within the system, or, reject the system entirely by going to form communes or whatever? Without the inherent contradictions that cause the system to cannibalize itself (depression of wages depressing buying power depressing profit leading to collapse). What creative legal structures can we develop to ensure that copyright can exist forever, but only for your friends? And how can we make sure that this power structure propagates indefinitely to our friends and family? Maybe this means a GPT5 recommending the construction of a Bene Gesserit style cult? Maybe it stumbles upon incredibly effective human-programming Memes, such as what Neal Stephenson warned about in Snowcrash and Fall / Reamde (can't remember which, whichever involved the collapse of the internet to distributed bots). My solution comes from Stephenson and Doctorow: distribute the technology freely. The sooner everyone has access to the incredible propagandic power of these technologies, the sooner we can develop counter measures. The task of the power-aggregator (fascist or whatever) will cease being "give me effective propaganda to easily control a population," it will be "give me propaganda that will bypass the population's GPT filter solutions." An arms race, but a more equal playing field.

throwaway99111 4y ago
>My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power. So, this is potentially a crackpot theory, but hear me out. I've noticed that a lot of the hype runners, posts claiming their businesses are optimizing this with chatgpt, or they've replaced most of their coding with chatgpt outputs have been generally accounts here that were created in 2021 or later. While before the last couple of months, you should generally assume good faith on HN and elsewhere, the fact is mass astroturfing now is very much possible on HN and elsewhere with chatGPT, and why wouldn't OpenAI want to hype their product and get as many users on board? They certainly could and would not be noticeable. You don't need nefarious political goals if you're just trying to sell something. You just need to create enough hype and illusion to grow your product. SV companies have done it before, and now it's automated, to a degree. One explanation (beyond basic confirmation bias) is that a lot of people joined the crypto wave and knew the orange site was the place where tech people hung out, and a lot of those types ditched crypto and are riding the AI wave, but I really cannot be sure anymore.
- ChatGTP 4y ago
  I kind of feel you, everything seems really hyped, more than it should be. I mean I still don't even have any idea what ChatGPTs actual use case is besides look super impressive. It seems like it can do a bunch of stuff, sometimes really well, other times spectacularly bad. It's not an a true AGI, but it's kind of marketed like one. I think we're all completely baffled by it? I mean the thing can talk right? So we're all fired?
  - komali2 4y ago
    As per usual everyone else keeps coming up with the ideas that when I hear, I think, "damn, seems obvious, wish I had thought of that." Sometimes it's because it's a good idea, sometimes because it's an idea that will probably get funding, which makes it a good idea :P Examples: 1. Ingesting a shitload of unread emails after you come back from vacation, then telling you briefly what important things you missed 2. The same idea but for slack 3. Ingesting request-for-cost response emails from various suppliers and outputting the data in a machine-readable way so it can be easily ingested by another API 4. Generating "individual" lesson plans based on student needs (I've heard 5 different pitches around this, majority for language learning) 5. student tutor chatbot There's some other ideas that were too stupid (or more probably, too innovative for dumb old me to comprehend) to remember. At the very least it has people thinking and being imaginative, which I think is pretty cool. I like getting into political debates with the thing.

noobermin 4y ago

To be honest, neither has most people here, most people working at openai, or the hypemen around AI.

XorNot 4y ago
I'm pretty sure the people working at OpenAI have thought about it, and decided that it's better they get there first then some autocracy.
still_grokking 4y ago
I don't think so. Almost all people promoting this stuff do that for their very own advantage in the first place. They want to be the ones with the lever, because they know that this will become a very powerful lever.

Animats 4y ago

> And one possibility might be that AIs could “improve themselves” to produce a single “apex intelligence” that would in a sense dominate everything else. That's not what worries me. It's when AIs learn to distribute work to other AIs and humans. Once that starts to work, machines are going to be better at coordination than humans, simply because they have more I/O bandwidth.

echelon 4y ago
No more middle managers! But seriously, taking middle management self-promotion out of the equation could reduce some of the rotten malinvestment and political in-fighting that happens in large organizations.
- still_grokking 4y ago
  To the AI all humans are "middle managers" though…
- ta1243 4y ago
  Or it could result in the first distopia of "Manna", where humans are there for cleaning, but there's no middle management. https://marshallbrain.com/manna1 https://marshallbrain.com/manna1
still_grokking 4y ago
Yes, being able to perform orders of magnitude more APM than the opponent is a sure win, even when the particular actions aren't optimal.
Faark 4y ago
Imagine an AI spawning many parallel instances of itself, all feeding back into the same conscience. So many coordination problems just gone. All the resources humans spend fighting each other in some form or another... And entirely depending only on itself. No need for any external being. That could prove very deadly to us, since "we need each other" is the core pillar forming our society. A king needs his peasants. An AI? Could just boot up another instance...

ly3xqhl8g9 4y ago

15,687 words in that article and none of them are wisdom or metaphysics. As long as we won't have a change in our metaphysics (assigning value and deriving meaning from 'having a job' and so on), and we still proud ourselves in being "jobs creators" [1], it's all for nothing, i.e., more shareholder value. Because of our current metaphysics, instead of living in a world in which we just can't wait and go full throttle towards a future in which AIs take all the jobs, we fear and dread even the minor but very plausible day in which narrow AI will do the trivial task of driving autonomously all the vehicles on all the roads, because that would imply that 200+ millions of jobs will disappear as if they never were. [1] "Last May, Samsung outlined a plan to pour more than $350 billion into its businesses and create tens of thousands of new jobs through 2026", https://edition.cnn.com/2023/03/15/tech/korea-chips-investment-hnk-intl/index.html https://edition.cnn.com/2023/03/15/tech/korea-chips-investme...

jart 4y ago

> > For the first time in history, it’s become realistic to truly automate intellectual tasks. The leverage this provides is completely unprecedented. > > The essay also mentions how humans are still the best at manual labor, and how government can work with humans directing the AI as it runs things every more efficiently. It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy. Which wolves wielded leverage over the dog community, after humans determined that dogs were better suited to helping them achieve their goals?

slowmovintarget 4y ago

The worst part of this is we haven't produced intelligence. We've produced very good autocomplete. There's no motivating force governing what it spits out, or what inputs it takes in, short of the people prompting or training it. That means it can only do as it's told, and not what is meant by what it's told, and that class of errors, when executed with high efficiency and sophistication, makes all sorts of bad things happen.

aatd86 4y ago
Maybe we don't want to produce autonomous intelligence but only human-prompted intelligence. What is the dopaminergic equivalent in AI terms? What may motivate the AI (as a new kind of lifeform)? Perhaps that we want the AI to remain a mere extension of us... The motivation component coming from humans. Then the issue of responsability for the output is also solved. It's the human that prompted the AI.
- 411111111111111 4y ago
  > Maybe we don't want to produce autonomous intelligence but only human-prompted intelligence. Why wouldn't we want these, if we could achieve that? I've only ever heard FUD arguments against it, and all of them would apply to your suggestion as well. Heck, I'd say true AI is safer then a highly effective autonomous bot that only does what it's told. You'd always be only a single command away from accidentally wiping out all organic life by ordering something like "get some iron"
  - aatd86 4y ago
    So you would run the risk to let it autonomously decide that humans are a threat to it and all humans must be eliminated or coerced into slavery à la Matrix ? The concern doesn't look very far-fetched to me. If the goal of any natural life-form/object is to maximize its energy level while having a stable structure (whether an an atom, a molecule, RNA, a cell, the body, your kinship, a company, a nation, humanity, a solar system etc...), then what would need to be encoded in those AI programs to make them autonomous? And then what decisions would they make as completely different, life forms with their own decision making? Would they empathize with us (which means they would have to have an idea of the human experience) or would they treat us as a lion treats an antelope, as we treat a cow, etc.?

ravenstine 4y ago

> It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy. Bingo. We don't know the answer to these questions yet, and to assume they are solved in a way that is generally equitable to humans is utterly insane.