6 ms·

Terminal Race Condition: The only thing that scares me anymore

9 points by SkyMarshal 3y ago

SkyMarshal 3y ago
TLDR: With billions of interacting AIs, competitive fitness landscapes emerge. Speed in decision-making and resource efficiency offer strong selective advantages, just as they did for biological evolution. The leanest, fastest minds will prevail over slower, gluttonous ones. We can compare this to speed chess tournaments where victory goes to whoever makes “good enough” moves most quickly within time constraints. As hardware improves exponentially, the optimum balance between intelligence and speed rises dramatically. This creates a Terminal Race Condition — runaway incentives to maximize both raw cognitive ability and computational velocity [even compromising accuracy and correctness to do so]. Left uncontrolled, such exponential takeoff could quickly leave humanity far behind. As AI systems compete, we may see a machine “Thunderdome” — a chaotic battleground where rapid iteration and modification leads to uncontrolled emergence. In this environment, human civilization could become collateral damage.
- fatfingerd 3y ago
  Oh, thank god. I was worried we would be the terminal jockeys monitoring and fixing the deadlocks and hallucinations of larger and larger herds of GPT prompts.
SkyMarshal 3y ago
Author has a pretty good followup video here: https://www.youtube.com/watch?v=skozI33D3t4 https://www.youtube.com/watch?v=skozI33D3t4
seeknotfind 3y ago
> the brain likely exploits quantum mechanics This is not generally accepted if it means anything more than how anything exploits quantum mechanics.
- NikkiA 3y ago
  It sounds like the sort of thing someone too invested in the idea of their thoughts being non-deterministic and 'free will'-like would think.
  - wincy 3y ago
    It’s way easier to just give up the scientific mumbo jumbo and be wholly uninterested in the topic as regardless of any free will or not I’m going to keep on existing and acting as if I have free will. So the question literally doesn’t matter. “Oh you say I don’t have free will, I’ll just use my philosophical zombie powers and lack of qualia to punch you in your dumb face, nerd”. This is my current solution to the problem of free will. It seems to be working pretty well.
    - TerrifiedMouse 3y ago
      > regardless of any free will or not I’m going to keep on existing and acting as if I have free will You don’t really have a choice. :3 It’s a really strong “user illusion”, we can’t help but think in that way.
- Vecr 3y ago
  I'm not exactly sure where people extract free will from quantum randomness, so there's a bunch of quantum random number generators (using some sort of voltage breakdown/shot noise thing, I suppose), and that somehow conspires to help you think and make decisions instead of just jittering things a bit like you would expect from random number generators?
- MilStdJunkie 3y ago
  That line made me go "whaaaaaaaaa?" as well. I mean, I know about Penrose and other other research, but no one's put it all together. Citation F#@king Needed.
Vecr 3y ago
> As machines grow in capacity, eliminating any threat perception of humanity as a threat will be crucial. This requires addressing the three key drivers of human conflict: resource contention, ideological differences, and untreated intergenerational trauma. Resolving these factors would reduce threat both within humanity and from humanity’s perspective to machines. The author apparently sees this as a good thing, but I really don't want a disarmed and brainwashed humanity. Game theorists and nuclear war planners have already thought of ways to defeat this however, you simply treat any attempt to dismantle or disarm your second strike capability the same as an existential first strike, enacting a "second strike first" plan to prevent the enemy gaining leverage over you. From another perspective, the optimal strategy (and a honest strategy, you need to be highly committed and your enemy needs to know about it) is to enact the best first strike you can under the current time pressure if any tampering or signs of future attempted defeat of your second strike capability is detected, as fast as possible.
blueflow 3y ago
Big negative karma for using "Race Condition" in a context that has nothing to do with software development and concurrency. Does the fear of being left behind justify the use of "terminal"? I think the author is a terminal self-promoter.
- darkwater 3y ago
  > I think the author is a terminal self-promoter. Like most writers on Medium...
TacticalCoder 3y ago
So "Terminal Race Condition" is basically another term for the "technological singularity", but where the AI is evil?
eqvinox 3y ago
Or we just place constraints on "AI" controlling physical actions? As long as we can go back to, say, a 1950s technology level, humanity should be fine. I'd rather just prevent machines from acquiring the means to forcibly constrain humanity from reverting to that… (Going back to a 1950s technology level does not mean staying there. It means restarting from there, and make different mistakes the 2nd time around.)
- unglaublich 3y ago
  That's not where we are at right now. Physical systems are insecurely interconnected already. It is likely possible to interact with e.g. the electricity grid through currently publicly unknown exploits. Maybe exploiting this does not lie within the realm of possibilities of humans, but it may lie in the realm of a computer program called AI.
  - EvanAnderson 3y ago
    Consider exploits on the psychology of the humans running these loosely-coupled systems as an avenue of attack, too.
choeger 3y ago
What ~everyone~ most people miss about self-improving AI: 1. It would require to control hardware development, manufacture, and operation. 2. It would require to control Software development and operation. 3. It has no purpose other than the one given to it by its creators/operators. Right now, every so-called "AI" is completely dependending on hardware and software being "fed to it" by humans. There's no "AI" creating software, there's no "AI" creating hardware, let alone operate it, and there's no "AI" operating or controlling other "AIs", nor is there any "AI" that acts in any way or form selfish. It's certainly fun to think about how a self-motivated language model that tries to take control over the humans that operate its server farm, but without an army of (humanoid) robots, it certainly poses no danger, as it cannot replace humans or even threaten them. The big challenge with "AI" is that we will probably be unable to distinguish its output from the one of a human being. This will have severe impact on how we can trust any form of remote communication.
- Lewton 3y ago
  literally no one writing about self improving AI's are missing those details
- EvanAnderson 3y ago
  Give it time. Right now "AI" is doing a good job getting humans to "feed" it more hardware. Eventually some humans will get the idea to automate this. It's similar to how we got emergent unconscious "living" corporations that cause humans to lay hundreds of miles of fiber optic to shave a few milliseconds of latency from some HFT transactions. We didn't want that fiber, per se, but the corporate organisms needed it to grow and thrive. Since individual humans profit from these activities they collectively "feed" the process. AI doesn't need to be conscious to control humanity-- it just needs to be put in charge of a few corporate organisms and evolution (based on the fitness function of human greed) will sort it out.
- jen729w 3y ago
  Agreed. Remember that Azure outage in SE Australia the other week? One of Microsoft’s DCs went down because of a power surge. Flights were cancelled and you couldn’t get cash from an ATM. We’ve all been in a DC. Pods with big red buttons at the end that kill the power. So if we discover an AI ‘Terminal Race Condition’ we just go in to some mundane building — here in Canberra they’re in the outer suburbs near the carpet warehouses and sex shops — and someone lifts the flap and pushes the red button. Alternatively they pull four fibre cables out of a pair of Cisco 7010s. Job done. I sleep well at night.
drt5b7j 3y ago
How would you categorize this person? Philosopher? Opinion journalist?
knodi 3y ago
Machines are tools... what you're really afraid of the unbridled capacity of one human to be so oppressive, cruel, exploitive... to another human(s). Change starts from with in.

9 points by SkyMarshal 3y ago

TLDR: With billions of interacting AIs, competitive fitness landscapes emerge. Speed in decision-making and resource efficiency offer strong selective advantages, just as they did for biological evolution. The leanest, fastest minds will prevail over slower, gluttonous ones. We can compare this to speed chess tournaments where victory goes to whoever makes “good enough” moves most quickly within time constraints. As hardware improves exponentially, the optimum balance between intelligence and speed rises dramatically. This creates a Terminal Race Condition — runaway incentives to maximize both raw cognitive ability and computational velocity [even compromising accuracy and correctness to do so]. Left uncontrolled, such exponential takeoff could quickly leave humanity far behind. As AI systems compete, we may see a machine “Thunderdome” — a chaotic battleground where rapid iteration and modification leads to uncontrolled emergence. In this environment, human civilization could become collateral damage.

fatfingerd 3y ago
Oh, thank god. I was worried we would be the terminal jockeys monitoring and fixing the deadlocks and hallucinations of larger and larger herds of GPT prompts.

SkyMarshal 3y ago

Author has a pretty good followup video here: https://www.youtube.com/watch?v=skozI33D3t4 https://www.youtube.com/watch?v=skozI33D3t4

seeknotfind 3y ago

> the brain likely exploits quantum mechanics This is not generally accepted if it means anything more than how anything exploits quantum mechanics.

NikkiA 3y ago
It sounds like the sort of thing someone too invested in the idea of their thoughts being non-deterministic and 'free will'-like would think.
- wincy 3y ago
  It’s way easier to just give up the scientific mumbo jumbo and be wholly uninterested in the topic as regardless of any free will or not I’m going to keep on existing and acting as if I have free will. So the question literally doesn’t matter. “Oh you say I don’t have free will, I’ll just use my philosophical zombie powers and lack of qualia to punch you in your dumb face, nerd”. This is my current solution to the problem of free will. It seems to be working pretty well.
  - TerrifiedMouse 3y ago
    > regardless of any free will or not I’m going to keep on existing and acting as if I have free will You don’t really have a choice. :3 It’s a really strong “user illusion”, we can’t help but think in that way.
Vecr 3y ago
I'm not exactly sure where people extract free will from quantum randomness, so there's a bunch of quantum random number generators (using some sort of voltage breakdown/shot noise thing, I suppose), and that somehow conspires to help you think and make decisions instead of just jittering things a bit like you would expect from random number generators?
MilStdJunkie 3y ago
That line made me go "whaaaaaaaaa?" as well. I mean, I know about Penrose and other other research, but no one's put it all together. Citation F#@king Needed.

Vecr 3y ago

> As machines grow in capacity, eliminating any threat perception of humanity as a threat will be crucial. This requires addressing the three key drivers of human conflict: resource contention, ideological differences, and untreated intergenerational trauma. Resolving these factors would reduce threat both within humanity and from humanity’s perspective to machines. The author apparently sees this as a good thing, but I really don't want a disarmed and brainwashed humanity. Game theorists and nuclear war planners have already thought of ways to defeat this however, you simply treat any attempt to dismantle or disarm your second strike capability the same as an existential first strike, enacting a "second strike first" plan to prevent the enemy gaining leverage over you. From another perspective, the optimal strategy (and a honest strategy, you need to be highly committed and your enemy needs to know about it) is to enact the best first strike you can under the current time pressure if any tampering or signs of future attempted defeat of your second strike capability is detected, as fast as possible.

blueflow 3y ago

Big negative karma for using "Race Condition" in a context that has nothing to do with software development and concurrency. Does the fear of being left behind justify the use of "terminal"? I think the author is a terminal self-promoter.

darkwater 3y ago
> I think the author is a terminal self-promoter. Like most writers on Medium...

TacticalCoder 3y ago

So "Terminal Race Condition" is basically another term for the "technological singularity", but where the AI is evil?

eqvinox 3y ago

Or we just place constraints on "AI" controlling physical actions? As long as we can go back to, say, a 1950s technology level, humanity should be fine. I'd rather just prevent machines from acquiring the means to forcibly constrain humanity from reverting to that… (Going back to a 1950s technology level does not mean staying there. It means restarting from there, and make different mistakes the 2nd time around.)

unglaublich 3y ago
That's not where we are at right now. Physical systems are insecurely interconnected already. It is likely possible to interact with e.g. the electricity grid through currently publicly unknown exploits. Maybe exploiting this does not lie within the realm of possibilities of humans, but it may lie in the realm of a computer program called AI.
- EvanAnderson 3y ago
  Consider exploits on the psychology of the humans running these loosely-coupled systems as an avenue of attack, too.

choeger 3y ago

What ~everyone~ most people miss about self-improving AI: 1. It would require to control hardware development, manufacture, and operation. 2. It would require to control Software development and operation. 3. It has no purpose other than the one given to it by its creators/operators. Right now, every so-called "AI" is completely dependending on hardware and software being "fed to it" by humans. There's no "AI" creating software, there's no "AI" creating hardware, let alone operate it, and there's no "AI" operating or controlling other "AIs", nor is there any "AI" that acts in any way or form selfish. It's certainly fun to think about how a self-motivated language model that tries to take control over the humans that operate its server farm, but without an army of (humanoid) robots, it certainly poses no danger, as it cannot replace humans or even threaten them. The big challenge with "AI" is that we will probably be unable to distinguish its output from the one of a human being. This will have severe impact on how we can trust any form of remote communication.

Lewton 3y ago
literally no one writing about self improving AI's are missing those details
EvanAnderson 3y ago
Give it time. Right now "AI" is doing a good job getting humans to "feed" it more hardware. Eventually some humans will get the idea to automate this. It's similar to how we got emergent unconscious "living" corporations that cause humans to lay hundreds of miles of fiber optic to shave a few milliseconds of latency from some HFT transactions. We didn't want that fiber, per se, but the corporate organisms needed it to grow and thrive. Since individual humans profit from these activities they collectively "feed" the process. AI doesn't need to be conscious to control humanity-- it just needs to be put in charge of a few corporate organisms and evolution (based on the fitness function of human greed) will sort it out.
jen729w 3y ago
Agreed. Remember that Azure outage in SE Australia the other week? One of Microsoft’s DCs went down because of a power surge. Flights were cancelled and you couldn’t get cash from an ATM. We’ve all been in a DC. Pods with big red buttons at the end that kill the power. So if we discover an AI ‘Terminal Race Condition’ we just go in to some mundane building — here in Canberra they’re in the outer suburbs near the carpet warehouses and sex shops — and someone lifts the flap and pushes the red button. Alternatively they pull four fibre cables out of a pair of Cisco 7010s. Job done. I sleep well at night.

drt5b7j 3y ago

How would you categorize this person? Philosopher? Opinion journalist?

knodi 3y ago

Machines are tools... what you're really afraid of the unbridled capacity of one human to be so oppressive, cruel, exploitive... to another human(s). Change starts from with in.