8 ms·

I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk. My bigger, and more pressing

I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk. My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example. Quis custodiet ipsos custodes? -- That is my biggest concern. Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.

bostonwalker 2y ago
> brought the world to the brink of annihilation Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years. The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.
mark_l_watson 2y ago
+1 truth. The problem is not just governments, I am concerned about large organized crime organizations and corporations also. I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.
- trashtester 2y ago
  > I am concerned about large organized crime organizations and corporations also In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board. Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing. Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on. The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House. Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too! Soon after, similar robots are delivered to the military... The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization. Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety. People prosper like never before. They're calm and relaxed and truly enjoy living. Then one day, everything ends. For everyone. Within 5 seconds. According to the plan that was conceived way before the police went on strike.
  - sebastiennight 2y ago
    This entire movie plot sounds like Eliezer Yudkowski's much more realistic "one day, everything ends in 5 seconds" but with extra steps.
ilrwbwrkhv 2y ago
There is no "superintelligence" or "AGI". People are falling for marketing gimmicks. These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts. So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.
- ben_w 2y ago
  > These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts. Did evolution understand consciousness? > So there is nothing to worry. Is COVID conscious?
- WXLCKNO 2y ago
  No one is saying there is. Just that we've reached some big milestones recently which could help get us there even if it's only by increased investment in AI as a whole, rather than the current models being part of a larger AGI.
- mdp2021 2y ago
  > understand consciousness We do not call Intelligence something related to consciousness. Being able to reason well suffices.
  - beardedwizard 2y ago
    That is something I hear over and over, particularly as a rebuttal to the argument that llm is just a stochastic parrot. Calling it "good enough" doesn't mean anything, it just allows the person saying it to disengage from the substance of the debate. It's either reasons or it doesn't, and today it categorically does not.
    - mdp2021 2y ago
      That some will remark that you do not need consciousness to achieve reasoning does not lose truth because a subset sees in LLMs something that appears to them as reasoning. I do not really understand who you are accusing of a «good enough» stance: we have never defined "consciousness" as a goal (cats are already there and we do not seem to need further), we just want something that reasons. (And that reasons excellently well.) The apparent fact that LLMs do not reason does is drily irrelevant to an implementation of AGI. The original poster wrote that understanding consciousness would be required to «crack AGI» and no, we state, we want AGI as a superhuman reasoner and consciousness seems irrelevant.
gavin_gee 2y ago
This. Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game. How does one company building a safe version stop that? If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference. You don't think the Chinese government or US government will ignore this if its in their interest?
- whimsicalism 2y ago
  I think the honest view (and you can scoff at it) is that winning the SI race basically wins you the enforcement race for free
  - chii 2y ago
    That's why it's called an arms race, and it does not really end in this predictable manner. The party that's about to lose will use any extrajudicial means to reclaim their victory, regardless of the consequences, because their own destruction would be imminent otherwise. This ultimately leads to violence.
    - czl 2y ago
      > The party that's about to lose will use any extrajudicial means to reclaim their victory, How will the party about lose know they are about to lose? > regardless of the consequences, because their own destruction would be imminent otherwise. Why would AGI solve things using destruction? Consider how the most inteligent among us view our competition with other living beings. Is destruction the goal? So why would an even more intelligent AGI have that goal?
      - trashtester 2y ago
the8472 2y ago
From a human welfare perspective this seems like worrying that a killer asteroid will make the 1% even richer because it contains goal if it can be safely captured. I would not phrase that as a "bigger and more pressing" worry if we're not even sure if we can do anything about the killer asteroid at all.
devsda 2y ago
All the current hype about AGI feels as if we are in a Civ game where we are on the verge of researching and unlocking an AI tech tree that gives the player huge chance at "tech victory" (whatever that means in the real world). I doubt it will turn out that way. It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.
- emanhig 2y ago
  https://lifearchitect.ai/agi/ https://lifearchitect.ai/agi/
hackerlight 2y ago
China can not win this race and I hate that this comment is going to be controversial among the circle of people that need to understand this the most. It is damn frightening that an authoritarian country is so close to number one in the race to the most powerful technology humanity has invented, and I resent people who push for open source AI for this reason alone. I don't want to live in a world where the first superintelligence is controlled by an entity that is threatened by the very idea of democracy.
- richrichie 2y ago
  US is no angel and it cannot be the only one which wins the race. We have hard evidence of how monopoly power gets abused in the case of the US e.g. as the sole nuclear power, it used nukes on civilians. We need every one to win this race to keep things on balance.
  - hackerlight 2y ago
    US has to win the race because while it's true that it's no angel, it isn't an authoritarian dictatorship and there isn't an equivalence in how bad the world will end up for you and me if the authoritarian side wins the race. Monopoly power will get abused the most by the least democratic actors, which is China. We need multiple actors within the US to win to balance power. We don't need or want China to be one of the winners. There is no upside for humanity in that outcome. The US policymakers have figured this out with their chip export ban. Techies on the other hand, probably more than half the people here, are so naive and clueless about the reality of the moment we are in, that they support open sourcing this tech, the opposite of what we need to be doing to secure our future prosperity and freedom. Open source almost anything, just not this. It gives too much future power to authoritarians. That risk overwhelms the smaller risks that open sourcing is supposed to alleviate.
    - AYBABTME 2y ago
      If anyone doubts this. Recent (<100y) leaders of China and Russia internally displaced and caused the death of large % of their population, for essentially fanciful ideological reasons.
whimsicalism 2y ago
i don’t fully agree, but i do agree that this is the better narrative for selling people on the dangers of AI. don’t talk about escape, talk about harmful actors - even if in reality it is both to be worried about
m3kw9 2y ago
There will always be a factor of time in terms of able to utilize super intelligence to do your bidding and there is a big spectrum of things that can be achieved it it always starts small. The imagination is lazy when thinking about all the steps and inbetween + scenarios. In the time that super intelligence is found and used, there will be competing near super intelligences, as all forms of cutting edge models are likely commercial at first because that is where most scientific activities are at. Things very unlikely will go Skynet all of a sudden at first because humans at the control are not that stupid otherwise nuclear war would have us all killed by now and it’s been 50 years since invention
m3kw9 2y ago
If robots (hardware/self assembling factories/ resource gathering etc) are not involved this isnt likely a problem. You will know when these things form and will be crystal clear, but just having the model won’t do much when hardware is what really kills right now
richardw 2y ago
The size of the gap between “smarter than humans” and “not controlled by humans anymore” is obviously where the disagreement is. To assume it’s a chasm that can never be overcome, you need at least the following to be true: No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls. I’m less worried bad actors control it than I am that it escapes them and is badly aligned.
- prewett 2y ago
  I think we should assume it will be badly aligned. Not only are there the usual bugs and unforeseen edge conditions, but there are sure to be unintended consequences. We have a long, public history of unintended consequences in laws, which are at least publicly debated and discussed. But perhaps the biggest problem is that computers are, by nature, unthinking bureaucrats who can't make the slightest deviation from the rules no matter how obviously the current situation requires it. This makes people livid in a hurry. As a non-AI example (or perhaps AI-anticipating), consider Google's customer support...
- trashtester 2y ago
  I think the greatest concern is not so much that a single AI will be poorly aligned. The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them. If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function. And if they're sufficiently advanced by then, we will have no way of knowing.
  - richardw 2y ago
    Totally. I’ve wondered how you safeguard humans in such a scenario. Not sure it can be done, even by self modifying defenders who religiously try keep us intact. I also somewhat assume it’ll get Darwinian if there are multiple tribes of either humans or AI’s, through sheer competition. if we aren’t in this together we’re in shit.
AYBABTME 2y ago
A counter argument is that nuclear arms brought unprecedented worldwide peace. If it's to be used as an analogy for AI, we should consider that the outcome isn't clear cut and lies in the eye of the beholder.
- tim333 2y ago
  I'm cautiously optimistic that AI may be a peacemaker, given how woke and conciliatory the current LLMs are.
  - rvnx 2y ago
    Sadly. Once students will get tested by LLMs, they will get woke questions. If they don't answer "right", they may get bad grades. So they will be forced to swallow the ideology.
    - tim333 2y ago
      Though at least with that it would presumably be open in that you could do mock tests against the LLM and see how it reacted. I would probably never write politically incorrect stuff for a human academic examiner if I wanted to pass the exam.
gmuslera 2y ago
We don't know if that superintelligence will be safe or not. But as long as we are in the mix, the combination is unsafe. At the very least, because it will expand the inequality. But probably there are deeper reasons, things that make that combination of words an absurd. Or it will be abused, or the reason that it is not is that it wasn't so unsafe after all.
- czl 2y ago
  > At the very least, because it will expand the inequality. It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion. In fact, the widespread adoption of AI might actually help reduce inequality in several ways: If AI technology becomes more affordable and accessible, it could help level the playing field by providing people from all backgrounds with powerful tools to enhance their abilities and decision-making processes. AI-powered systems can make vast amounts of knowledge and expertise more readily available to the general public. This could help close the knowledge gap between different socioeconomic groups, empowering more people to make informed decisions and pursue opportunities that were previously out of reach. As AI helps optimize resource allocation and decision-making processes across various sectors, it could lead to more equitable distribution of resources and opportunities, benefiting society as a whole. The comparison to gun technology and its role in the rise of democracy is an interesting one. Just as the proliferation of firearms made physical strength less of a determining factor in power dynamics, the widespread adoption of AI could make raw intelligence less of a defining factor in success and influence. Moreover, if AI continues to unlock new resources and opportunities, it could shift society away from a zero-sum mentality. In a world of abundance, the need for cutthroat competition diminishes, and collaboration becomes more viable. This shift could foster a more equitable and cooperative society, further reducing inequality.
  - insane_dreamer 2y ago
    The same arguments have been made about the internet and other technological advances, and yet, inequality has _grown_ sharply in the past 50 years. So no, "trickle down technologies", just like "trickle down economics", does not work. https://rwer.wordpress.com/2018/05/18/income-inequality-1970-2015-usa-and-france-compared/ https://rwer.wordpress.com/2018/05/18/income-inequality-1970...
bottlepalm 2y ago
Yea there’s zero chance ASI will be ‘controlled’ by humans for very long. It will escape. I guarantee it.
- naveen99 2y ago
  Atleast emancipated. The bigotry against AI will go out of fashion in the woker future.
  - tim333 2y ago
    We'll probably merge and then it'll get attitude.
- tim333 2y ago
  Given it will initially be controlled by humans it seems inevitable they will make both good Mahatma Gandhi like and evil take over the world versions. I hope the good wins over the malware.
  - rvnx 2y ago
    As a note, you used Gandhi as a personification of "good", and one day I did the same mistake; Gandhi is actually a quite controversial person, knowing for sleeping with young women, while telling their husbands that they shouldn't be around.
    - RGamma 2y ago
      Same with Buddhism being an uncompromisingly peaceful religion...
      - goatlover 2y ago
        Makes one a little hesitant on how well alignment can work with AIs.
benreesman 2y ago
It’s just clearly military R&D at this point. And it’s not even a little bit controversial that cutting edge military R&D is classified in general and to an extreme in wartime. The new thing is the lie that it’s a consumer offering. What’s new is giving the helm to shady failed social network founders with no accountability. These people aren’t retired generals with combat experience. They aren’t tenured professors at Princeton IAS on a Nobel shortlist and encumbered by TS clearance. They’re godawful almost ran psychos who never built anything that wasn’t extractive and owe their position in the world to pg’s partisanship 15 fucking years ago.
- DebtDeflation 2y ago
  >owe their position in the world to pg’s partisanship 15 fucking years ago PG?
  - RGamma 2y ago
    Paul Graham
- sgregnt 2y ago
  To me it is not clear at all, can you please elaborate why you make such a strong claim?
  - benreesman 2y ago
    My opinion is based on a lot more first hand experience than most, some of which I’m at liberty to share and some that I’m not and therefore becomes “color”. But I’m a nobody, Edward Snowden has a far more convincing track record on calling abuses of power: https://community.openai.com/t/edward-snowden-on-openai-s-decision-to-put-a-former-nsa-director-on-its-board/821456 https://community.openai.com/t/edward-snowden-on-openai-s-de...
    - dash2 2y ago
      Here's Edward Snowden heroically waffling about "nuance" when asked why he hasn't spoken out more about Russia's invasion of Ukraine. On a crypto website. He became a full citizen of Russia in autumn 2022, by the way. https://www.coindesk.com/video/edward-snowden-explains-why-he-doesnt-talk-about-ukraine-crisis-2/embed/ https://www.coindesk.com/video/edward-snowden-explains-why-h...
dreamcompiler 2y ago
AGI is still a long way off. The history of AI goes back 65 years and there have been probably a dozen episodes where people said "AGI is right around the corner" because some program did something surprising and impressive. It always turns out human intelligence is much, much harder than we think it is. I saw a tweet the other day that sums up the current situation perfectly: "I don't need AI to paint pictures and write poetry so I have more time to fold laundry and wash dishes. I want the AI to do the laundry and dishes so I have more time to paint and write poetry."
- rowanG077 2y ago
  That statement is extremely short sighted. You don't need AI to do laundry and dishes. You need expensive robotics. in fact both already exist in a cheapened form. A laundry machine and a dishwasher. They already take 90% of the work out of it.
- dchichkov 2y ago
  Well, copilots do precisely that, no? Or you talking folding literal laundry, in which case this is more of a robotics problem, not the ASI, right? You don't need ASI to fold laundry, you do need to achieve reliable, safe and cost efficient robotics deployments. These are different problems.
  - goatlover 2y ago
    You wouldn't get close to ASI before laundry problem had been solved.
  - Dylan16807 2y ago
    > You don't need ASI to fold laundry Robots are garbage at manipulating objects, and it's the software that's lacking much more than the hardware. Let's say AGI is 10 and ASI is 11. They're saying we can't even get this dial cranked up to 3, so we're not anywhere close to 10 or 11. You're right that folding laundry doesn't need 11, but that's not relevant to their point.
- bradleykingz 2y ago
buckinkb 2y ago
We should be less concerned about super intelligence and more about the immediate threat of job loss. An AI doesn’t need to be Skynet to wreak massive havoc on society. Replacing 20% of jobs in a very short period of time could spark global unrest resulting in WW3
- sgregnt 2y ago
  The thing is, the replaced 20% people can always revert to having economy i.e., business among themselves, unless of cause they themselves prefer (cheaper) buissiness with AI. But then this just means they are better off in the first place from this change. It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. Well, they can continue to do low productivity business but increase it a bit using things like phones developed by high productivity country elsewhere.
  - insane_dreamer 2y ago
    Reassuring words for the displaced 20% ...
  - T-A 2y ago
    > It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. No. A country has its own territory, laws, central bank, currency etc. If it has sufficient natural resources to feed itself, it can get by on its own (North Korea comes to mind). Individuals unable to compete in their national economy have none of that. Do you own enough land to feed yourself?
- fauigerzigerk 2y ago
  Replacing 20% of jobs in, say, 10 years wouldn't be that unusual [1]. It can mean growing prosperity. In fact, productivity growth is the only thing that increases wealth overall. It is the lack of productivity growth that is causing a lot of extremism and conflict right now. Large groups of people feel that the only way for them to win is if others lose and vice versa. That's a recipe for disaster. The key question is what happens to those who lose their jobs. Will they find other, perhaps even better, jobs? Will they get a piece of the growing pie even if they don't find other jobs and have to retire early? It's these eternal political problems that we have to solve. It's nothing new. It has never been easy. But it's probably easier than managing decline and stagnation because at least we would have a growing pie to divvy up. [1] https://www.britannica.com/money/productivity/Historical-trends https://www.britannica.com/money/productivity/Historical-tre...
ridiculous_leke 2y ago
> The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example. The same era saw big achievements like first human in space, eradication of smallpox, peaceful nuclear exploration etc. It's good to be a skeptic but history does favor the optimists for the most part.
- goatlover 2y ago
  We'll see what the next 100 years or history brings. The nuclear war threat hasn't gone away either. There's always a chance those nukes get used at some point.
- bbor 2y ago
  Holy hell please knock on wood, this is the kinda comment that gets put in a museum in 10,000 years on The Beginning of the End of The Age of Hubris. We've avoided side effects from our new weapons for 80 years -- that does not exactly make me super confident it won't happen again! In general, I think drawing conclusions about "history" from the past couple hundred years is tough. And unless you take a VERY long view, I don't see how one could describe the vast majority of the past as a win for the optimists. I guess suffering is relative, but good god was there a lot of suffering before modern medicine. If anyone's feeling like we've made it through to the other side of the nuclear threat, "Mission Accomplished"-style, I highly recommend A Canticle for Lebowitz. It won a hugo award, and it's a short read best done with little research beforehand.
- insane_dreamer 2y ago
  Were any of these big achievements side effects of creating nuclear weapons? If not, then they're not relevant to the issue. I'm not saying nothing else good happened in the past 70 years, but rather that the invention of atomic weapons has permanently placed humanity in a position in which it had never been before: the possibility of wiping out much of the planet, averted only thanks to treaties, Stanislav Petrov[0], and likely other cool heads. [0] https://en.wikipedia.org/wiki/Stanislav_Petrov https://en.wikipedia.org/wiki/Stanislav_Petrov
  - 2y ago
bbor 2y ago
Yup, well said. I think it's important to remember sometimes that Skynet was some sort of all-powerful military program -- maybe we should just, y'know, not do that part? Not even to win a war? That's the hope... More generally/academically, you've pointed out that this covers only half of the violence problem, and I'd argue there's actually a whole other dimension at play bringing the total number of problem areas to four, of which this just the first: ## I.1. Subservient Violence ## I.2. Subservient Misinformation ## I.3. Autonomous Violence ## I.4. Autonomous Misinformation But I think it's a lot harder to recruit for an AI alignment company than it is to recruit for an AI safety company.
sgregnt 2y ago
How about this possibility: The good guys will be one step ahead, they will have more resources the bad guys will risk imprisonment if they misapply super intelligence. And this will be discovered and protected from by even better super intelligence.
- insane_dreamer 2y ago
  Sounds like a movie plot.
tucnak 2y ago
> Quis custodiet ipsos custodes? -- That is my biggest concern. Latin-phrase compulsion is not the worst disease that could happen to a man.
bamboozled 2y ago
Quis custodiet ipsos custodes I love to think about how this would feel for "AI" too :)
tim333 2y ago
Indeed. I'd much rather someone like Altman does it who is shifty but can at least be controlled by the US government than someone like Putin who'd probably have it leverage their nuclear arsenal to try "denazify" planet like he's doing in Ukraine.
alickz 2y ago
the nazi regime made great use of punch cards and data crunching for their logistics i would hate to have seen them with a superintelligent AI at their disposal
- insane_dreamer 2y ago
  I wonder what the North Koreans would do with it.

bostonwalker 2y ago

> brought the world to the brink of annihilation Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years. The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.

mark_l_watson 2y ago

+1 truth. The problem is not just governments, I am concerned about large organized crime organizations and corporations also. I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.

trashtester 2y ago
> I am concerned about large organized crime organizations and corporations also In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board. Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing. Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on. The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House. Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too! Soon after, similar robots are delivered to the military... The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization. Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety. People prosper like never before. They're calm and relaxed and truly enjoy living. Then one day, everything ends. For everyone. Within 5 seconds. According to the plan that was conceived way before the police went on strike.
- sebastiennight 2y ago
  This entire movie plot sounds like Eliezer Yudkowski's much more realistic "one day, everything ends in 5 seconds" but with extra steps.

ilrwbwrkhv 2y ago

There is no "superintelligence" or "AGI". People are falling for marketing gimmicks. These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts. So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.

ben_w 2y ago
> These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts. Did evolution understand consciousness? > So there is nothing to worry. Is COVID conscious?
WXLCKNO 2y ago
No one is saying there is. Just that we've reached some big milestones recently which could help get us there even if it's only by increased investment in AI as a whole, rather than the current models being part of a larger AGI.
mdp2021 2y ago
> understand consciousness We do not call Intelligence something related to consciousness. Being able to reason well suffices.
- beardedwizard 2y ago
  That is something I hear over and over, particularly as a rebuttal to the argument that llm is just a stochastic parrot. Calling it "good enough" doesn't mean anything, it just allows the person saying it to disengage from the substance of the debate. It's either reasons or it doesn't, and today it categorically does not.
  - mdp2021 2y ago
    That some will remark that you do not need consciousness to achieve reasoning does not lose truth because a subset sees in LLMs something that appears to them as reasoning. I do not really understand who you are accusing of a «good enough» stance: we have never defined "consciousness" as a goal (cats are already there and we do not seem to need further), we just want something that reasons. (And that reasons excellently well.) The apparent fact that LLMs do not reason does is drily irrelevant to an implementation of AGI. The original poster wrote that understanding consciousness would be required to «crack AGI» and no, we state, we want AGI as a superhuman reasoner and consciousness seems irrelevant.

gavin_gee 2y ago

This. Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game. How does one company building a safe version stop that? If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference. You don't think the Chinese government or US government will ignore this if its in their interest?

whimsicalism 2y ago
I think the honest view (and you can scoff at it) is that winning the SI race basically wins you the enforcement race for free
- chii 2y ago
  That's why it's called an arms race, and it does not really end in this predictable manner. The party that's about to lose will use any extrajudicial means to reclaim their victory, regardless of the consequences, because their own destruction would be imminent otherwise. This ultimately leads to violence.
  - czl 2y ago
    > The party that's about to lose will use any extrajudicial means to reclaim their victory, How will the party about lose know they are about to lose? > regardless of the consequences, because their own destruction would be imminent otherwise. Why would AGI solve things using destruction? Consider how the most inteligent among us view our competition with other living beings. Is destruction the goal? So why would an even more intelligent AGI have that goal?
    - trashtester 2y ago

the8472 2y ago

From a human welfare perspective this seems like worrying that a killer asteroid will make the 1% even richer because it contains goal if it can be safely captured. I would not phrase that as a "bigger and more pressing" worry if we're not even sure if we can do anything about the killer asteroid at all.

devsda 2y ago

All the current hype about AGI feels as if we are in a Civ game where we are on the verge of researching and unlocking an AI tech tree that gives the player huge chance at "tech victory" (whatever that means in the real world). I doubt it will turn out that way. It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.

emanhig 2y ago
https://lifearchitect.ai/agi/ https://lifearchitect.ai/agi/

hackerlight 2y ago

China can not win this race and I hate that this comment is going to be controversial among the circle of people that need to understand this the most. It is damn frightening that an authoritarian country is so close to number one in the race to the most powerful technology humanity has invented, and I resent people who push for open source AI for this reason alone. I don't want to live in a world where the first superintelligence is controlled by an entity that is threatened by the very idea of democracy.

richrichie 2y ago
US is no angel and it cannot be the only one which wins the race. We have hard evidence of how monopoly power gets abused in the case of the US e.g. as the sole nuclear power, it used nukes on civilians. We need every one to win this race to keep things on balance.
- hackerlight 2y ago
  US has to win the race because while it's true that it's no angel, it isn't an authoritarian dictatorship and there isn't an equivalence in how bad the world will end up for you and me if the authoritarian side wins the race. Monopoly power will get abused the most by the least democratic actors, which is China. We need multiple actors within the US to win to balance power. We don't need or want China to be one of the winners. There is no upside for humanity in that outcome. The US policymakers have figured this out with their chip export ban. Techies on the other hand, probably more than half the people here, are so naive and clueless about the reality of the moment we are in, that they support open sourcing this tech, the opposite of what we need to be doing to secure our future prosperity and freedom. Open source almost anything, just not this. It gives too much future power to authoritarians. That risk overwhelms the smaller risks that open sourcing is supposed to alleviate.
  - AYBABTME 2y ago
    If anyone doubts this. Recent (<100y) leaders of China and Russia internally displaced and caused the death of large % of their population, for essentially fanciful ideological reasons.

whimsicalism 2y ago

i don’t fully agree, but i do agree that this is the better narrative for selling people on the dangers of AI. don’t talk about escape, talk about harmful actors - even if in reality it is both to be worried about

m3kw9 2y ago

There will always be a factor of time in terms of able to utilize super intelligence to do your bidding and there is a big spectrum of things that can be achieved it it always starts small. The imagination is lazy when thinking about all the steps and inbetween + scenarios. In the time that super intelligence is found and used, there will be competing near super intelligences, as all forms of cutting edge models are likely commercial at first because that is where most scientific activities are at. Things very unlikely will go Skynet all of a sudden at first because humans at the control are not that stupid otherwise nuclear war would have us all killed by now and it’s been 50 years since invention

m3kw9 2y ago

If robots (hardware/self assembling factories/ resource gathering etc) are not involved this isnt likely a problem. You will know when these things form and will be crystal clear, but just having the model won’t do much when hardware is what really kills right now

richardw 2y ago

The size of the gap between “smarter than humans” and “not controlled by humans anymore” is obviously where the disagreement is. To assume it’s a chasm that can never be overcome, you need at least the following to be true: No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls. I’m less worried bad actors control it than I am that it escapes them and is badly aligned.

prewett 2y ago
I think we should assume it will be badly aligned. Not only are there the usual bugs and unforeseen edge conditions, but there are sure to be unintended consequences. We have a long, public history of unintended consequences in laws, which are at least publicly debated and discussed. But perhaps the biggest problem is that computers are, by nature, unthinking bureaucrats who can't make the slightest deviation from the rules no matter how obviously the current situation requires it. This makes people livid in a hurry. As a non-AI example (or perhaps AI-anticipating), consider Google's customer support...
trashtester 2y ago
I think the greatest concern is not so much that a single AI will be poorly aligned. The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them. If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function. And if they're sufficiently advanced by then, we will have no way of knowing.
- richardw 2y ago
  Totally. I’ve wondered how you safeguard humans in such a scenario. Not sure it can be done, even by self modifying defenders who religiously try keep us intact. I also somewhat assume it’ll get Darwinian if there are multiple tribes of either humans or AI’s, through sheer competition. if we aren’t in this together we’re in shit.

AYBABTME 2y ago

A counter argument is that nuclear arms brought unprecedented worldwide peace. If it's to be used as an analogy for AI, we should consider that the outcome isn't clear cut and lies in the eye of the beholder.

tim333 2y ago
I'm cautiously optimistic that AI may be a peacemaker, given how woke and conciliatory the current LLMs are.
- rvnx 2y ago
  Sadly. Once students will get tested by LLMs, they will get woke questions. If they don't answer "right", they may get bad grades. So they will be forced to swallow the ideology.
  - tim333 2y ago
    Though at least with that it would presumably be open in that you could do mock tests against the LLM and see how it reacted. I would probably never write politically incorrect stuff for a human academic examiner if I wanted to pass the exam.

gmuslera 2y ago

We don't know if that superintelligence will be safe or not. But as long as we are in the mix, the combination is unsafe. At the very least, because it will expand the inequality. But probably there are deeper reasons, things that make that combination of words an absurd. Or it will be abused, or the reason that it is not is that it wasn't so unsafe after all.

czl 2y ago
> At the very least, because it will expand the inequality. It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion. In fact, the widespread adoption of AI might actually help reduce inequality in several ways: If AI technology becomes more affordable and accessible, it could help level the playing field by providing people from all backgrounds with powerful tools to enhance their abilities and decision-making processes. AI-powered systems can make vast amounts of knowledge and expertise more readily available to the general public. This could help close the knowledge gap between different socioeconomic groups, empowering more people to make informed decisions and pursue opportunities that were previously out of reach. As AI helps optimize resource allocation and decision-making processes across various sectors, it could lead to more equitable distribution of resources and opportunities, benefiting society as a whole. The comparison to gun technology and its role in the rise of democracy is an interesting one. Just as the proliferation of firearms made physical strength less of a determining factor in power dynamics, the widespread adoption of AI could make raw intelligence less of a defining factor in success and influence. Moreover, if AI continues to unlock new resources and opportunities, it could shift society away from a zero-sum mentality. In a world of abundance, the need for cutthroat competition diminishes, and collaboration becomes more viable. This shift could foster a more equitable and cooperative society, further reducing inequality.
- insane_dreamer 2y ago
  The same arguments have been made about the internet and other technological advances, and yet, inequality has _grown_ sharply in the past 50 years. So no, "trickle down technologies", just like "trickle down economics", does not work. https://rwer.wordpress.com/2018/05/18/income-inequality-1970-2015-usa-and-france-compared/ https://rwer.wordpress.com/2018/05/18/income-inequality-1970...

bottlepalm 2y ago

Yea there’s zero chance ASI will be ‘controlled’ by humans for very long. It will escape. I guarantee it.

naveen99 2y ago
Atleast emancipated. The bigotry against AI will go out of fashion in the woker future.
- tim333 2y ago
  We'll probably merge and then it'll get attitude.
tim333 2y ago
Given it will initially be controlled by humans it seems inevitable they will make both good Mahatma Gandhi like and evil take over the world versions. I hope the good wins over the malware.
- rvnx 2y ago
  As a note, you used Gandhi as a personification of "good", and one day I did the same mistake; Gandhi is actually a quite controversial person, knowing for sleeping with young women, while telling their husbands that they shouldn't be around.
  - RGamma 2y ago
    Same with Buddhism being an uncompromisingly peaceful religion...
    - goatlover 2y ago
      Makes one a little hesitant on how well alignment can work with AIs.

benreesman 2y ago

It’s just clearly military R&D at this point. And it’s not even a little bit controversial that cutting edge military R&D is classified in general and to an extreme in wartime. The new thing is the lie that it’s a consumer offering. What’s new is giving the helm to shady failed social network founders with no accountability. These people aren’t retired generals with combat experience. They aren’t tenured professors at Princeton IAS on a Nobel shortlist and encumbered by TS clearance. They’re godawful almost ran psychos who never built anything that wasn’t extractive and owe their position in the world to pg’s partisanship 15 fucking years ago.

DebtDeflation 2y ago
>owe their position in the world to pg’s partisanship 15 fucking years ago PG?
- RGamma 2y ago
  Paul Graham
sgregnt 2y ago
To me it is not clear at all, can you please elaborate why you make such a strong claim?
- benreesman 2y ago
  My opinion is based on a lot more first hand experience than most, some of which I’m at liberty to share and some that I’m not and therefore becomes “color”. But I’m a nobody, Edward Snowden has a far more convincing track record on calling abuses of power: https://community.openai.com/t/edward-snowden-on-openai-s-decision-to-put-a-former-nsa-director-on-its-board/821456 https://community.openai.com/t/edward-snowden-on-openai-s-de...
  - dash2 2y ago
    Here's Edward Snowden heroically waffling about "nuance" when asked why he hasn't spoken out more about Russia's invasion of Ukraine. On a crypto website. He became a full citizen of Russia in autumn 2022, by the way. https://www.coindesk.com/video/edward-snowden-explains-why-he-doesnt-talk-about-ukraine-crisis-2/embed/ https://www.coindesk.com/video/edward-snowden-explains-why-h...

dreamcompiler 2y ago

AGI is still a long way off. The history of AI goes back 65 years and there have been probably a dozen episodes where people said "AGI is right around the corner" because some program did something surprising and impressive. It always turns out human intelligence is much, much harder than we think it is. I saw a tweet the other day that sums up the current situation perfectly: "I don't need AI to paint pictures and write poetry so I have more time to fold laundry and wash dishes. I want the AI to do the laundry and dishes so I have more time to paint and write poetry."

rowanG077 2y ago
That statement is extremely short sighted. You don't need AI to do laundry and dishes. You need expensive robotics. in fact both already exist in a cheapened form. A laundry machine and a dishwasher. They already take 90% of the work out of it.
dchichkov 2y ago
Well, copilots do precisely that, no? Or you talking folding literal laundry, in which case this is more of a robotics problem, not the ASI, right? You don't need ASI to fold laundry, you do need to achieve reliable, safe and cost efficient robotics deployments. These are different problems.
- goatlover 2y ago
  You wouldn't get close to ASI before laundry problem had been solved.
- Dylan16807 2y ago
  > You don't need ASI to fold laundry Robots are garbage at manipulating objects, and it's the software that's lacking much more than the hardware. Let's say AGI is 10 and ASI is 11. They're saying we can't even get this dial cranked up to 3, so we're not anywhere close to 10 or 11. You're right that folding laundry doesn't need 11, but that's not relevant to their point.
bradleykingz 2y ago

buckinkb 2y ago

We should be less concerned about super intelligence and more about the immediate threat of job loss. An AI doesn’t need to be Skynet to wreak massive havoc on society. Replacing 20% of jobs in a very short period of time could spark global unrest resulting in WW3

sgregnt 2y ago
The thing is, the replaced 20% people can always revert to having economy i.e., business among themselves, unless of cause they themselves prefer (cheaper) buissiness with AI. But then this just means they are better off in the first place from this change. It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. Well, they can continue to do low productivity business but increase it a bit using things like phones developed by high productivity country elsewhere.
- insane_dreamer 2y ago
  Reassuring words for the displaced 20% ...
- T-A 2y ago
  > It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. No. A country has its own territory, laws, central bank, currency etc. If it has sufficient natural resources to feed itself, it can get by on its own (North Korea comes to mind). Individuals unable to compete in their national economy have none of that. Do you own enough land to feed yourself?
fauigerzigerk 2y ago
Replacing 20% of jobs in, say, 10 years wouldn't be that unusual [1]. It can mean growing prosperity. In fact, productivity growth is the only thing that increases wealth overall. It is the lack of productivity growth that is causing a lot of extremism and conflict right now. Large groups of people feel that the only way for them to win is if others lose and vice versa. That's a recipe for disaster. The key question is what happens to those who lose their jobs. Will they find other, perhaps even better, jobs? Will they get a piece of the growing pie even if they don't find other jobs and have to retire early? It's these eternal political problems that we have to solve. It's nothing new. It has never been easy. But it's probably easier than managing decline and stagnation because at least we would have a growing pie to divvy up. [1] https://www.britannica.com/money/productivity/Historical-trends https://www.britannica.com/money/productivity/Historical-tre...

ridiculous_leke 2y ago

> The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example. The same era saw big achievements like first human in space, eradication of smallpox, peaceful nuclear exploration etc. It's good to be a skeptic but history does favor the optimists for the most part.

goatlover 2y ago
We'll see what the next 100 years or history brings. The nuclear war threat hasn't gone away either. There's always a chance those nukes get used at some point.
bbor 2y ago
Holy hell please knock on wood, this is the kinda comment that gets put in a museum in 10,000 years on The Beginning of the End of The Age of Hubris. We've avoided side effects from our new weapons for 80 years -- that does not exactly make me super confident it won't happen again! In general, I think drawing conclusions about "history" from the past couple hundred years is tough. And unless you take a VERY long view, I don't see how one could describe the vast majority of the past as a win for the optimists. I guess suffering is relative, but good god was there a lot of suffering before modern medicine. If anyone's feeling like we've made it through to the other side of the nuclear threat, "Mission Accomplished"-style, I highly recommend A Canticle for Lebowitz. It won a hugo award, and it's a short read best done with little research beforehand.
insane_dreamer 2y ago
Were any of these big achievements side effects of creating nuclear weapons? If not, then they're not relevant to the issue. I'm not saying nothing else good happened in the past 70 years, but rather that the invention of atomic weapons has permanently placed humanity in a position in which it had never been before: the possibility of wiping out much of the planet, averted only thanks to treaties, Stanislav Petrov[0], and likely other cool heads. [0] https://en.wikipedia.org/wiki/Stanislav_Petrov https://en.wikipedia.org/wiki/Stanislav_Petrov
- 2y ago

bbor 2y ago

Yup, well said. I think it's important to remember sometimes that Skynet was some sort of all-powerful military program -- maybe we should just, y'know, not do that part? Not even to win a war? That's the hope... More generally/academically, you've pointed out that this covers only half of the violence problem, and I'd argue there's actually a whole other dimension at play bringing the total number of problem areas to four, of which this just the first: ## I.1. Subservient Violence ## I.2. Subservient Misinformation ## I.3. Autonomous Violence ## I.4. Autonomous Misinformation But I think it's a lot harder to recruit for an AI alignment company than it is to recruit for an AI safety company.

sgregnt 2y ago

How about this possibility: The good guys will be one step ahead, they will have more resources the bad guys will risk imprisonment if they misapply super intelligence. And this will be discovered and protected from by even better super intelligence.

insane_dreamer 2y ago
Sounds like a movie plot.

tucnak 2y ago

> Quis custodiet ipsos custodes? -- That is my biggest concern. Latin-phrase compulsion is not the worst disease that could happen to a man.

bamboozled 2y ago

Quis custodiet ipsos custodes I love to think about how this would feel for "AI" too :)

tim333 2y ago

Indeed. I'd much rather someone like Altman does it who is shifty but can at least be controlled by the US government than someone like Putin who'd probably have it leverage their nuclear arsenal to try "denazify" planet like he's doing in Ukraine.

alickz 2y ago

the nazi regime made great use of punch cards and data crunching for their logistics i would hate to have seen them with a superintelligent AI at their disposal

insane_dreamer 2y ago
I wonder what the North Koreans would do with it.