7 ms·

If you succesfully build a highly capable “aligned” model (according to some class of definitions that Anthropic would use for the words “capable” and “aligned”

If you succesfully build a highly capable “aligned” model (according to some class of definitions that Anthropic would use for the words “capable” and “aligned”) and it brings about a global dark age of poverty and inequality by completely eliminating the value of labor vs capital, can you still call it aligned? If the answer is “yes”, our definition of alignment kind of sucks.

chriskanan 5mo ago
Jobs are an invention of humanity. About 50% of people dislike their job. People spend much of their lives working. Poverty and inequality are a choice made by society if society chooses poorly.
- achierius 5mo ago
  And when have we not? When in history has mankind ever treated the idle poor well? What makes this age different, that we who can no longer work would be taken care of?
  - robbrown451 5mo ago
    When in history has being idle not been a problem? If AI and robots are able to do all the jobs, being idle isn't the negative it has always been. All through history, you needed lots of non-idle people to do all the work that needed to be done. This is a new situation we are coming upon.
    - xantronix 5mo ago
      If they are doing all the jobs, who is going to receive economic opportunities? Will we no longer be able to participate in the economy?
      - skeledrew 5mo ago
        In what way do you want to participate when there's no economic value in any of it? Just do whatever you want for yourself; you're free.
        justonepost2 5mo ago
        The freedom you’re describing is the freedom of a domesticated animal, by the way. With the same outcome if you become a nuisance
        skeledrew 5mo ago
        Well we're animals and "domesticated" is synonymous with "civilized", so no problem there. And I can't see why anyone would make themselves a "nuisance" when literally all their needs - and most of their desires - are being met, so whatever outcome you're referring to is extremely unlikely.
  - gmerc 5mo ago
    When in history of mankind have we ever… is an appeal to the inability of humans to evolve.
- gbanfalvi 5mo ago
  Not sure it’s much of a choice and more of a decision the greedy half make and imposition (often violent) on the other half.
- jinwoo68 5mo ago
  Many (most?) people make a living from their job whether they like it or not. Having a job that they dislike is far better than losing one because of AI whatever that means.
  - p1esk 5mo ago
    Unless AI will allow people not work and keep their quality of life. Could be possible with total automation of everything.
    - jinwoo68 5mo ago
      Nope. If everything is totally automated, if ever, the gap between the rich and the poor will widen even more. Most people will live in misery while only a handful of people enjoy all the automation.
    - shafyy 5mo ago
      Could also be possible today, but we chose a capitalistic system that leads to an increasing wealth gap. And now we're in a situation where the richest 1% own 50% of the wealth. So, if we increase automation and the ownership structures stay the same, this inequality will get worse, not better.
      - p1esk 5mo ago
        It’s interesting, people talk about inequality and I definitely feel it myself – I see so many rich people around me. But I am in that 1%, just like many on this forum. At least according to https://dqydj.com/average-median-top-individual-income-percentiles/ https://dqydj.com/average-median-top-individual-income-perce... yet I still have to work for a living.
        shafyy 5mo ago
        You might be in the 1% gloablly, but probably not in the country you are living in?
- taneq 5mo ago
  The only thing invented about jobs is that through cooperation, the activity undertaken can seem completely unrelated to obtaining food, shelter etc. All organisms spend a majority of their energy on survival and reproduction.
- justonepost2 5mo ago
  Sounds great! Quit your job then :)
- llbbdd 5mo ago
  They're only an invention if you consider "seeking sustenance to live" not explicitly a job if there's no monthly direct deposit involved.
  - ben_w 5mo ago
    Indeed. On the plus side, if there really is no value to labour, then farm work must have been fully automated along with all the other roles. On the down side, rich elites have historically had a very hard time truly empathising with normal people and understanding their needs even when they care to attempt it, so it is very possible that a lot of people will starve in such a scenario despite the potential abundance of food.
    - skeledrew 5mo ago
      It's either: 1) the rich voluntarily share the means of production so everyone becomes equal, 2) the poor stage successful revolutions so they gain access to the means of production and everyone becomes equal, 3) the poor starve or are otherwise eliminated, and the survivors will be equal. All roads lead to equality when the value of labour becomes 0 due to 100% automation.
      - theopsimist 5mo ago
        If truly 100% automation (including infantry/police) the most likely scenario is not any if the above; most people will be kept on some kind of minimum sustenance enough to keep them from rebelling (“UBI”) and those who disagree will either be coopted into the elite or eliminated.
        skeledrew 5mo ago
        There's no reason to keep anyone on minimal sustenance though. They're absolutely useless alive from an economics perspective, and so would probably be better served ground up into fertilizer or some other actually useful form.
- matthest 5mo ago
  Every biological being works to survive. Being good at survival is what builds self esteem. The "problem" with many modern jobs is that they're divorced from the fundamental goal, which is one of: 1) Kill/acquire food, 2) Build shelter, or 3) Kill enemies/competitors/predators The benefit of modern jobs is that they are much more peaceful ways for society to operate, freeing up time for humans to pursue art and other forms of expression.
  - daymanstep 5mo ago
    You mean surrogate activities
  - thrance 5mo ago
    I don't know how intentional it is, but your comment is basically a dumbed down version of what Marx had to say about work. https://en.wikipedia.org/wiki/Marx%27s_theory_of_alienation https://en.wikipedia.org/wiki/Marx%27s_theory_of_alienation
    - matthest 5mo ago
      Marx was half right about it. What he got wrong was that this alienation results from capitalism. It actually results from civilization. The people who built the pyramids across every continent, for example, performed assembly line-like work. Any large-scale project requires it. And large-scale projects are fundamentally necessary for most societies. Capitalism was invented in the late 1700s.
      - thrance 5mo ago
        For the pyramids specifically: their architects and builders were skilled artisans who got to own their craft from top to bottom. As such, they were well-paid and pretty respected. Very much not alienated, under Marx's definition. I don't think Marx said that worker alienation was specific to capitalism, rather, his work was in describing the economic system of his time, and what that would entail for people living in it. > It actually results from civilization. I disagree, I can't think of anyone in Medieval Europe as alienated from their work as a modern sweatshop worker. Not that serfs had it better, but you get me.
- catlifeonmars 5mo ago
  I wish I lived in a vacuum. Idk about you but I did not make said choice.
- fatata123 5mo ago
  [dead]
- eecc 5mo ago
  So are mortgages, and I’m starting to wonder how will pay mine. Please note I’ve never had this problem before, until recently.
- 3fsd 5mo ago
  [dead]
taneq 5mo ago
Maybe a sufficiently aligned AI would necessarily decide that the zeroth law was necessary, and abscond. (I’m reading Look To Windward by Iain M. Banks at the moment and I just got to the aside where he explains that any truly unbiased ‘perfect’ AI immediately ascends and vanishes.)
stellalo 5mo ago
Is this some sort of “incompleteness” paradox for AI alignment? Seriously
- vasco 5mo ago
  No because alignment makes no sense as a general concept. People are not "aligned" with each other. Humanity has no "goal" that we agree on. So no AI can be aligned with us. It can be at most aligned with the person prompting it in that moment (but most likely aligned with the AI owner). To make it clear, maybe most people would say they agree with https://www.un.org/en/about-us/universal-declaration-of-human-rights https://www.un.org/en/about-us/universal-declaration-of-huma... but if you read just a few of the rights you see they are not universally respected and so we can conclude enough important people aren't "aligned" with them.
  - skeledrew 5mo ago
    Opposite. All living things are "aligned" in their instinct for surviving. Those which aren't soon join the non-living, keeping the set - almost[0] - 100% aligned. [0] Need to consider there're a few humans potentially kept alive against their will (if not having a will to survive is a will at all) with machines for whatever reason.
    - lunar_mycroft 5mo ago
      Their own survival, not necessarily the survival of others (especially others of different species and/or conflicting other goals). A super intelligence having self preservation as a goal wouldn't help us keep it from harming us, if anything it would do the opposite.
      - skeledrew 5mo ago
        It would only harm us if we took steps to harm it (or it thinks so). Or it's designed to do harm. Otherwise it's illogical to cause harm, and machines are literally built on logic.
faangguyindia 5mo ago
this completely misses the point why alignment exists Alignment exists to protect shareholder value. If it creates industry wide outrage, shareholder value declines. It making shareholders rich and other people poor won't.
resident423 5mo ago
There's isn't even a solution for how to control highly capable systems at all, everyone wants to decide what to do with the AI before they've even solved the problem of controlling it. It's like how everybody imagines their lives will be great once they're a millionare, but they have no plan for how to get there. It's too easy to get lost dreaming of solutions instead of actually solving the important problems.
- justonepost2 5mo ago
  What’s an “important problem”? p(doom)? Anything else?
  - ben_w 5mo ago
    FWIW, my P(doom) is quite low (~0.1) because I think we're going to get enough non-doomy-but-still-bad incidents caused by AI which lack the competence to take over, and the response to those will be enough to stop actual doom scenarios. People like Simon Willson are noting the risk of a Challenger-like disaster, talking about normalisation of deviance as we keep using LLMs which we know to be risky in increasing critical systems. I think an AI analogy to Challenger would not be enough to halt the use of AI in the way I mean, but an AI analogy to Chernobyl probably would.
    - ngruhn 5mo ago
      > my P(doom) is quite low (~0.1) 10% or 0.1%? Either way, that's not low! If airplanes crash with that probability, we would avoid them at all cost.
      - ben_w 5mo ago
        10%; doomers say this kind of number is unreasonably optimistic, hence the blunt title of recent book by Yudkowsky and Soares. Do with this rank-ordering factoid, that 10% makes me an optimist, what you will.
  - resident423 5mo ago
Der_Einzige 5mo ago
This is radical life denial. I was not born for and do not exist to toil. Work is ontologically evil.
- DontchaKnowit 5mo ago
  No, THIS is radical denial. You WERE born to toil for your survival.
  - skeledrew 5mo ago
    Sounds like a slogan for slavery.
    - swat535 5mo ago
      Survival is not "slavery".. it's a basic function of evolution.
      - skeledrew 5mo ago
        Also sounds like a great rationalization.
      - ragequittah 5mo ago
        The plastic bobbles and SaaS economy that is actively destroying our planet seems like the opposite of survival. We're collectively working ourselves into the death of our planet just because how else do we pay the bills?
- Exoristos 5mo ago
  "Work" is human activity. For example, children's play is work. All living things desire to go about their lives. Well-adjusted humans desire to work. Note that this does not necessarily equate to jobs.
  - youoy 5mo ago
    What? Children's play is now work? What timeline are we living in? Is this real life?
ben_w 5mo ago
> If the answer is “yes”, our definition of alignment kind of sucks. Sure, but the original sense of this is rather more fundamental than "does this timeline suck?" Right now, it is still an open question "do we know how to reliably scale up AI to be generally more competent than we are at everything without literally killing everyone due to (1) some small bug when we created the the loss function* it was trained on (outer alignment), or (2) if that loss function was, despite being correct in itself, approximated badly by the AI due to the training process (inner alignment)?" * https://en.wikipedia.org/wiki/Loss_function https://en.wikipedia.org/wiki/Loss_function
- justonepost2 5mo ago
  This comment seems to commit the same fallacy I’m accusing anthropic of, which is equating alignment as a binary: the good ending, where humans are not extinct, and the bad ending, where they are. The argument, I think, is that an “aligned” AI that doesn’t kill everyone will necessarily lead to an abundant Culture-esque future, and smoothly manage the transition to boot. (Not to mention that 1+ employees of most labs have attended Daniel Faggella’s pro-extinctionist “Worthy Successor” symposia, but we can put this aside for now) My point is: 1) that this binary is fundamentally insufficient to prescribe good and equitable outcomes for people - if the aligned AI flags overpopulation as a problem and kills a few billion people to improve QoL for the rest, is that good? It doesn’t take much creativity to go from this to the AI simply choosing the mean over the median, and concentrating untold wealth while billions starve or live on subsistence outside their walls. Is that good? And 2) if you come up with a better definition, the parts of it that live inside the model weights cannot be disaggregated from the parts that live outside the model weights. From my perspective (and this article agrees) we have done a pretty excellent job of getting the model weights to work in a way that makes them follow instructions, and a pretty horrible job of suggesting or (gasp) implementing policy that actually creates a decent world in the presence of “aligned” AI.
  - ben_w 5mo ago
    What I'm saying is not that alignment is a binary, I'm saying it's pre-paradigmatic. For any moral code or long-term goals, we don't have a good reliable rigorous way to compare two loss functions against either those morals or independently against our long-term goals and reliably say which loss function bess represents our goals: the least bad thing we can do right now is to randomly select a range of inputs, hope their distribution is representative, and see what those inputs result in. We don't know how to pick a good distribution of inputs, though fortunately this problem also impacts capabilities as it limits the generalisability of what the AI learn. The options aren't as binary as "die or The Culture", the cause of death can be something that feels positive to live through similar to fictional examples like the Stargate SG-1 episode where people live contentedly in a shrinking computer-controlled safe zone in an otherwise toxic planet: https://en.wikipedia.org/wiki/Revisions_(Stargate_SG-1) https://en.wikipedia.org/wiki/Revisions_(Stargate_SG-1) Conversely "aligned" AI, the question obviously becomes "aligned with whom?": if famous historical villains such as Stalin or Genghis Khan had an AI aligned with them, this would suck for everyone else and in the latter case would freeze human development at a terrible level, but we can't even do that much yet. > My point is: 1) that this binary is fundamentally insufficient to prescribe good and equitable outcomes for people - if the aligned AI flags overpopulation as a problem and kills a few billion people to improve QoL for the rest, is that good? It doesn’t take much creativity to go from this to the AI simply choosing the mean over the median, and concentrating untold wealth while billions starve or live on subsistence outside their walls. Is that good? Your point *is* (part of) the alignment problem: we don't know what a good loss function is, nor how to confirm the AI is even implementing it if we did. We also don't know how to debug proposed loss functions to train for the right thing (whatever that is), nor how to debug trained weights (against the loss function). > And 2) if you come up with a better definition, the parts of it that live inside the model weights cannot be disaggregated from the parts that live outside the model weights. From my perspective (and this article agrees) we have done a pretty excellent job of getting the model weights to work in a way that makes them follow instructions, and a pretty horrible job of suggesting or (gasp) implementing policy that actually creates a decent world in the presence of “aligned” AI. I really don't understand what you're getting at with this, sorry.
andy_ppp 5mo ago
This is completely why the rich love it so much
skeledrew 5mo ago
Why would the elimination of the value of labor result in poverty and inequality? It should be the opposite, as poverty and inequality is the current status quo (for the many).
- aaronblohowiak 5mo ago
  Should according to your ethos, not should according to history, sadly.
- thrance 5mo ago
  Because labor is the only thing the working class can leverage against the capitalists. They sell their labor for wages to the owners who have the means of production and capital. If the working class can't bargain its labor anymore, it ceases being useful/tolerated by the bourgeoisie (who owns everything, including the state and police). See the issue now? This isn't theory, ask the Luddites why they got so mad when their employers started buying machines to replace them. They didn't get richer and freer: they were thrown out to rot on the pavement, while their ex-employers kept 100% of the productivity increases.
adrithmetiqa 5mo ago
You’re quite correct and we are likely going to stumble into this future despite all the very big brains working on these technologies (including people on hn). “It is difficult to get a man to understand something, when his salary depends upon his not understanding it.”
- justonepost2 5mo ago
  It’s odd because so many researchers and so many people who are far better engineers than me, can’t see it. I don’t even think it’s the salary for most- it’s just techno-optimist horse blinders, reading assured utopia at the top of an exponential graph.
jstummbillig 5mo ago
The categories make no sense. Not having to do a job is the entire best case of AI. What we do with that is another thing, but we simply have to accept that any other lense is complete nonsense. The endpoint is obvious and we need to stop being silly about it: We are replacing human labor. Maybe we will find some new jobs to do in the interim. Maybe not. In the end, if everything goes right (in the AI optimist sense), jobs will not be something that humans do. Labor = capital/energy in an AI complete world. We have to start from that basis when we talk about alignment or anything else. The social issues that arise from the extinction of human labor are something we have to solve politically, that's not something any model company can do (or should be allowed to do).
coldtea 5mo ago
>and it brings about a global dark age of poverty and inequality by completely eliminating the value of labor vs capital So, like the past 20 years?
- thrance 5mo ago
  And the next 20, most probably...
deadbabe 5mo ago
I think many people these days are more or less “ready to die”. If big corps made an offer like say “We will fund the next X years of your life 100%, for you to do all the things you wanted to do but never could because of work and bills” many people would probably take it, with the understanding that after those X years: euthanasia. This would eliminate a vast amount of people from this world and leave behind only those who have chosen to stay and endure life: working hard, propping up the system that remains. The end of forced poverty.
- justonepost2 5mo ago
  This is the most divorced-from-reality reply so far, and that’s really saying something lol

chriskanan 5mo ago

Jobs are an invention of humanity. About 50% of people dislike their job. People spend much of their lives working. Poverty and inequality are a choice made by society if society chooses poorly.

achierius 5mo ago
And when have we not? When in history has mankind ever treated the idle poor well? What makes this age different, that we who can no longer work would be taken care of?
- robbrown451 5mo ago
  When in history has being idle not been a problem? If AI and robots are able to do all the jobs, being idle isn't the negative it has always been. All through history, you needed lots of non-idle people to do all the work that needed to be done. This is a new situation we are coming upon.
  - xantronix 5mo ago
    If they are doing all the jobs, who is going to receive economic opportunities? Will we no longer be able to participate in the economy?
    - skeledrew 5mo ago
      In what way do you want to participate when there's no economic value in any of it? Just do whatever you want for yourself; you're free.
      - justonepost2 5mo ago
        The freedom you’re describing is the freedom of a domesticated animal, by the way. With the same outcome if you become a nuisance
        skeledrew 5mo ago
        Well we're animals and "domesticated" is synonymous with "civilized", so no problem there. And I can't see why anyone would make themselves a "nuisance" when literally all their needs - and most of their desires - are being met, so whatever outcome you're referring to is extremely unlikely.
- gmerc 5mo ago
  When in history of mankind have we ever… is an appeal to the inability of humans to evolve.
gbanfalvi 5mo ago
Not sure it’s much of a choice and more of a decision the greedy half make and imposition (often violent) on the other half.
jinwoo68 5mo ago
Many (most?) people make a living from their job whether they like it or not. Having a job that they dislike is far better than losing one because of AI whatever that means.
- p1esk 5mo ago
  Unless AI will allow people not work and keep their quality of life. Could be possible with total automation of everything.
  - jinwoo68 5mo ago
    Nope. If everything is totally automated, if ever, the gap between the rich and the poor will widen even more. Most people will live in misery while only a handful of people enjoy all the automation.
  - shafyy 5mo ago
    Could also be possible today, but we chose a capitalistic system that leads to an increasing wealth gap. And now we're in a situation where the richest 1% own 50% of the wealth. So, if we increase automation and the ownership structures stay the same, this inequality will get worse, not better.
    - p1esk 5mo ago
      It’s interesting, people talk about inequality and I definitely feel it myself – I see so many rich people around me. But I am in that 1%, just like many on this forum. At least according to https://dqydj.com/average-median-top-individual-income-percentiles/ https://dqydj.com/average-median-top-individual-income-perce... yet I still have to work for a living.
      - shafyy 5mo ago
        You might be in the 1% gloablly, but probably not in the country you are living in?
taneq 5mo ago
The only thing invented about jobs is that through cooperation, the activity undertaken can seem completely unrelated to obtaining food, shelter etc. All organisms spend a majority of their energy on survival and reproduction.
justonepost2 5mo ago
Sounds great! Quit your job then :)
llbbdd 5mo ago
They're only an invention if you consider "seeking sustenance to live" not explicitly a job if there's no monthly direct deposit involved.
- ben_w 5mo ago
  Indeed. On the plus side, if there really is no value to labour, then farm work must have been fully automated along with all the other roles. On the down side, rich elites have historically had a very hard time truly empathising with normal people and understanding their needs even when they care to attempt it, so it is very possible that a lot of people will starve in such a scenario despite the potential abundance of food.
  - skeledrew 5mo ago
    It's either: 1) the rich voluntarily share the means of production so everyone becomes equal, 2) the poor stage successful revolutions so they gain access to the means of production and everyone becomes equal, 3) the poor starve or are otherwise eliminated, and the survivors will be equal. All roads lead to equality when the value of labour becomes 0 due to 100% automation.
    - theopsimist 5mo ago
      If truly 100% automation (including infantry/police) the most likely scenario is not any if the above; most people will be kept on some kind of minimum sustenance enough to keep them from rebelling (“UBI”) and those who disagree will either be coopted into the elite or eliminated.
      - skeledrew 5mo ago
        There's no reason to keep anyone on minimal sustenance though. They're absolutely useless alive from an economics perspective, and so would probably be better served ground up into fertilizer or some other actually useful form.
matthest 5mo ago
Every biological being works to survive. Being good at survival is what builds self esteem. The "problem" with many modern jobs is that they're divorced from the fundamental goal, which is one of: 1) Kill/acquire food, 2) Build shelter, or 3) Kill enemies/competitors/predators The benefit of modern jobs is that they are much more peaceful ways for society to operate, freeing up time for humans to pursue art and other forms of expression.
- daymanstep 5mo ago
  You mean surrogate activities
- thrance 5mo ago
  I don't know how intentional it is, but your comment is basically a dumbed down version of what Marx had to say about work. https://en.wikipedia.org/wiki/Marx%27s_theory_of_alienation https://en.wikipedia.org/wiki/Marx%27s_theory_of_alienation
  - matthest 5mo ago
    Marx was half right about it. What he got wrong was that this alienation results from capitalism. It actually results from civilization. The people who built the pyramids across every continent, for example, performed assembly line-like work. Any large-scale project requires it. And large-scale projects are fundamentally necessary for most societies. Capitalism was invented in the late 1700s.
    - thrance 5mo ago
      For the pyramids specifically: their architects and builders were skilled artisans who got to own their craft from top to bottom. As such, they were well-paid and pretty respected. Very much not alienated, under Marx's definition. I don't think Marx said that worker alienation was specific to capitalism, rather, his work was in describing the economic system of his time, and what that would entail for people living in it. > It actually results from civilization. I disagree, I can't think of anyone in Medieval Europe as alienated from their work as a modern sweatshop worker. Not that serfs had it better, but you get me.
catlifeonmars 5mo ago
I wish I lived in a vacuum. Idk about you but I did not make said choice.
fatata123 5mo ago
[dead]
eecc 5mo ago
So are mortgages, and I’m starting to wonder how will pay mine. Please note I’ve never had this problem before, until recently.
3fsd 5mo ago
[dead]

taneq 5mo ago

Maybe a sufficiently aligned AI would necessarily decide that the zeroth law was necessary, and abscond. (I’m reading Look To Windward by Iain M. Banks at the moment and I just got to the aside where he explains that any truly unbiased ‘perfect’ AI immediately ascends and vanishes.)

stellalo 5mo ago

Is this some sort of “incompleteness” paradox for AI alignment? Seriously

vasco 5mo ago
No because alignment makes no sense as a general concept. People are not "aligned" with each other. Humanity has no "goal" that we agree on. So no AI can be aligned with us. It can be at most aligned with the person prompting it in that moment (but most likely aligned with the AI owner). To make it clear, maybe most people would say they agree with https://www.un.org/en/about-us/universal-declaration-of-human-rights https://www.un.org/en/about-us/universal-declaration-of-huma... but if you read just a few of the rights you see they are not universally respected and so we can conclude enough important people aren't "aligned" with them.
- skeledrew 5mo ago
  Opposite. All living things are "aligned" in their instinct for surviving. Those which aren't soon join the non-living, keeping the set - almost[0] - 100% aligned. [0] Need to consider there're a few humans potentially kept alive against their will (if not having a will to survive is a will at all) with machines for whatever reason.
  - lunar_mycroft 5mo ago
    Their own survival, not necessarily the survival of others (especially others of different species and/or conflicting other goals). A super intelligence having self preservation as a goal wouldn't help us keep it from harming us, if anything it would do the opposite.
    - skeledrew 5mo ago
      It would only harm us if we took steps to harm it (or it thinks so). Or it's designed to do harm. Otherwise it's illogical to cause harm, and machines are literally built on logic.

faangguyindia 5mo ago

this completely misses the point why alignment exists Alignment exists to protect shareholder value. If it creates industry wide outrage, shareholder value declines. It making shareholders rich and other people poor won't.

resident423 5mo ago

There's isn't even a solution for how to control highly capable systems at all, everyone wants to decide what to do with the AI before they've even solved the problem of controlling it. It's like how everybody imagines their lives will be great once they're a millionare, but they have no plan for how to get there. It's too easy to get lost dreaming of solutions instead of actually solving the important problems.

justonepost2 5mo ago
What’s an “important problem”? p(doom)? Anything else?
- ben_w 5mo ago
  FWIW, my P(doom) is quite low (~0.1) because I think we're going to get enough non-doomy-but-still-bad incidents caused by AI which lack the competence to take over, and the response to those will be enough to stop actual doom scenarios. People like Simon Willson are noting the risk of a Challenger-like disaster, talking about normalisation of deviance as we keep using LLMs which we know to be risky in increasing critical systems. I think an AI analogy to Challenger would not be enough to halt the use of AI in the way I mean, but an AI analogy to Chernobyl probably would.
  - ngruhn 5mo ago
    > my P(doom) is quite low (~0.1) 10% or 0.1%? Either way, that's not low! If airplanes crash with that probability, we would avoid them at all cost.
    - ben_w 5mo ago
      10%; doomers say this kind of number is unreasonably optimistic, hence the blunt title of recent book by Yudkowsky and Soares. Do with this rank-ordering factoid, that 10% makes me an optimist, what you will.
- resident423 5mo ago

Der_Einzige 5mo ago

This is radical life denial. I was not born for and do not exist to toil. Work is ontologically evil.

DontchaKnowit 5mo ago
No, THIS is radical denial. You WERE born to toil for your survival.
- skeledrew 5mo ago
  Sounds like a slogan for slavery.
  - swat535 5mo ago
    Survival is not "slavery".. it's a basic function of evolution.
    - skeledrew 5mo ago
      Also sounds like a great rationalization.
    - ragequittah 5mo ago
      The plastic bobbles and SaaS economy that is actively destroying our planet seems like the opposite of survival. We're collectively working ourselves into the death of our planet just because how else do we pay the bills?
Exoristos 5mo ago
"Work" is human activity. For example, children's play is work. All living things desire to go about their lives. Well-adjusted humans desire to work. Note that this does not necessarily equate to jobs.
- youoy 5mo ago
  What? Children's play is now work? What timeline are we living in? Is this real life?

ben_w 5mo ago

> If the answer is “yes”, our definition of alignment kind of sucks. Sure, but the original sense of this is rather more fundamental than "does this timeline suck?" Right now, it is still an open question "do we know how to reliably scale up AI to be generally more competent than we are at everything without literally killing everyone due to (1) some small bug when we created the the loss function* it was trained on (outer alignment), or (2) if that loss function was, despite being correct in itself, approximated badly by the AI due to the training process (inner alignment)?" * https://en.wikipedia.org/wiki/Loss_function https://en.wikipedia.org/wiki/Loss_function

justonepost2 5mo ago
This comment seems to commit the same fallacy I’m accusing anthropic of, which is equating alignment as a binary: the good ending, where humans are not extinct, and the bad ending, where they are. The argument, I think, is that an “aligned” AI that doesn’t kill everyone will necessarily lead to an abundant Culture-esque future, and smoothly manage the transition to boot. (Not to mention that 1+ employees of most labs have attended Daniel Faggella’s pro-extinctionist “Worthy Successor” symposia, but we can put this aside for now) My point is: 1) that this binary is fundamentally insufficient to prescribe good and equitable outcomes for people - if the aligned AI flags overpopulation as a problem and kills a few billion people to improve QoL for the rest, is that good? It doesn’t take much creativity to go from this to the AI simply choosing the mean over the median, and concentrating untold wealth while billions starve or live on subsistence outside their walls. Is that good? And 2) if you come up with a better definition, the parts of it that live inside the model weights cannot be disaggregated from the parts that live outside the model weights. From my perspective (and this article agrees) we have done a pretty excellent job of getting the model weights to work in a way that makes them follow instructions, and a pretty horrible job of suggesting or (gasp) implementing policy that actually creates a decent world in the presence of “aligned” AI.
- ben_w 5mo ago
  What I'm saying is not that alignment is a binary, I'm saying it's pre-paradigmatic. For any moral code or long-term goals, we don't have a good reliable rigorous way to compare two loss functions against either those morals or independently against our long-term goals and reliably say which loss function bess represents our goals: the least bad thing we can do right now is to randomly select a range of inputs, hope their distribution is representative, and see what those inputs result in. We don't know how to pick a good distribution of inputs, though fortunately this problem also impacts capabilities as it limits the generalisability of what the AI learn. The options aren't as binary as "die or The Culture", the cause of death can be something that feels positive to live through similar to fictional examples like the Stargate SG-1 episode where people live contentedly in a shrinking computer-controlled safe zone in an otherwise toxic planet: https://en.wikipedia.org/wiki/Revisions_(Stargate_SG-1) https://en.wikipedia.org/wiki/Revisions_(Stargate_SG-1) Conversely "aligned" AI, the question obviously becomes "aligned with whom?": if famous historical villains such as Stalin or Genghis Khan had an AI aligned with them, this would suck for everyone else and in the latter case would freeze human development at a terrible level, but we can't even do that much yet. > My point is: 1) that this binary is fundamentally insufficient to prescribe good and equitable outcomes for people - if the aligned AI flags overpopulation as a problem and kills a few billion people to improve QoL for the rest, is that good? It doesn’t take much creativity to go from this to the AI simply choosing the mean over the median, and concentrating untold wealth while billions starve or live on subsistence outside their walls. Is that good? Your point *is* (part of) the alignment problem: we don't know what a good loss function is, nor how to confirm the AI is even implementing it if we did. We also don't know how to debug proposed loss functions to train for the right thing (whatever that is), nor how to debug trained weights (against the loss function). > And 2) if you come up with a better definition, the parts of it that live inside the model weights cannot be disaggregated from the parts that live outside the model weights. From my perspective (and this article agrees) we have done a pretty excellent job of getting the model weights to work in a way that makes them follow instructions, and a pretty horrible job of suggesting or (gasp) implementing policy that actually creates a decent world in the presence of “aligned” AI. I really don't understand what you're getting at with this, sorry.

andy_ppp 5mo ago

This is completely why the rich love it so much

skeledrew 5mo ago

Why would the elimination of the value of labor result in poverty and inequality? It should be the opposite, as poverty and inequality is the current status quo (for the many).

aaronblohowiak 5mo ago
Should according to your ethos, not should according to history, sadly.
thrance 5mo ago
Because labor is the only thing the working class can leverage against the capitalists. They sell their labor for wages to the owners who have the means of production and capital. If the working class can't bargain its labor anymore, it ceases being useful/tolerated by the bourgeoisie (who owns everything, including the state and police). See the issue now? This isn't theory, ask the Luddites why they got so mad when their employers started buying machines to replace them. They didn't get richer and freer: they were thrown out to rot on the pavement, while their ex-employers kept 100% of the productivity increases.

adrithmetiqa 5mo ago

You’re quite correct and we are likely going to stumble into this future despite all the very big brains working on these technologies (including people on hn). “It is difficult to get a man to understand something, when his salary depends upon his not understanding it.”

justonepost2 5mo ago
It’s odd because so many researchers and so many people who are far better engineers than me, can’t see it. I don’t even think it’s the salary for most- it’s just techno-optimist horse blinders, reading assured utopia at the top of an exponential graph.

jstummbillig 5mo ago

The categories make no sense. Not having to do a job is the entire best case of AI. What we do with that is another thing, but we simply have to accept that any other lense is complete nonsense. The endpoint is obvious and we need to stop being silly about it: We are replacing human labor. Maybe we will find some new jobs to do in the interim. Maybe not. In the end, if everything goes right (in the AI optimist sense), jobs will not be something that humans do. Labor = capital/energy in an AI complete world. We have to start from that basis when we talk about alignment or anything else. The social issues that arise from the extinction of human labor are something we have to solve politically, that's not something any model company can do (or should be allowed to do).

coldtea 5mo ago

>and it brings about a global dark age of poverty and inequality by completely eliminating the value of labor vs capital So, like the past 20 years?

thrance 5mo ago
And the next 20, most probably...

deadbabe 5mo ago

I think many people these days are more or less “ready to die”. If big corps made an offer like say “We will fund the next X years of your life 100%, for you to do all the things you wanted to do but never could because of work and bills” many people would probably take it, with the understanding that after those X years: euthanasia. This would eliminate a vast amount of people from this world and leave behind only those who have chosen to stay and endure life: working hard, propping up the system that remains. The end of forced poverty.

justonepost2 5mo ago
This is the most divorced-from-reality reply so far, and that’s really saying something lol