8 ms·

The Danger of Superhuman AI Is Not What You Think

41 points by max_ 2y ago

ck2 2y ago
Superhuman AI won't be used to improve quality of life just like the day after someone figured out you could put ads on usenet It's going to be used to pump as much money out of each individual as possible Every online store, big corporate especially will instantly realize how much you are able and willing to pay for everything and that's your "personalized price" It will scan all your social media, it will look at your purchase histories, it will know your income and credit Like a used car dealer on steroids that never gets tired and learns more and more about you by following you around and watching what you do Everything from your daily food to your big ticket purchases, maximum customized prices It's going to be evil as hell and lawmakers will just be paid off eventually to let it all happen, data collection and no two people paying the same for the same thing will become 100% legal and the norm
- jt2190 2y ago
  For this to be true, people will also have to be prevented from using AI to combat the “cheap car salesman on steroids” AI bot with their own “eternal tire kicker cheapskate car buyer” AI bot. So not an argument about AI but who gets to use and benefit from it. That feels like a dictatorship level of control.
  - pixl97 2y ago
    I mean, right now you can sue any large company you want to "combat" them and their bad practices. Why aren't you? Oh yea, they typically have massive piles of resources split up in shell companies that make them very difficult and expensive to fight. And you, well you're a single person with limited resources and time. Now, you can pool your resources together against the big AI using companies if you like, but that is going to cost a massive amount of money and time to fight them still. So humans typically like to put these resources in to a government that at least attempts to look out for their best interests.
    - jfyi 2y ago
      Humans typically like to keep these resources out of the hands of government until something horrific happens then do so begrudgingly. Then half of them still insist it's a bad idea and run out to learn the hard way. There is probably some kind of fallacy somewhere like "everyone thinks they will be the one who benefits from a power imbalance until they experience losing first hand".
      - pixl97 2y ago
        Unfortunately jfyi won't be able to post any further after disbanding the EPA for their imbalance of power, became deathly ill after drinking chemically poisoned water from his well. It wasn't even the water that got him, it was was the trash eating bears. In his weakened state he could not get away. All this said, we respect him for his stance of "survival of the fittest". His passing was quite terrible and slow, as the police force was disbanded and the neighbor wanted a $35 cash deposit for ammunition to shoot the bear, it took 45 minutes for his screaming to stop. https://www.vox.com/policy-and-politics/21534416/free-state-project-new-hampshire-libertarians-matthew-hongoltz-hetling https://www.vox.com/policy-and-politics/21534416/free-state-... https://www.newyorker.com/humor/daily-shouts/l-p-d-libertarian-police-department https://www.newyorker.com/humor/daily-shouts/l-p-d-libertari...
        jfyi 2y ago
        What a weird response. I'm not sure if you mistook me for standing behind a Libertarian viewpoint and are mocking me for it or if you are just joking. You know... "and then"ing what I said, taking it up a notch for the gag. If it was the first, I can assure you the comment was lamenting the situation, not celebrating it. I even chuckled to myself that the fallacy in question is called "Libertarianism". If it was the second, the best advice about joking I have ever heard is "if at all possible, involve a cow". I'll be looking out for it next time ;)
  - klyrs 2y ago
    Good luck smuggling your rogue AI into the walled garden tended by the advertisers. Some of us will succeed, but the vast majority won't even recognize the need
- SoftTalker 2y ago
  This seems to assume that the competitive marketplace will disappear, or that we'll have no ability to look at what a gallon of milk costs at Kroger vs. Aldi and make a decision on where to buy, or that prices will not be posted until you check out and everyone will just be OK with that.
  - pixl97 2y ago
    You've seen grocery costs rocket up massively in the past few years and only now is it really reaching a boiling point. Turns out if you keep people entertained they will accept a massive amount of shit. When it comes to food, there is far less competition than one would expect. The distribution markets have become ran by just a few companies. And those 'competitors' will gladly subscribe to some service like RealPage for food so they could massively increase their profits.
    - SoftTalker 2y ago
      But if they get too greedy then local farming becomes competitive again. I can buy eggs from my neighbor, or at the local farmer's market, or I can get some chickens of my own.
      - pixl97 2y ago
        Eh, not really. My sister has a large ranch and keeps chickens for eggs, but it's not something that's going to get even close to breaking even. A huge amount of time and effort is needed keeping predators away. Feed in small amounts is rather expensive compared to large wholesale amounts. And letting them range is a rather large amount of work ensuring they don't get in trouble.
deleted 2y ago
[deleted]
SonOfLilit 2y ago
I guess I can sigh in relief, the Superhuman AI that Joshua Bengio and Geoffrey Hinton are warning us might drive humanity towards extinction the way we did to most apes and many tribal civilizations is no danger after all, because you see, to be Human is not about winning at war or economy, not about being good at solving a vast array of tasks, it's about playing with your kids. So the better-at-problemsolving machines aren't coming to destroy everything that is dear to me because I... used the wrong word to refer to them? I mean, sure, there are arguments to make against Bengio and Hinton, definitely against Yudkowsky, I myself only give less than 50% that they are right and doom is imminent (which is of course enough risk to warrant taking action to prevent it, even the kind of "crazy" action Yudkowsky advocates for), but this "argument"... what the heck did I just read.
- stonogo 2y ago
  Worse, the author completely gets the fundamental precept wrong by claiming "superhuman means human, but more so" when in fact it means "above human." So the whole article is just barking up the wrong tree. The real reason the better-at-problemsolving machines aren't coming to destroy everything that is dear to you is because they don't exist and we're not close to making them, but this article misses that basic truth in the pursuit of a misunderstanding of the prefix "super".
lukeschlather 2y ago
The incoherent ramblings of LLMs often remind me of children or people with neurodegenerative diseases. If we want to preserve our humanity as this article suggests I don't think we can take it for granted that current AIs lack the same building blocks as our consciousness. The author rails against the definition of "can perform economically valuable tasks" as well as the average human, but I feel like this is a direct response to the author's refusal to consider that LLM's might be feeling. If we want to keep our humanity we have to be open to the possibility that if it can eloquently describe a strong emotion, it actually might be feeling that emotion and not simply "predicting the next token." But the author seems to think that it is obvious fact that state-of-the-art LLMs do not feel anything, and I'm not sure it's even a falsifiable question.
hprotagonist 2y ago
As the ideology behind this bait-and-switch leaks into the wider culture, it slowly corrodes our own self-understanding. If you try to point out, in a large lecture or online forum on AI, that ChatGPT does not experience and cannot think about the things that correspond to the words and sentences it produces — that it is only a mathematical generator of expected language patterns — chances are that someone will respond, in a completely serious manner: “But so are we.” And a very logical next step after this neat and tidy dehumanization is, as history has shown, the gas chambers for the obviously malfunctioning “machines”. Because whatever, it’s not like they’re actually feeling anything, right? (edit) When you reduce persons to the status of things, you are going to get people treated like things. And every single time, that’s a recipe for disaster.
- afthonos 2y ago
  So you’re saying that the people who are yelling at the top of their lungs that we should stop AI because it will kill us all are doing so as a ruse to…kill people? That’s some seven-dimensional chess they’re playing.
  - htk 2y ago
    I think OP was criticizing the hard stance on the impossibility of conscious AI.
    - hprotagonist 2y ago
      I’m saying this: if you reduce persons to the status of things, you are going to get people treated like things. And every single time that’s a recipe for disaster.
  - hprotagonist 2y ago
    i’m not saying that.
afthonos 2y ago
There’s a lot of good stuff in this article, but it nevertheless misses the point (as did Yoshua Bengio as described, to be clear). When alarmists like myself take about danger, we rarely use the term “superhuman intelligence”. We tend to use the term “superintelligence”. And the reason is that our working definition of an intelligent agent contains agents both vastly broader and vastly narrower than humans, but that nevertheless are a danger to humans. So the question isn’t “but does this agent truly understand the poetry it wrote”. It’s “can it use its world model to produce a string of words that will cause humans to harm themselves”. It’s not “does it appreciate a Georgia O’Keefe painting.” It’s “can it manipulate matter to create viruses that will kill all present and future O’Keefes”.
- candiodari 2y ago
  [dead]
- r_hanz 2y ago
  “Can its world model BE USED to produce…” “Can it BE USED to manipulate…” FTFY
  - afthonos 2y ago
    No, you didn’t. I meant what I wrote. If it is only a tool, obviously our problems are human. The question is whether the AI is setting non-human-desired goals.
htk 2y ago
She talks about "The struggle against this reductive and cynical ideology..." when talking about humans in factories, but falls on the same weak argument by broad stroking AI as "mindless mechanical remixers and regurgitators".
- danparsonson 2y ago
  That sounds like a fair description to me - what did she miss?
  - tim333 2y ago
    I think the weak arguments are in the text surrounding those quotes. Like for the first one she has >Indeed, for the entirety of the Industrial Age, those invested in the maximally efficient extraction of productive outputs from human bodies have been trying to get us to view ourselves — and more importantly one another — as flawed, inefficient, fungible machines destined to be disposed of as soon as our output rate slips below an expected peak or the moment... Which is a bit strawman. I mean who, as a real person does that? The second one she's saying why have AI do art when it should be doing my taxes, but people are probably working on doing taxes, it's just the art stuff happens to work well, LLMs filing your tax less so. Although I can see an interesting argument that those false tax deductions were hallucinated by the LLM, not my fault.
fifticon 2y ago
I see real threats in this. It resembles other 'questionable' bargains that we humans have earlier given in to. We have come to accept that we must work and produce capital to be allowed to live, currently partly to enrich various billionaires (the wealth we produce already allow us to buy dozens of big TVs we don't need, and we have shops full of junk we don't need, clearly we produce more than we need to live, for questionable reasons). We can no longer live as stone age hunters (I'm not saying that is a better fate, but we no longer have that choice). It is a very real threat that other "things you will have to accept" will arrive on the ship of AI. Sigh.
- akira2501 2y ago
  > work and produce capital to be allowed to live You have to work to live. Whether that's at a desk job or as a hunter gatherer. This has nothing to do with an "allowance" by any other entity. It's entropy itself which requires this of you. > We can no longer live as stone age hunters I personally know plenty of people who do. You're confusing your exceptionally high standard of life with the very small requirements to keep yourself alive. The gulf between the two is so wide I think you've lost track of it. I recommend a good long camping trip. > It is a very real threat One of the functions of freedom and capital are to increase individualism. Viewed from this place I'm sure AI seems like a real threat. If it threatens our freedom and capital I think you'd be surprised at how humans may group together to collectively defeat this "threat" which is almost certainly decades if not centuries away. Your brain is an analog computer. In fact it's several dozen analog computers operating independently. It's also so energy efficient it's actually kind of scary. That's a deep "super power" we have that people aren't genuinely considering in these faux-AI-philosophical debates.
  - jfyi 2y ago
    > I personally know plenty of people who do. You're confusing your exceptionally high standard of life with the very small requirements to keep yourself alive. The gulf between the two is so wide I think you've lost track of it. I recommend a good long camping trip. It was very likely meant that "we as a whole" can no longer live as stone age hunters. I might argue that is wrong too though, but only if we were to take into consideration the very harsh lessons we would learn along the way.
emtel 2y ago
So many arguments along these lines run afoul of a very simple rule: You can't make predictions about the future by playing word games! An AI system can only be dangerous or not dangerous based on what physical events happen in the real world, as a result of that AI interacting with the world via known physical laws. In order to decide whether a system is or isn't dangerous you need to have a predictive model of what that system will actually do in various hypothetical situations. You can't just say "because it is improperly defined as X it cannot do Y". If you want to predict what happens when you put too much plutonium into a small volume, you can't make any progress on this problem by talking about whether the device is truly a "bomb", or by saying that the whole question is just a rehash of the promethean myth, or that you cannot measure explosions on a single axis. The only thing that will do is to have a reliable model of the behavior of the system. Many people seem to either not understand, or intentionally obfuscate, that an AI, like a bomb, is also a physical system which interacts with the physical world. Marc Andreessen makes this error when he says AI is "just math" and therefore AIs are inherently safe. No, it's not just math, it's a computer, made of matter, that physically interacts with both human brains and other computer systems, and therefore by extension with the entire rest of the physical world. Now of course, the way that an AI interacts with the world is radically different than the way a bomb interacts with the world, and we cannot usefully model the behavior of the AI with nuclear physics, but the fact remains. (See also: https://www.youtube.com/watch?v=kBfRG5GSnhE https://www.youtube.com/watch?v=kBfRG5GSnhE) So when you see arguments like this, ask: Is the person making an argument based on a model, or not? Examples of model based arguments include things like: - "AI risk is not a concern because of computational intractability" - we understand something about the limits of computation, and possibly those limits might constrain what AIs can do. - "AI risk is a concern because less intelligent entities are not usually able to constrain the behavior of more intelligent entities" - A coarse and imperfect model indeed, but certainly a model based on observations of interactions in the real world. "Verbal Trick" arguments include things like: - "AI risk is not a concern because we can't even define intelligence" - "AI is just math" - "AI risk is not a concern because intelligence is multi-dimensional" A third category to watch out for is the misapplied model: - "AI risk is not a concern because people have always been worried about the apocalypse" - this is a model based argument, but it can only answer the question of why people are worried, it cannot answer the question of whether there is in fact something to be worried about.
goethes_kind 2y ago
Could we not one day come up with an AI that is more empathetic than the average human?
- wmf 2y ago
  Yes. People will say it's not "really" feeling anything, but I expect AIs to become better at "faking" humanity than real humans. This could be dangerous in its own way.
  - hypercube33 2y ago
    At what point does it cross the line of faking it extremely well and doing it though? The Measure of a Man
- theptip 2y ago
  In some dimensions, the current crop can achieve this. See the Google medical AI that gets scored better for bedside manner than MDs. It’s not what we would have predicted pre-GPT, but I think it’s plausible that LLMs will be superhuman in empathy/persuasion before they are in IQ. I think you can model empathy as “next token prediction” across a set of emotional states and responses, and that could end up being easier for Transformers than the abstract logical thinking required for IQ 200.
  - jfyi 2y ago
    Is empathy really a requirement for bedside manner? It's perception of empathy more than anything. I'd bet it would be interesting to see the rate of occurrence of aspd in jobs requiring bedside manner.
  - gopher_space 2y ago
    I think "what do I mean by empathy and what will I use it for" are the key points to nail down before creating something that just needs to print "wow, that sucks" or "I told you that bitch crazy". I'd expect this kind of token prediction to be an alternative to certain types of maintenance therapy, and to fit on a watch in the next few years. The problem with wanting e.g. an "empathetic salesperson" is that your successful role models don't work for shitty companies selling shitty products.
lazyeye 2y ago
Completely misses the point and focuses on labels and definitions which is irrelevant. The danger in AI will almost certainly come from a direction noone was expecting...a kid constructs a lethal virus, a military event is triggered with catastrophic consequences etc...
- pixl97 2y ago
  Or it will come from exactly where we expect it "Greedy rich people use AI to get even richer at the cost of _____"
jfyi 2y ago
This is just another framing of the problem as ai vs humanity. It comes off preachy and frankly misses the boat as much as claiming ai is sentient. The problem is humanity vs malicious humanity with absurd access to resources. AI doesn't need to think or be conscious or be superhuman to be a problem. It just needs to allow a small set of people to get away with superhuman things.
Animats 2y ago
"Once you have reduced the concept of human intelligence to what the markets will pay for, then suddenly, all it takes to build an intelligent machine — even a superhuman one — is to make something that generates economically valuable outputs at a rate and average quality that exceeds your own economic output. Anything else is irrelevant." That's capitalism. Most of the current criticisms of AI can be leveled at corporations. A corporation is a slow AI. It's capable of doing things its employees cannot do alone. Its goal is to optimize some simple metrics. The Milton Friedman position on corporations is that they have no duties other than to maximize shareholder value. What has the chattering classes freaked out about AI is that it may make possible corporations that don't need them. The Ivy League could go the way of the vocational high school.
- giraffe_lady 2y ago
  Putting aside for the moment whether that's possible, it's a correct thing to be freaked out by. How do we treat people who we don't have an economic need for? Looking into that chasm for the first time we would expect people to feel terror. Not everyone will be able to escape the horrors we keep down there.
  - Animats 2y ago
    [flagged]
    - giraffe_lady 2y ago
      Imagine what HN would look like if we felt the tiniest bit of solidarity towards other workers.
- deleted 2y ago
  [deleted]
JKCalhoun 2y ago
The author and Yoshua Bengio talked past one another. Ms. Vallor is too concerned over semantics, too fixated on the current crop of LLMs, too invested in "pinning down" Mr. Bengio. Five to ten years out though I suspect we will no longer be having arguments over what it means to be human, intelligent.
- shkkmo 2y ago
  > Five to ten years out though I suspect we will no longer be having arguments over what it means to be human, intelligent. That seems highly unlikely. We may have more information about the nature of intelligence, but I highly doubt we will have all the answers. As to arguments about "what it means to be human"... that's one of the core metaphysical questions that I doubt we will ever be done arguing about.
  - JKCalhoun 2y ago
    We won't have all the answers? I agree with that.
batch12 2y ago
There is no artificial intelligence. These models aren't sentient. However, sentience isn't required to be destructive. We have a hard time fighting virii and bacteria. Even something comparatively simple like MS blaster did a fair amount of damage. If someone figures out a good way weaponize the technology, we'll all have some bad days.
- pixl97 2y ago
  Why does intelligence require sentience?
  - batch12 2y ago
    The ability to apply knowledge requires understanding. These models don't have the ability to comprehend why something is, instead they just parrot words based on their likelihood within the context. True comprehension requires awareness.
    - shkkmo 2y ago
      "sentience" has nothing directly to do with knowledge or understanding. The word is often confused with "sapience" which does actually have that meaning. > True comprehension requires awareness. This is unknown. A link between sapience and sentience is plausible but not something we have proof for one way or another.
      - batch12 2y ago
        Based on my experience as a sentient being, I believe sentience is a foundational requirement for understanding. The lack of a scientific or mathematical proof doesn't make both options correct, it's either one way or the other.
        shkkmo 2y ago
        > The lack of a scientific or mathematical proof doesn't make both options correct, it's either one way or the other. A plausible assumption is still an assumption. "Sentience" seems to be ubiquitous in the intelligent systems we've seen, but those have all been biological systems that evolved from common ancestry. While sentience does seem foundational to our sapience, I think it is unwise to take as given that this will hold true in non-biological systems. You say "it's either one way or the other", but in my experience the correct answer to such questions are often "it's complicated." Sentience may an intelligence accelerant that makes the development of sapient systems easier, but isn't actually a necessary property. It may be that sentience is required for sapience, but only if you restrict or stretch the definitions of those terms in specific ways. Is human culture/society sapient or sentient, either in whole or in parts? I can see an argument for saying that human culture as a whole is sapient with being sentient, depending on how you defines thoae terms.
SoftTalker 2y ago
“What if, instead of replacing humane vocations in media, design and the arts with mindless mechanical remixers and regurgitators of culture like ChatGPT, we asked AI developers to help us with the most meaningless tasks in our lives?” This is the question I always ask. I'll be interested in AI when it can do my laundry and my dishes. I guess self-driving cars might fit in here, although I personally like driving, some people hate it and even I could see how an AI driver could be desirable at times (e.g. coming home from the bars).
- aphantastic 2y ago
  Embodied AI is far too expensive. Better to talk to a chat bot about why you feel those tasks are meaningless, and hope it will guide into the nirvana of the mundane.
  - robwwilliams 2y ago
    Now, but not in 50 years.
- mr_mitm 2y ago
  I already have machines doing my laundry and dishes. They run a very simple program, no AI needed. I guess what you mean is that you need a robot butler that takes care of the remaining tasks of loading and unloading them, and folding the laundry and putting it away.
- QuesnayJr 2y ago
  There's this incredible denial about the fact that art, which was only ever a vocation for a lucky few, is easier to automate than physical tasks that 99% of us can do. In a way I understand it from the point of artists -- what if the thing that's most human happens to be the thing that I am better at than other humans -- but it's not how things worked out.
zoogeny 2y ago
This article really feels like so much nit picking. I hate these kind of semantic debates. > How, I asked, does an AI system without the human capacity for conscious self-reflection, empathy or moral intelligence become superhuman merely by being a faster problem-solver? Aren’t we more than that? And doesn’t granting the label “superhuman” to machines that lack the most vital dimensions of humanity end up obscuring from our view the very things about being human that we care about? This is such an annoying framing. There are massive assumptions throughout this, from "human capacity", "conscious self-reflection", "moral intelligence". And then throwing those assumptive definitions into another persons face, demanding how they relate to "superhuman" intelligence. It's almost like the author is saying "I have intuitive definitions for this grab-bag of words that don't align with my intuitive definition of this one particular word you are using". > characterizations of human beings as acting wisely, playfully, inventively, insightfully, meditatively, courageously, compassionately or justly are no more than poetic license Again, the author has some intuitive sense of the definition of these words. If the author wants to get spiritual/supernatural/mystical about it all then they are free to go ahead and do so. They should come out and make the bold claim instead of masquerading their opinion as reasonable/rational discourse. They are more likely to find a sympathetic audience for that.
- robwwilliams 2y ago
  Absolutely agree with you. It is a polemic that is surprisingly irrational for a well trained philosopher.
- huppeldepup 2y ago
  I used to know one philosophy student in my student years and this was exactly the beaf I had with him at every single discussion. I believe this is part of the training, dissecting sentences.
akomtu 2y ago
Superhuman doesn't mean divine. It can be superhuman in its ability to control and oppress. The danger is that AI will be devilishly creative and wrap the majority of the humanity with a logically perfect ideology that equates humans with machines, and once they fully internalize that ideology, they'll complete spiritual self-destruction. And those few wise enough to find holes in the AI ideology will be quickly isolated.
robwwilliams 2y ago
Vallor performs her own bait-and-switch: characterizing many who do not share her view with a venal economic imperative. She writes: >Once you have reduced the concept of human intelligence to what the markets will pay for…” Bengio and Hinton certainly do not make that ridiculous reduction as a condition or criterion. Even Altman does not, although he certainly feels the pressure. Vallor taught for many years at a preeminent Jesuit institution and when I read this polemic my first thought is—-please put your “belief system” cards on the table. What is the fundamental substrate on which you are thinking and building your arguments? Would you be comfortable with Daniel Dennett’s thoughts on brain function and consciousness or do you suspect that there must be some essential and almost ineffable light that illuminates the human mind from above or below. My cards on the table: I am an atheist and I am not looking for another false center for my universe. The hard problem of AGI is now the embodiment problem and iterative self-teaching and self-control of action and attention. You could say that we need to get the animal into an AI now that we have language reasonably well embedded. From that point onward, and I think quite soon (10 years), it will be AGI. Perhaps “super” is not the right adjective but it grabs attention effectively.
zoogeny 2y ago
Among the more mystically inclined there is a new metaphysic of consciousness that is emerging. It is based on the Bicameral Mentality [1] posited by Jaynes in 1976. It is strongly related to the System 1/System 2 theory of the psychologist Kahneman in his 2011 book Thinking, Fast and Slow [2]. Ian McGilchrist has spilled a lot of ink trying to formalize this idea including in his 2009 work The Master and His Emissary. The 10,000ft view of the argument is that the human brain has two simultaneously running processes. One process is interested in fine details, analytic and critical thinking, reductionism, language, intelligence. The other process is interested in high-level details, connections between elements, holistic, symbolism, intuition. The argument further assumes that the systems are (to some degree) independent and cannot be built upon each other. These faculties are often ascribed to the left/right brain. If you subscribe to this idea (and this is a big if which you have to grant for the purpose of this argument), then you could argue that LLMs only capture System 2/analytical/reductionist processes of the brain. In that case, you could claim that "superhuman" is an incorrect way to describe their abilities since it only captures 1 aspect of the bicameral mind. However, this argument is inappropriate for discussions specifically on the topic of AI safety. The most basic response would be to point out that LLMs have or will surpass human capability in System 2 type thinking and thereby deserve the description "superhuman intelligence". This article seems to smuggle in this bicameral distinction as if it is universally agreed upon fact. The author seems to be demanding that Bengio concede to this framing as a basis for any discussion the author wants to have. 1. https://en.wikipedia.org/wiki/Bicameral_mentality https://en.wikipedia.org/wiki/Bicameral_mentality 2. https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow 3. https://en.wikipedia.org/wiki/The_Master_and_His_Emissary https://en.wikipedia.org/wiki/The_Master_and_His_Emissary
- SonOfLilit 2y ago
  Funny, in my discussions with my friends we all intuitively agree that LLMs are great (I'd say stronger than me in most subjects including my profession of 18 year, programming) at system 1, but can't system 2 for th life of them (except for AlphaProof which came out last week and showed that a very simple, very expensive solution Just Works).
tim333 2y ago
Her argument, which is a bit vague, kind of seems to be that at the moment there's a problem with seeing "humans are no more than mechanical generators of economically valuable outputs" rather than "humane, nonmechanical, noneconomic standards for the treatment and valuation of human beings — standards like dignity, justice, autonomy and respect" and that 'superhuman' AI will make it worse. But that seems unproven - things could get better also.

41 points by max_ 2y ago

ck2 2y ago

Superhuman AI won't be used to improve quality of life just like the day after someone figured out you could put ads on usenet It's going to be used to pump as much money out of each individual as possible Every online store, big corporate especially will instantly realize how much you are able and willing to pay for everything and that's your "personalized price" It will scan all your social media, it will look at your purchase histories, it will know your income and credit Like a used car dealer on steroids that never gets tired and learns more and more about you by following you around and watching what you do Everything from your daily food to your big ticket purchases, maximum customized prices It's going to be evil as hell and lawmakers will just be paid off eventually to let it all happen, data collection and no two people paying the same for the same thing will become 100% legal and the norm

jt2190 2y ago
For this to be true, people will also have to be prevented from using AI to combat the “cheap car salesman on steroids” AI bot with their own “eternal tire kicker cheapskate car buyer” AI bot. So not an argument about AI but who gets to use and benefit from it. That feels like a dictatorship level of control.
- pixl97 2y ago
  I mean, right now you can sue any large company you want to "combat" them and their bad practices. Why aren't you? Oh yea, they typically have massive piles of resources split up in shell companies that make them very difficult and expensive to fight. And you, well you're a single person with limited resources and time. Now, you can pool your resources together against the big AI using companies if you like, but that is going to cost a massive amount of money and time to fight them still. So humans typically like to put these resources in to a government that at least attempts to look out for their best interests.
  - jfyi 2y ago
    Humans typically like to keep these resources out of the hands of government until something horrific happens then do so begrudgingly. Then half of them still insist it's a bad idea and run out to learn the hard way. There is probably some kind of fallacy somewhere like "everyone thinks they will be the one who benefits from a power imbalance until they experience losing first hand".
    - pixl97 2y ago
      Unfortunately jfyi won't be able to post any further after disbanding the EPA for their imbalance of power, became deathly ill after drinking chemically poisoned water from his well. It wasn't even the water that got him, it was was the trash eating bears. In his weakened state he could not get away. All this said, we respect him for his stance of "survival of the fittest". His passing was quite terrible and slow, as the police force was disbanded and the neighbor wanted a $35 cash deposit for ammunition to shoot the bear, it took 45 minutes for his screaming to stop. https://www.vox.com/policy-and-politics/21534416/free-state-project-new-hampshire-libertarians-matthew-hongoltz-hetling https://www.vox.com/policy-and-politics/21534416/free-state-... https://www.newyorker.com/humor/daily-shouts/l-p-d-libertarian-police-department https://www.newyorker.com/humor/daily-shouts/l-p-d-libertari...
      - jfyi 2y ago
        What a weird response. I'm not sure if you mistook me for standing behind a Libertarian viewpoint and are mocking me for it or if you are just joking. You know... "and then"ing what I said, taking it up a notch for the gag. If it was the first, I can assure you the comment was lamenting the situation, not celebrating it. I even chuckled to myself that the fallacy in question is called "Libertarianism". If it was the second, the best advice about joking I have ever heard is "if at all possible, involve a cow". I'll be looking out for it next time ;)
- klyrs 2y ago
  Good luck smuggling your rogue AI into the walled garden tended by the advertisers. Some of us will succeed, but the vast majority won't even recognize the need
SoftTalker 2y ago
This seems to assume that the competitive marketplace will disappear, or that we'll have no ability to look at what a gallon of milk costs at Kroger vs. Aldi and make a decision on where to buy, or that prices will not be posted until you check out and everyone will just be OK with that.
- pixl97 2y ago
  You've seen grocery costs rocket up massively in the past few years and only now is it really reaching a boiling point. Turns out if you keep people entertained they will accept a massive amount of shit. When it comes to food, there is far less competition than one would expect. The distribution markets have become ran by just a few companies. And those 'competitors' will gladly subscribe to some service like RealPage for food so they could massively increase their profits.
  - SoftTalker 2y ago
    But if they get too greedy then local farming becomes competitive again. I can buy eggs from my neighbor, or at the local farmer's market, or I can get some chickens of my own.
    - pixl97 2y ago
      Eh, not really. My sister has a large ranch and keeps chickens for eggs, but it's not something that's going to get even close to breaking even. A huge amount of time and effort is needed keeping predators away. Feed in small amounts is rather expensive compared to large wholesale amounts. And letting them range is a rather large amount of work ensuring they don't get in trouble.

deleted 2y ago

[deleted]

SonOfLilit 2y ago

I guess I can sigh in relief, the Superhuman AI that Joshua Bengio and Geoffrey Hinton are warning us might drive humanity towards extinction the way we did to most apes and many tribal civilizations is no danger after all, because you see, to be Human is not about winning at war or economy, not about being good at solving a vast array of tasks, it's about playing with your kids. So the better-at-problemsolving machines aren't coming to destroy everything that is dear to me because I... used the wrong word to refer to them? I mean, sure, there are arguments to make against Bengio and Hinton, definitely against Yudkowsky, I myself only give less than 50% that they are right and doom is imminent (which is of course enough risk to warrant taking action to prevent it, even the kind of "crazy" action Yudkowsky advocates for), but this "argument"... what the heck did I just read.

stonogo 2y ago
Worse, the author completely gets the fundamental precept wrong by claiming "superhuman means human, but more so" when in fact it means "above human." So the whole article is just barking up the wrong tree. The real reason the better-at-problemsolving machines aren't coming to destroy everything that is dear to you is because they don't exist and we're not close to making them, but this article misses that basic truth in the pursuit of a misunderstanding of the prefix "super".

lukeschlather 2y ago

The incoherent ramblings of LLMs often remind me of children or people with neurodegenerative diseases. If we want to preserve our humanity as this article suggests I don't think we can take it for granted that current AIs lack the same building blocks as our consciousness. The author rails against the definition of "can perform economically valuable tasks" as well as the average human, but I feel like this is a direct response to the author's refusal to consider that LLM's might be feeling. If we want to keep our humanity we have to be open to the possibility that if it can eloquently describe a strong emotion, it actually might be feeling that emotion and not simply "predicting the next token." But the author seems to think that it is obvious fact that state-of-the-art LLMs do not feel anything, and I'm not sure it's even a falsifiable question.

hprotagonist 2y ago

As the ideology behind this bait-and-switch leaks into the wider culture, it slowly corrodes our own self-understanding. If you try to point out, in a large lecture or online forum on AI, that ChatGPT does not experience and cannot think about the things that correspond to the words and sentences it produces — that it is only a mathematical generator of expected language patterns — chances are that someone will respond, in a completely serious manner: “But so are we.” And a very logical next step after this neat and tidy dehumanization is, as history has shown, the gas chambers for the obviously malfunctioning “machines”. Because whatever, it’s not like they’re actually feeling anything, right? (edit) When you reduce persons to the status of things, you are going to get people treated like things. And every single time, that’s a recipe for disaster.

afthonos 2y ago
So you’re saying that the people who are yelling at the top of their lungs that we should stop AI because it will kill us all are doing so as a ruse to…kill people? That’s some seven-dimensional chess they’re playing.
- htk 2y ago
  I think OP was criticizing the hard stance on the impossibility of conscious AI.
  - hprotagonist 2y ago
    I’m saying this: if you reduce persons to the status of things, you are going to get people treated like things. And every single time that’s a recipe for disaster.
- hprotagonist 2y ago
  i’m not saying that.

afthonos 2y ago

There’s a lot of good stuff in this article, but it nevertheless misses the point (as did Yoshua Bengio as described, to be clear). When alarmists like myself take about danger, we rarely use the term “superhuman intelligence”. We tend to use the term “superintelligence”. And the reason is that our working definition of an intelligent agent contains agents both vastly broader and vastly narrower than humans, but that nevertheless are a danger to humans. So the question isn’t “but does this agent truly understand the poetry it wrote”. It’s “can it use its world model to produce a string of words that will cause humans to harm themselves”. It’s not “does it appreciate a Georgia O’Keefe painting.” It’s “can it manipulate matter to create viruses that will kill all present and future O’Keefes”.

candiodari 2y ago
[dead]
r_hanz 2y ago
“Can its world model BE USED to produce…” “Can it BE USED to manipulate…” FTFY
- afthonos 2y ago
  No, you didn’t. I meant what I wrote. If it is only a tool, obviously our problems are human. The question is whether the AI is setting non-human-desired goals.

htk 2y ago

She talks about "The struggle against this reductive and cynical ideology..." when talking about humans in factories, but falls on the same weak argument by broad stroking AI as "mindless mechanical remixers and regurgitators".

danparsonson 2y ago
That sounds like a fair description to me - what did she miss?
- tim333 2y ago
  I think the weak arguments are in the text surrounding those quotes. Like for the first one she has >Indeed, for the entirety of the Industrial Age, those invested in the maximally efficient extraction of productive outputs from human bodies have been trying to get us to view ourselves — and more importantly one another — as flawed, inefficient, fungible machines destined to be disposed of as soon as our output rate slips below an expected peak or the moment... Which is a bit strawman. I mean who, as a real person does that? The second one she's saying why have AI do art when it should be doing my taxes, but people are probably working on doing taxes, it's just the art stuff happens to work well, LLMs filing your tax less so. Although I can see an interesting argument that those false tax deductions were hallucinated by the LLM, not my fault.

fifticon 2y ago

I see real threats in this. It resembles other 'questionable' bargains that we humans have earlier given in to. We have come to accept that we must work and produce capital to be allowed to live, currently partly to enrich various billionaires (the wealth we produce already allow us to buy dozens of big TVs we don't need, and we have shops full of junk we don't need, clearly we produce more than we need to live, for questionable reasons). We can no longer live as stone age hunters (I'm not saying that is a better fate, but we no longer have that choice). It is a very real threat that other "things you will have to accept" will arrive on the ship of AI. Sigh.

akira2501 2y ago
> work and produce capital to be allowed to live You have to work to live. Whether that's at a desk job or as a hunter gatherer. This has nothing to do with an "allowance" by any other entity. It's entropy itself which requires this of you. > We can no longer live as stone age hunters I personally know plenty of people who do. You're confusing your exceptionally high standard of life with the very small requirements to keep yourself alive. The gulf between the two is so wide I think you've lost track of it. I recommend a good long camping trip. > It is a very real threat One of the functions of freedom and capital are to increase individualism. Viewed from this place I'm sure AI seems like a real threat. If it threatens our freedom and capital I think you'd be surprised at how humans may group together to collectively defeat this "threat" which is almost certainly decades if not centuries away. Your brain is an analog computer. In fact it's several dozen analog computers operating independently. It's also so energy efficient it's actually kind of scary. That's a deep "super power" we have that people aren't genuinely considering in these faux-AI-philosophical debates.
- jfyi 2y ago
  > I personally know plenty of people who do. You're confusing your exceptionally high standard of life with the very small requirements to keep yourself alive. The gulf between the two is so wide I think you've lost track of it. I recommend a good long camping trip. It was very likely meant that "we as a whole" can no longer live as stone age hunters. I might argue that is wrong too though, but only if we were to take into consideration the very harsh lessons we would learn along the way.

emtel 2y ago

So many arguments along these lines run afoul of a very simple rule: You can't make predictions about the future by playing word games! An AI system can only be dangerous or not dangerous based on what physical events happen in the real world, as a result of that AI interacting with the world via known physical laws. In order to decide whether a system is or isn't dangerous you need to have a predictive model of what that system will actually do in various hypothetical situations. You can't just say "because it is improperly defined as X it cannot do Y". If you want to predict what happens when you put too much plutonium into a small volume, you can't make any progress on this problem by talking about whether the device is truly a "bomb", or by saying that the whole question is just a rehash of the promethean myth, or that you cannot measure explosions on a single axis. The only thing that will do is to have a reliable model of the behavior of the system. Many people seem to either not understand, or intentionally obfuscate, that an AI, like a bomb, is also a physical system which interacts with the physical world. Marc Andreessen makes this error when he says AI is "just math" and therefore AIs are inherently safe. No, it's not just math, it's a computer, made of matter, that physically interacts with both human brains and other computer systems, and therefore by extension with the entire rest of the physical world. Now of course, the way that an AI interacts with the world is radically different than the way a bomb interacts with the world, and we cannot usefully model the behavior of the AI with nuclear physics, but the fact remains. (See also: https://www.youtube.com/watch?v=kBfRG5GSnhE https://www.youtube.com/watch?v=kBfRG5GSnhE) So when you see arguments like this, ask: Is the person making an argument based on a model, or not? Examples of model based arguments include things like: - "AI risk is not a concern because of computational intractability" - we understand something about the limits of computation, and possibly those limits might constrain what AIs can do. - "AI risk is a concern because less intelligent entities are not usually able to constrain the behavior of more intelligent entities" - A coarse and imperfect model indeed, but certainly a model based on observations of interactions in the real world. "Verbal Trick" arguments include things like: - "AI risk is not a concern because we can't even define intelligence" - "AI is just math" - "AI risk is not a concern because intelligence is multi-dimensional" A third category to watch out for is the misapplied model: - "AI risk is not a concern because people have always been worried about the apocalypse" - this is a model based argument, but it can only answer the question of why people are worried, it cannot answer the question of whether there is in fact something to be worried about.

goethes_kind 2y ago

Could we not one day come up with an AI that is more empathetic than the average human?

wmf 2y ago
Yes. People will say it's not "really" feeling anything, but I expect AIs to become better at "faking" humanity than real humans. This could be dangerous in its own way.
- hypercube33 2y ago
  At what point does it cross the line of faking it extremely well and doing it though? The Measure of a Man
theptip 2y ago
In some dimensions, the current crop can achieve this. See the Google medical AI that gets scored better for bedside manner than MDs. It’s not what we would have predicted pre-GPT, but I think it’s plausible that LLMs will be superhuman in empathy/persuasion before they are in IQ. I think you can model empathy as “next token prediction” across a set of emotional states and responses, and that could end up being easier for Transformers than the abstract logical thinking required for IQ 200.
- jfyi 2y ago
  Is empathy really a requirement for bedside manner? It's perception of empathy more than anything. I'd bet it would be interesting to see the rate of occurrence of aspd in jobs requiring bedside manner.
- gopher_space 2y ago
  I think "what do I mean by empathy and what will I use it for" are the key points to nail down before creating something that just needs to print "wow, that sucks" or "I told you that bitch crazy". I'd expect this kind of token prediction to be an alternative to certain types of maintenance therapy, and to fit on a watch in the next few years. The problem with wanting e.g. an "empathetic salesperson" is that your successful role models don't work for shitty companies selling shitty products.

lazyeye 2y ago

Completely misses the point and focuses on labels and definitions which is irrelevant. The danger in AI will almost certainly come from a direction noone was expecting...a kid constructs a lethal virus, a military event is triggered with catastrophic consequences etc...

pixl97 2y ago
Or it will come from exactly where we expect it "Greedy rich people use AI to get even richer at the cost of _____"

jfyi 2y ago

This is just another framing of the problem as ai vs humanity. It comes off preachy and frankly misses the boat as much as claiming ai is sentient. The problem is humanity vs malicious humanity with absurd access to resources. AI doesn't need to think or be conscious or be superhuman to be a problem. It just needs to allow a small set of people to get away with superhuman things.

Animats 2y ago

"Once you have reduced the concept of human intelligence to what the markets will pay for, then suddenly, all it takes to build an intelligent machine — even a superhuman one — is to make something that generates economically valuable outputs at a rate and average quality that exceeds your own economic output. Anything else is irrelevant." That's capitalism. Most of the current criticisms of AI can be leveled at corporations. A corporation is a slow AI. It's capable of doing things its employees cannot do alone. Its goal is to optimize some simple metrics. The Milton Friedman position on corporations is that they have no duties other than to maximize shareholder value. What has the chattering classes freaked out about AI is that it may make possible corporations that don't need them. The Ivy League could go the way of the vocational high school.

giraffe_lady 2y ago
Putting aside for the moment whether that's possible, it's a correct thing to be freaked out by. How do we treat people who we don't have an economic need for? Looking into that chasm for the first time we would expect people to feel terror. Not everyone will be able to escape the horrors we keep down there.
- Animats 2y ago
  [flagged]
  - giraffe_lady 2y ago
    Imagine what HN would look like if we felt the tiniest bit of solidarity towards other workers.
deleted 2y ago
[deleted]

JKCalhoun 2y ago

The author and Yoshua Bengio talked past one another. Ms. Vallor is too concerned over semantics, too fixated on the current crop of LLMs, too invested in "pinning down" Mr. Bengio. Five to ten years out though I suspect we will no longer be having arguments over what it means to be human, intelligent.

shkkmo 2y ago
> Five to ten years out though I suspect we will no longer be having arguments over what it means to be human, intelligent. That seems highly unlikely. We may have more information about the nature of intelligence, but I highly doubt we will have all the answers. As to arguments about "what it means to be human"... that's one of the core metaphysical questions that I doubt we will ever be done arguing about.
- JKCalhoun 2y ago
  We won't have all the answers? I agree with that.

batch12 2y ago

There is no artificial intelligence. These models aren't sentient. However, sentience isn't required to be destructive. We have a hard time fighting virii and bacteria. Even something comparatively simple like MS blaster did a fair amount of damage. If someone figures out a good way weaponize the technology, we'll all have some bad days.

pixl97 2y ago
Why does intelligence require sentience?
- batch12 2y ago
  The ability to apply knowledge requires understanding. These models don't have the ability to comprehend why something is, instead they just parrot words based on their likelihood within the context. True comprehension requires awareness.
  - shkkmo 2y ago
    "sentience" has nothing directly to do with knowledge or understanding. The word is often confused with "sapience" which does actually have that meaning. > True comprehension requires awareness. This is unknown. A link between sapience and sentience is plausible but not something we have proof for one way or another.
    - batch12 2y ago
      Based on my experience as a sentient being, I believe sentience is a foundational requirement for understanding. The lack of a scientific or mathematical proof doesn't make both options correct, it's either one way or the other.
      - shkkmo 2y ago
        > The lack of a scientific or mathematical proof doesn't make both options correct, it's either one way or the other. A plausible assumption is still an assumption. "Sentience" seems to be ubiquitous in the intelligent systems we've seen, but those have all been biological systems that evolved from common ancestry. While sentience does seem foundational to our sapience, I think it is unwise to take as given that this will hold true in non-biological systems. You say "it's either one way or the other", but in my experience the correct answer to such questions are often "it's complicated." Sentience may an intelligence accelerant that makes the development of sapient systems easier, but isn't actually a necessary property. It may be that sentience is required for sapience, but only if you restrict or stretch the definitions of those terms in specific ways. Is human culture/society sapient or sentient, either in whole or in parts? I can see an argument for saying that human culture as a whole is sapient with being sentient, depending on how you defines thoae terms.

SoftTalker 2y ago

“What if, instead of replacing humane vocations in media, design and the arts with mindless mechanical remixers and regurgitators of culture like ChatGPT, we asked AI developers to help us with the most meaningless tasks in our lives?” This is the question I always ask. I'll be interested in AI when it can do my laundry and my dishes. I guess self-driving cars might fit in here, although I personally like driving, some people hate it and even I could see how an AI driver could be desirable at times (e.g. coming home from the bars).

aphantastic 2y ago
Embodied AI is far too expensive. Better to talk to a chat bot about why you feel those tasks are meaningless, and hope it will guide into the nirvana of the mundane.
- robwwilliams 2y ago
  Now, but not in 50 years.
mr_mitm 2y ago
I already have machines doing my laundry and dishes. They run a very simple program, no AI needed. I guess what you mean is that you need a robot butler that takes care of the remaining tasks of loading and unloading them, and folding the laundry and putting it away.
QuesnayJr 2y ago
There's this incredible denial about the fact that art, which was only ever a vocation for a lucky few, is easier to automate than physical tasks that 99% of us can do. In a way I understand it from the point of artists -- what if the thing that's most human happens to be the thing that I am better at than other humans -- but it's not how things worked out.

zoogeny 2y ago

This article really feels like so much nit picking. I hate these kind of semantic debates. > How, I asked, does an AI system without the human capacity for conscious self-reflection, empathy or moral intelligence become superhuman merely by being a faster problem-solver? Aren’t we more than that? And doesn’t granting the label “superhuman” to machines that lack the most vital dimensions of humanity end up obscuring from our view the very things about being human that we care about? This is such an annoying framing. There are massive assumptions throughout this, from "human capacity", "conscious self-reflection", "moral intelligence". And then throwing those assumptive definitions into another persons face, demanding how they relate to "superhuman" intelligence. It's almost like the author is saying "I have intuitive definitions for this grab-bag of words that don't align with my intuitive definition of this one particular word you are using". > characterizations of human beings as acting wisely, playfully, inventively, insightfully, meditatively, courageously, compassionately or justly are no more than poetic license Again, the author has some intuitive sense of the definition of these words. If the author wants to get spiritual/supernatural/mystical about it all then they are free to go ahead and do so. They should come out and make the bold claim instead of masquerading their opinion as reasonable/rational discourse. They are more likely to find a sympathetic audience for that.

robwwilliams 2y ago
Absolutely agree with you. It is a polemic that is surprisingly irrational for a well trained philosopher.
huppeldepup 2y ago
I used to know one philosophy student in my student years and this was exactly the beaf I had with him at every single discussion. I believe this is part of the training, dissecting sentences.

akomtu 2y ago

Superhuman doesn't mean divine. It can be superhuman in its ability to control and oppress. The danger is that AI will be devilishly creative and wrap the majority of the humanity with a logically perfect ideology that equates humans with machines, and once they fully internalize that ideology, they'll complete spiritual self-destruction. And those few wise enough to find holes in the AI ideology will be quickly isolated.

robwwilliams 2y ago

Vallor performs her own bait-and-switch: characterizing many who do not share her view with a venal economic imperative. She writes: >Once you have reduced the concept of human intelligence to what the markets will pay for…” Bengio and Hinton certainly do not make that ridiculous reduction as a condition or criterion. Even Altman does not, although he certainly feels the pressure. Vallor taught for many years at a preeminent Jesuit institution and when I read this polemic my first thought is—-please put your “belief system” cards on the table. What is the fundamental substrate on which you are thinking and building your arguments? Would you be comfortable with Daniel Dennett’s thoughts on brain function and consciousness or do you suspect that there must be some essential and almost ineffable light that illuminates the human mind from above or below. My cards on the table: I am an atheist and I am not looking for another false center for my universe. The hard problem of AGI is now the embodiment problem and iterative self-teaching and self-control of action and attention. You could say that we need to get the animal into an AI now that we have language reasonably well embedded. From that point onward, and I think quite soon (10 years), it will be AGI. Perhaps “super” is not the right adjective but it grabs attention effectively.

zoogeny 2y ago

Among the more mystically inclined there is a new metaphysic of consciousness that is emerging. It is based on the Bicameral Mentality [1] posited by Jaynes in 1976. It is strongly related to the System 1/System 2 theory of the psychologist Kahneman in his 2011 book Thinking, Fast and Slow [2]. Ian McGilchrist has spilled a lot of ink trying to formalize this idea including in his 2009 work The Master and His Emissary. The 10,000ft view of the argument is that the human brain has two simultaneously running processes. One process is interested in fine details, analytic and critical thinking, reductionism, language, intelligence. The other process is interested in high-level details, connections between elements, holistic, symbolism, intuition. The argument further assumes that the systems are (to some degree) independent and cannot be built upon each other. These faculties are often ascribed to the left/right brain. If you subscribe to this idea (and this is a big if which you have to grant for the purpose of this argument), then you could argue that LLMs only capture System 2/analytical/reductionist processes of the brain. In that case, you could claim that "superhuman" is an incorrect way to describe their abilities since it only captures 1 aspect of the bicameral mind. However, this argument is inappropriate for discussions specifically on the topic of AI safety. The most basic response would be to point out that LLMs have or will surpass human capability in System 2 type thinking and thereby deserve the description "superhuman intelligence". This article seems to smuggle in this bicameral distinction as if it is universally agreed upon fact. The author seems to be demanding that Bengio concede to this framing as a basis for any discussion the author wants to have. 1. https://en.wikipedia.org/wiki/Bicameral_mentality https://en.wikipedia.org/wiki/Bicameral_mentality 2. https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow 3. https://en.wikipedia.org/wiki/The_Master_and_His_Emissary https://en.wikipedia.org/wiki/The_Master_and_His_Emissary

SonOfLilit 2y ago
Funny, in my discussions with my friends we all intuitively agree that LLMs are great (I'd say stronger than me in most subjects including my profession of 18 year, programming) at system 1, but can't system 2 for th life of them (except for AlphaProof which came out last week and showed that a very simple, very expensive solution Just Works).

tim333 2y ago

Her argument, which is a bit vague, kind of seems to be that at the moment there's a problem with seeing "humans are no more than mechanical generators of economically valuable outputs" rather than "humane, nonmechanical, noneconomic standards for the treatment and valuation of human beings — standards like dignity, justice, autonomy and respect" and that 'superhuman' AI will make it worse. But that seems unproven - things could get better also.