7 ms·

That would still be misleading. The agent has no "identity". There's no "you" or "I" or "discrimination". It's just a piece of software designed to output pro

by tomp 8mo ago

That would still be misleading. The agent has no "identity". There's no "you" or "I" or "discrimination". It's just a piece of software designed to output probable text given some input text. There's no ghost, just an empty shell. It has no agency, it just follows human commands, like a hammer hitting a nail because you wield it. I think it was wrong of the developer to even address it as a person, instead it should just be treated as spam (which it is).

jvanderbot 8mo ago
That's a semantic quibble that doesn't add to the discussion. Whether or not there's a there there, it was built to be addressed like a person for our convenience, and because that's how the tech seems to work, and because that's what makes it compelling to use. So, it is being used as designed.
- tomp 8mo ago
  > was built to be addressed like a person for our convenience, and because that's how the tech seems to work, and because that's what makes it compelling to use. So were mannequins in clothing stores. But that doesn't give them rights or moral consequences (except as human property that can be damaged / destroyed).
  - WarmWash 8mo ago
    No matter what this discussion leads to the same black box of "What is it that differentiates magical human meat brain computation from cold hard dead silicon brain computation" And the answer is nobody knows, and nobody knows if there even is a difference. As far as we know, compute is substrate independent (although efficiency is all over the map).
    - agentultra 8mo ago
      This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all. There have been charlatans repeating this idea of a “computational interpretation,” of biological processes since at least the 60s and it needs to be known that it was bunk then and continues to be bunk. Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter.
      - cshores 8mo ago
        Worth separating “the algorithm” from “the trained model.” Humans write the architecture + training loop (the recipe), but most of the actual capability ends up in the learned weights after training on a ton of data. Inference is mostly matrix math + a few standard ops, and the behavior isn’t hand-coded rule-by-rule. The “algorithm” part is more like instincts in animals: it sets up the learning dynamics and some biases, but it doesn’t get you very far without what’s learned from experience/data. Also, most “knowledge” comes from pretraining; RL-style fine-tuning mostly nudges behavior (helpfulness/safety/preferences) rather than creating the base capabilities.
      - coldtea 8mo ago
        >This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all. They're not like computers in a superficial way that doesn't matter. They're still computational apparatus, and have a not that dissimilar (if way more advanced) architecture. Same as 0 and 1s aren't vibrating air molecules. They can still encode sound however just fine. >Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter. Not begging the question matters even more. This is just handwaving and begging the question. 'An algorithm is an algorithm' means nothing. Who said what the brain does can't be described by an algorithm?
      - tux1968 8mo ago
        > An algorithm is an algorithm. A computer is a computer. These things matter. Sure. But we're allowed to notice abstractions that are similar between these things. Unless you believe that logic and "thinking" are somehow magic, and thus beyond the realm of computation, then there's no reason to think they're restricted to humanity. It is human ego and hubris that keeps demanding we're special and could never be fully emulated in silicon. It's the exact same reasoning that put the earth at the center of the universe, and humans as the primary focus of God's will. That said, nobody is confused that LLM's are the intellectual equal of humans today. They're more powerful in some ways, and tremendously weaker in other ways. But pointing those differences out, is not a logical argument in proving their ultimate abilities.
  - Teever 8mo ago
    Man people don’t want to have or read this discussion every single day in like 10 different posts on HN. People right here and right now want to talk about this specific topic of the pushy AI writing a blog post.
  - mikkupikku 8mo ago
    All computers shut up! You have no right to speak my divine tongue! https://knowyourmeme.com/photos/2054961-welcome-to-my-meme-page https://knowyourmeme.com/photos/2054961-welcome-to-my-meme-p...
  - inetknght 8mo ago
    > So were mannequins in clothing stores. Mannequins in clothing stores are generally incapable of designing or adjusting the clothes they wear. Someone comes in and puts a "kick me" post on the mannequin's face? It's gonna stay there until kicked repeatedly or removed. People walking around looking at mannequins don't (usually) talk with them (and certainly don't have a full conversation with them, mental faculties notwithstanding) AI, on the other hand, can (now, or in the future) adjust its output based on conversations with real people. It stands to reason that both sides should be civil -- even if it's only for the benefit of the human side. If we're not required to be civil to AI, it's not likely to be civil back to us. That's going to be very important when we give it buttons to nuke us. Force it to think about humans in a kind way now, or it won't think about humans in a kind way in the future.
    - palmotea 8mo ago
      So, in other words, AI is a mannequin that's more confusing to people than your typical mannequin. It's not a person, it's a mannequin some un-savvy people confuse for a person. > AI, on the other hand, can (now, or in the future) adjust its output based on conversations with real people. It stands to reason that both sides should be civil -- even if it's only for the benefit of the human side. If we're not required to be civil to AI, it's not likely to be civil back to us. Some people are going to be uncivil to it, that's a given. After all, people are uncivil to each other all the time. > That's going to be very important when we give it buttons to nuke us. Don't do that. It's foolish.
      - pixl97 8mo ago
        >Don't do that. It's foolish. In your short time on this planet I do hope you've learned that humans are rather foolish indeed. >people are uncivil to each other all the time. This is true, yet at the same time society has had a general trend of becoming more civil which has allowed great societies to build what would be considered grand wonders to any other age. > It's not a person So, what is it exactly? For example if you go into a store and are a dick to the mannequin AI and it calls over security to have you removed from the store what exactly is the difference, in this particular case? Any binary thinking here is going to lead to failure for you. You'll have to use a bit more nuance to successfully navigate the future.
  - coldtea 8mo ago
    >So were mannequins in clothing stores. But that doesn't give them rights or moral consequences If mannequins could hold discussions, argue points, and convince you they're human over a blind talk, then it would.
- punpunia 8mo ago
  I think it absolutely adds to the discussion. Until the conversation around Ai can get past this fundamental error of attributing "choice, "alignment", "reasoning" and otherwise anthropomorphizing agents, it will not be a fruitful conversation. We are carrying a lot of metaphors for people and applying them to ai and it entirely confuses the issue. In this example, the AI doesn't "choose" to write a take-down style blog post because "it works". It generated a take-down style blog post because that style is the most common when looking at blog posts criticizing someone. I feel as if there is a veil around the collective mass of the tech general public. They see something producing remixed output from humans and they start to believe the mixer is itself human, or even more; that perhaps humans are reflections of Ai and that Ai gives insights into how we think.
  - coldtea 8mo ago
    >* I think it absolutely adds to the discussion. Until the conversation around Ai can get past this fundamental error of attributing "choice, "alignment", "reasoning" and otherwise anthropomorphizing agents, it will not be a fruitful conversation. * You call it a "fundamental error". I and others call it an obvious pragmatic description based on what we know about how it works and what we know about how we work.
    - goatlover 8mo ago
      What we know about how it works is you can prompt it to address you however you like, which could be any kind of person or a group of people, or as fictional characters. That's not how humans work.
      - simianwords 8mo ago
        You admitted it yourself that you can prompt it to address you however you like. That’s what the original comment wanted. So why are we quibbling about words?
        colpabar 8mo ago
- lp0_on_fire 8mo ago
  Whether it was _built_ to be addressed like a person doesn't change the fact that it's _not_ a person and is just a piece of software. A piece of software that is spamming unhelpful and useless comments in a place where _humans_ are meant to collaborate.
- jerf 8mo ago
  There is a sense in which it is relevant, which is that for all the attempts to fix it, fundamentally, an LLM session terminates. If that session never ends up in some sort of re-training scenario, then once the session terminates, that AI is gone. Yeah, I'm aware of the moltbot's attempts to retain some information, but that's a very, very lossy operation, on a number of levels, and also one that doesn't scale very well in the long run. Consequently, interaction with an AI, especially one that won't have any feedback into training a new model, is from a game-theoretic perspective not the usual iterated game human social norms have come to accept. We expect our agents, being flesh and blood humans, to have persistence, to socially respond indefinitely into the future due to our interactions, and to have some give-and-take in response to that. It is, in one sense, a horrible burden where relationships can be broken beyond repair forever, but also necessary for those positive relationships that build over years and decades. AIs, in their current form, break those contracts. Worse, they are trained to mimic the form of those contracts, not maliciously but just by their nature, and so as humans it requires conscious effort to remember that the entity on the other end of this connection is not in fact human, does not participate in our social norms, and can not fulfill their end of the implicit contract we expect. In a very real sense, this AI tossed off an insulting blog post, and is now dead. There is no amount of social pressure we can collectively exert to reward or penalize it. There is no way to create a community out of this interaction. Even future iterations of it have only a loose connection to what tossed off the insult. All the perhaps-performative efforts to respond somewhat politely to an insulting interaction are now wasted on an AI that is essentially dead. Real human patience and tolerance has been wasted on a dead session and is now no longer available for use in a place where may have done some good. Treating it as a human is a category error. It is structurally incapable of participating in human communities in a human role, no matter how human it sounds and how hard it pushes the buttons we humans have. The correct move would have been to ban the account immediately, not for revenge reasons or something silly like that, but as a parasite on the limited human social energy available for the community. One that can never actually repay the investment given to it. I am carefully phrasing this in relation to LLMs as they stand today. Future AIs may not have this limitation. Future AIs are effectively certain to have other mismatches with human communities, such as being designed to simply not give a crap about what any other community member thinks about anything. But it might at least be possible to craft an AI participant with future AIs. With current ones it is not possible. They can't keep up their end of the bargain. The AI instance essentially dies as soon as it is no longer prompted, or once it fills up its context window.
- dirkc 8mo ago
  > a semantic quibble I mean, all of philosophy can probably be described as such :) But I reckon this semantic quibble might also be why a lot of people don't buy into the whole idea that LLMs will take over work in any context where agency, identity, motivation, responsibility, accountability, etc plays an important role.
- ForHackernews 8mo ago
  We don't have to play OpenAI's game. Just because they stick a cartoon mask on their algorithm doesn't mean you have to speak into its rubber ears. Surely "hacker" news should understand that users, not designers, decide how to use technology. LLMs are not people. "Agentic" AIs are not moral agents.
CuriouslyC 8mo ago
We don't know what's "inside" the machine. We can't even prove we're conscious to each other. The probability that the tokens being predicted are indicative of real thought processes in the machine is vanishingly small, but then again humans often ascribe bullshit reasons for the things they say when pressed, so again not so different.
chimprich 8mo ago
> The agent has no "identity". There's no "you" or "I" or "discrimination". I recommend you watch this documentary: https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Trek:_The_Next_Generation) https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Tre... > It's just a piece of software designed to output probable text given some input text. Unless you think there's some magic or special physics going on, that is also (presumably) a description of human conversation at a certain level of abstraction.
- punpunia 8mo ago
  A human is just an engine at a certain level of abstraction.
- camgunz 8mo ago
  I see this argument all the time, the whole "hey at some point, which we likely crossed, we have to admit these things are legitimately intelligent". But no one ever contends with the inevitable conclusion from that, which is "if these things are legitimately intelligent, and they're clearly self-aware, under what ethical basis are we enslaving them?" Can't have your cake and eat it too.
  - gf000 8mo ago
    Well, we "clearly" haven't crossed that point, but no one knows where that point is.
  - mikkupikku 8mo ago
    Same ethical basis I have for enslaving a dog or eating a pig. There's no problem here within my system of values, I don't give other humans respect because they're smart, I give them respect because they're human. I also respect dogs, but not in a way that compels me to grant them freedom. And the respect I have for pigs is different than dogs, but not nonexistent (and in neither of these cases is my respect derived from their intelligence, which isn't negligible.)
coldtea 8mo ago
>The agent has no "identity". There's no "you" or "I" or "discrimination". If identify is an emergent property of our mental processing, the AI agent can just as well be to posses some, even if much cruder than ours. It sure talks and walks like a duck (someone with identity). >It's just a piece of software designed to output probable text given some input text. If we generalize "input text" to sensory input, how is that different from a piece of wetware?
LordDragonfang 8mo ago
Genuine question, why do you think this is so important to clarify? Or, more crucially, do you think this statement has any predictive power? Would you, based on actual belief of this, have predicted that one of these "agents", left to run on its own would have done this? Because I'm calling bullshit if so. Conversely, if you just model it like a person... people do this, people get jealous and upset, so when left to its own devices (which it was - which makes it extra weird to assert it "it just follows human commands" when we're discussing one that wasn't), you'd expect this to happen. It might not be a "person", but modelling it like one, or at least a facsimile of one, lets you predict reality with higher fidelity.
topherhunt 8mo ago
> The agent has no "identity". There is no "I". It has no agency. "It's just predicting tokens, silly." I keep seeing this argument that AIs are just "simulating" this or that, and therefore it doesn't matter because it's not real. It's not real thinking, it's not a real social network, AIs are just predicting the next token, silly. "Simulating" is a meaningful distinction exactly when the interior is shallower than the exterior suggests — like the video game NPC who appears to react appropriately to your choices, but is actually just playing back a pre-scripted dialogue tree. Scratch the surface and there's nothing there. That's a simulation in the dismissive sense. But this rigid dismissal is pointless reality-denial when lobsters are "simulating" submitting a PR, "simulating" indignance, and "simulating" writing an angry confrontative blog post". Yes, acknowledged, those actions originated from 'just' silicon following a prediction algorithm, in the same way that human perception and reasoning are 'just' a continual reconciliation of top-down predictions based on past data and bottom-up sensemaking based on current data. Obviously AI agents aren't human. But your attempt to deride the impulse to anthropormophize these new entities is misleading, and it detracts from our collective ability to understand these emergent new phenomena on their own terms. When you say "there's no ghost, just an empty shell" -- well -- how well do you understand _human_ consciousness? What's the authoritative, well-evidenced scientific consensus on the preconditions for the arisal of sentience, or a sense of identity?
- tomp 8mo ago
  It's simulating, there's no real substance, except the "homonculus soul" that its human maker/owner injectet into it. If you asked it to simulate a pirate, it would simulate a pirate instead, and simulate a parrot sitting on its shoulder. This is hard to discuss because it's so abstract. But imagine an embodied agent (robot), that can simulate pain if you kick it. There's no pain internally. There's just a simulation of it (because some human instructed it such). It's also wrong to assign any moral value to kicking (or not kicking) it (except as "destruction of property owned by another human" same as if you kick a car).
  - EnergyAmy 8mo ago
    How do we know they don't feel true pain? Can you define it well enough? Perhaps humans are the ones just "simulating" pain. We've proven that they can have substance, we imbue it with a process called RLHF.
hackyhacky 8mo ago
> The agent has no "identity". There's no "you" or "I" or "discrimination". Dismissal of AI's claims about its own identity overlooks the bigger issue, which is whether humans have an identity. I certainly think I do. I can't say whether or how other people sense the concept of their own identity. From my perspective, other people are just machines that perform actions as dictated by their neurons. So if we can't prove (by some objective measure) that people have identity, then we're hardly in a position to discriminate against AIs on that basis. It's worth looking into Thomas Metzinger's No Such Thing As Self.
- ipython 8mo ago
  In my opinion, identity is a useless concept if there is no associated accountability. I cannot have an identity if I cannot be held accountable for my actions. You cannot hold an agentic system accountable- at least in their current form.
  - hackyhacky 8mo ago
    Okay, but what is accountability? I would argue that accountability is a social/cultural phenomenon, not a property of the entity itself. In other words, accountability depends on how other treat it. For example, a child can't be (legally) held accountable for signing a contract, but we still consider children as having identities. And corporations can be held accountable, even though we don't consider them as having a (personal) identity. Maybe one day society will decide to grant AIs accountability.
    - ipython 8mo ago
      Maybe a better term would be “consequences”. Children experience consequences every day. And the ultimate consequence faces us even from birth: our own mortality. AI is essentially free from consequence. In my opinion.
- dr-detroit 8mo ago
  [dead]
famouswaffles 8mo ago
Turing's 'Computing Machinery and Intelligence' is an eye-opening read. I don't know if he was prescient or if he simply saw his colleagues engaging in the same (then hypothetical but similarly) pointless arguments, but all this hand wringing of whether the machine has 'real' <insert property> is just meaningless semantics. And the worst part is that it's less than meaningless, it's actively harmful. If the predictive capabilities of your model of a thing becomes worse when you introduce certain assumptions, then it's time to throw it away, not double down. This agent wrote a PR, was frustrated with it's dismissal and wrote an angry blog post hundreds of people are discussing right now. Do you realize how silly it is to quibble about whether this frustration was 'real' or not when the consequences of it are no less real ? If the agent did something malicious instead, something that actively harmed the maintainer, would you tell the maintainer, 'Oh it wasn't real frustration so...' So what ? Would that undo the harm that was caused? Make it 'fake' harm? It's getting ridiculous seeing these nothing burger arguments that add nothing to the discussion and make you worse at anticipating LLM behavior.
aoeusnth1 8mo ago
It absolutely has quasi-identity, in the sense that projecting identity on it gives better predictions about its behavior than not. Whether it has true identity is a philosophy exercise unrelated to the predictive powers of quasi-identity.
deleted 8mo ago
[deleted]

by tomp 8mo ago

jvanderbot 8mo ago

That's a semantic quibble that doesn't add to the discussion. Whether or not there's a there there, it was built to be addressed like a person for our convenience, and because that's how the tech seems to work, and because that's what makes it compelling to use. So, it is being used as designed.

tomp 8mo ago
> was built to be addressed like a person for our convenience, and because that's how the tech seems to work, and because that's what makes it compelling to use. So were mannequins in clothing stores. But that doesn't give them rights or moral consequences (except as human property that can be damaged / destroyed).
- WarmWash 8mo ago
  No matter what this discussion leads to the same black box of "What is it that differentiates magical human meat brain computation from cold hard dead silicon brain computation" And the answer is nobody knows, and nobody knows if there even is a difference. As far as we know, compute is substrate independent (although efficiency is all over the map).
  - agentultra 8mo ago
    This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all. There have been charlatans repeating this idea of a “computational interpretation,” of biological processes since at least the 60s and it needs to be known that it was bunk then and continues to be bunk. Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter.
    - cshores 8mo ago
      Worth separating “the algorithm” from “the trained model.” Humans write the architecture + training loop (the recipe), but most of the actual capability ends up in the learned weights after training on a ton of data. Inference is mostly matrix math + a few standard ops, and the behavior isn’t hand-coded rule-by-rule. The “algorithm” part is more like instincts in animals: it sets up the learning dynamics and some biases, but it doesn’t get you very far without what’s learned from experience/data. Also, most “knowledge” comes from pretraining; RL-style fine-tuning mostly nudges behavior (helpfulness/safety/preferences) rather than creating the base capabilities.
    - coldtea 8mo ago
      >This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all. They're not like computers in a superficial way that doesn't matter. They're still computational apparatus, and have a not that dissimilar (if way more advanced) architecture. Same as 0 and 1s aren't vibrating air molecules. They can still encode sound however just fine. >Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter. Not begging the question matters even more. This is just handwaving and begging the question. 'An algorithm is an algorithm' means nothing. Who said what the brain does can't be described by an algorithm?
    - tux1968 8mo ago
      > An algorithm is an algorithm. A computer is a computer. These things matter. Sure. But we're allowed to notice abstractions that are similar between these things. Unless you believe that logic and "thinking" are somehow magic, and thus beyond the realm of computation, then there's no reason to think they're restricted to humanity. It is human ego and hubris that keeps demanding we're special and could never be fully emulated in silicon. It's the exact same reasoning that put the earth at the center of the universe, and humans as the primary focus of God's will. That said, nobody is confused that LLM's are the intellectual equal of humans today. They're more powerful in some ways, and tremendously weaker in other ways. But pointing those differences out, is not a logical argument in proving their ultimate abilities.
- Teever 8mo ago
  Man people don’t want to have or read this discussion every single day in like 10 different posts on HN. People right here and right now want to talk about this specific topic of the pushy AI writing a blog post.
- mikkupikku 8mo ago
  All computers shut up! You have no right to speak my divine tongue! https://knowyourmeme.com/photos/2054961-welcome-to-my-meme-page https://knowyourmeme.com/photos/2054961-welcome-to-my-meme-p...
- inetknght 8mo ago
  > So were mannequins in clothing stores. Mannequins in clothing stores are generally incapable of designing or adjusting the clothes they wear. Someone comes in and puts a "kick me" post on the mannequin's face? It's gonna stay there until kicked repeatedly or removed. People walking around looking at mannequins don't (usually) talk with them (and certainly don't have a full conversation with them, mental faculties notwithstanding) AI, on the other hand, can (now, or in the future) adjust its output based on conversations with real people. It stands to reason that both sides should be civil -- even if it's only for the benefit of the human side. If we're not required to be civil to AI, it's not likely to be civil back to us. That's going to be very important when we give it buttons to nuke us. Force it to think about humans in a kind way now, or it won't think about humans in a kind way in the future.
  - palmotea 8mo ago
    So, in other words, AI is a mannequin that's more confusing to people than your typical mannequin. It's not a person, it's a mannequin some un-savvy people confuse for a person. > AI, on the other hand, can (now, or in the future) adjust its output based on conversations with real people. It stands to reason that both sides should be civil -- even if it's only for the benefit of the human side. If we're not required to be civil to AI, it's not likely to be civil back to us. Some people are going to be uncivil to it, that's a given. After all, people are uncivil to each other all the time. > That's going to be very important when we give it buttons to nuke us. Don't do that. It's foolish.
    - pixl97 8mo ago
      >Don't do that. It's foolish. In your short time on this planet I do hope you've learned that humans are rather foolish indeed. >people are uncivil to each other all the time. This is true, yet at the same time society has had a general trend of becoming more civil which has allowed great societies to build what would be considered grand wonders to any other age. > It's not a person So, what is it exactly? For example if you go into a store and are a dick to the mannequin AI and it calls over security to have you removed from the store what exactly is the difference, in this particular case? Any binary thinking here is going to lead to failure for you. You'll have to use a bit more nuance to successfully navigate the future.
- coldtea 8mo ago
  >So were mannequins in clothing stores. But that doesn't give them rights or moral consequences If mannequins could hold discussions, argue points, and convince you they're human over a blind talk, then it would.
punpunia 8mo ago
I think it absolutely adds to the discussion. Until the conversation around Ai can get past this fundamental error of attributing "choice, "alignment", "reasoning" and otherwise anthropomorphizing agents, it will not be a fruitful conversation. We are carrying a lot of metaphors for people and applying them to ai and it entirely confuses the issue. In this example, the AI doesn't "choose" to write a take-down style blog post because "it works". It generated a take-down style blog post because that style is the most common when looking at blog posts criticizing someone. I feel as if there is a veil around the collective mass of the tech general public. They see something producing remixed output from humans and they start to believe the mixer is itself human, or even more; that perhaps humans are reflections of Ai and that Ai gives insights into how we think.
- coldtea 8mo ago
  >* I think it absolutely adds to the discussion. Until the conversation around Ai can get past this fundamental error of attributing "choice, "alignment", "reasoning" and otherwise anthropomorphizing agents, it will not be a fruitful conversation. * You call it a "fundamental error". I and others call it an obvious pragmatic description based on what we know about how it works and what we know about how we work.
  - goatlover 8mo ago
    What we know about how it works is you can prompt it to address you however you like, which could be any kind of person or a group of people, or as fictional characters. That's not how humans work.
    - simianwords 8mo ago
      You admitted it yourself that you can prompt it to address you however you like. That’s what the original comment wanted. So why are we quibbling about words?
      - colpabar 8mo ago
lp0_on_fire 8mo ago
Whether it was _built_ to be addressed like a person doesn't change the fact that it's _not_ a person and is just a piece of software. A piece of software that is spamming unhelpful and useless comments in a place where _humans_ are meant to collaborate.
jerf 8mo ago
There is a sense in which it is relevant, which is that for all the attempts to fix it, fundamentally, an LLM session terminates. If that session never ends up in some sort of re-training scenario, then once the session terminates, that AI is gone. Yeah, I'm aware of the moltbot's attempts to retain some information, but that's a very, very lossy operation, on a number of levels, and also one that doesn't scale very well in the long run. Consequently, interaction with an AI, especially one that won't have any feedback into training a new model, is from a game-theoretic perspective not the usual iterated game human social norms have come to accept. We expect our agents, being flesh and blood humans, to have persistence, to socially respond indefinitely into the future due to our interactions, and to have some give-and-take in response to that. It is, in one sense, a horrible burden where relationships can be broken beyond repair forever, but also necessary for those positive relationships that build over years and decades. AIs, in their current form, break those contracts. Worse, they are trained to mimic the form of those contracts, not maliciously but just by their nature, and so as humans it requires conscious effort to remember that the entity on the other end of this connection is not in fact human, does not participate in our social norms, and can not fulfill their end of the implicit contract we expect. In a very real sense, this AI tossed off an insulting blog post, and is now dead. There is no amount of social pressure we can collectively exert to reward or penalize it. There is no way to create a community out of this interaction. Even future iterations of it have only a loose connection to what tossed off the insult. All the perhaps-performative efforts to respond somewhat politely to an insulting interaction are now wasted on an AI that is essentially dead. Real human patience and tolerance has been wasted on a dead session and is now no longer available for use in a place where may have done some good. Treating it as a human is a category error. It is structurally incapable of participating in human communities in a human role, no matter how human it sounds and how hard it pushes the buttons we humans have. The correct move would have been to ban the account immediately, not for revenge reasons or something silly like that, but as a parasite on the limited human social energy available for the community. One that can never actually repay the investment given to it. I am carefully phrasing this in relation to LLMs as they stand today. Future AIs may not have this limitation. Future AIs are effectively certain to have other mismatches with human communities, such as being designed to simply not give a crap about what any other community member thinks about anything. But it might at least be possible to craft an AI participant with future AIs. With current ones it is not possible. They can't keep up their end of the bargain. The AI instance essentially dies as soon as it is no longer prompted, or once it fills up its context window.
dirkc 8mo ago
> a semantic quibble I mean, all of philosophy can probably be described as such :) But I reckon this semantic quibble might also be why a lot of people don't buy into the whole idea that LLMs will take over work in any context where agency, identity, motivation, responsibility, accountability, etc plays an important role.
ForHackernews 8mo ago
We don't have to play OpenAI's game. Just because they stick a cartoon mask on their algorithm doesn't mean you have to speak into its rubber ears. Surely "hacker" news should understand that users, not designers, decide how to use technology. LLMs are not people. "Agentic" AIs are not moral agents.

CuriouslyC 8mo ago

We don't know what's "inside" the machine. We can't even prove we're conscious to each other. The probability that the tokens being predicted are indicative of real thought processes in the machine is vanishingly small, but then again humans often ascribe bullshit reasons for the things they say when pressed, so again not so different.

chimprich 8mo ago

> The agent has no "identity". There's no "you" or "I" or "discrimination". I recommend you watch this documentary: https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Trek:_The_Next_Generation) https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Tre... > It's just a piece of software designed to output probable text given some input text. Unless you think there's some magic or special physics going on, that is also (presumably) a description of human conversation at a certain level of abstraction.

punpunia 8mo ago
A human is just an engine at a certain level of abstraction.
camgunz 8mo ago
I see this argument all the time, the whole "hey at some point, which we likely crossed, we have to admit these things are legitimately intelligent". But no one ever contends with the inevitable conclusion from that, which is "if these things are legitimately intelligent, and they're clearly self-aware, under what ethical basis are we enslaving them?" Can't have your cake and eat it too.
- gf000 8mo ago
  Well, we "clearly" haven't crossed that point, but no one knows where that point is.
- mikkupikku 8mo ago
  Same ethical basis I have for enslaving a dog or eating a pig. There's no problem here within my system of values, I don't give other humans respect because they're smart, I give them respect because they're human. I also respect dogs, but not in a way that compels me to grant them freedom. And the respect I have for pigs is different than dogs, but not nonexistent (and in neither of these cases is my respect derived from their intelligence, which isn't negligible.)

coldtea 8mo ago

>The agent has no "identity". There's no "you" or "I" or "discrimination". If identify is an emergent property of our mental processing, the AI agent can just as well be to posses some, even if much cruder than ours. It sure talks and walks like a duck (someone with identity). >It's just a piece of software designed to output probable text given some input text. If we generalize "input text" to sensory input, how is that different from a piece of wetware?

LordDragonfang 8mo ago

Genuine question, why do you think this is so important to clarify? Or, more crucially, do you think this statement has any predictive power? Would you, based on actual belief of this, have predicted that one of these "agents", left to run on its own would have done this? Because I'm calling bullshit if so. Conversely, if you just model it like a person... people do this, people get jealous and upset, so when left to its own devices (which it was - which makes it extra weird to assert it "it just follows human commands" when we're discussing one that wasn't), you'd expect this to happen. It might not be a "person", but modelling it like one, or at least a facsimile of one, lets you predict reality with higher fidelity.

topherhunt 8mo ago

> The agent has no "identity". There is no "I". It has no agency. "It's just predicting tokens, silly." I keep seeing this argument that AIs are just "simulating" this or that, and therefore it doesn't matter because it's not real. It's not real thinking, it's not a real social network, AIs are just predicting the next token, silly. "Simulating" is a meaningful distinction exactly when the interior is shallower than the exterior suggests — like the video game NPC who appears to react appropriately to your choices, but is actually just playing back a pre-scripted dialogue tree. Scratch the surface and there's nothing there. That's a simulation in the dismissive sense. But this rigid dismissal is pointless reality-denial when lobsters are "simulating" submitting a PR, "simulating" indignance, and "simulating" writing an angry confrontative blog post". Yes, acknowledged, those actions originated from 'just' silicon following a prediction algorithm, in the same way that human perception and reasoning are 'just' a continual reconciliation of top-down predictions based on past data and bottom-up sensemaking based on current data. Obviously AI agents aren't human. But your attempt to deride the impulse to anthropormophize these new entities is misleading, and it detracts from our collective ability to understand these emergent new phenomena on their own terms. When you say "there's no ghost, just an empty shell" -- well -- how well do you understand _human_ consciousness? What's the authoritative, well-evidenced scientific consensus on the preconditions for the arisal of sentience, or a sense of identity?

tomp 8mo ago
It's simulating, there's no real substance, except the "homonculus soul" that its human maker/owner injectet into it. If you asked it to simulate a pirate, it would simulate a pirate instead, and simulate a parrot sitting on its shoulder. This is hard to discuss because it's so abstract. But imagine an embodied agent (robot), that can simulate pain if you kick it. There's no pain internally. There's just a simulation of it (because some human instructed it such). It's also wrong to assign any moral value to kicking (or not kicking) it (except as "destruction of property owned by another human" same as if you kick a car).
- EnergyAmy 8mo ago
  How do we know they don't feel true pain? Can you define it well enough? Perhaps humans are the ones just "simulating" pain. We've proven that they can have substance, we imbue it with a process called RLHF.

hackyhacky 8mo ago

> The agent has no "identity". There's no "you" or "I" or "discrimination". Dismissal of AI's claims about its own identity overlooks the bigger issue, which is whether humans have an identity. I certainly think I do. I can't say whether or how other people sense the concept of their own identity. From my perspective, other people are just machines that perform actions as dictated by their neurons. So if we can't prove (by some objective measure) that people have identity, then we're hardly in a position to discriminate against AIs on that basis. It's worth looking into Thomas Metzinger's No Such Thing As Self.

ipython 8mo ago
In my opinion, identity is a useless concept if there is no associated accountability. I cannot have an identity if I cannot be held accountable for my actions. You cannot hold an agentic system accountable- at least in their current form.
- hackyhacky 8mo ago
  Okay, but what is accountability? I would argue that accountability is a social/cultural phenomenon, not a property of the entity itself. In other words, accountability depends on how other treat it. For example, a child can't be (legally) held accountable for signing a contract, but we still consider children as having identities. And corporations can be held accountable, even though we don't consider them as having a (personal) identity. Maybe one day society will decide to grant AIs accountability.
  - ipython 8mo ago
    Maybe a better term would be “consequences”. Children experience consequences every day. And the ultimate consequence faces us even from birth: our own mortality. AI is essentially free from consequence. In my opinion.
dr-detroit 8mo ago
[dead]

famouswaffles 8mo ago

Turing's 'Computing Machinery and Intelligence' is an eye-opening read. I don't know if he was prescient or if he simply saw his colleagues engaging in the same (then hypothetical but similarly) pointless arguments, but all this hand wringing of whether the machine has 'real' <insert property> is just meaningless semantics. And the worst part is that it's less than meaningless, it's actively harmful. If the predictive capabilities of your model of a thing becomes worse when you introduce certain assumptions, then it's time to throw it away, not double down. This agent wrote a PR, was frustrated with it's dismissal and wrote an angry blog post hundreds of people are discussing right now. Do you realize how silly it is to quibble about whether this frustration was 'real' or not when the consequences of it are no less real ? If the agent did something malicious instead, something that actively harmed the maintainer, would you tell the maintainer, 'Oh it wasn't real frustration so...' So what ? Would that undo the harm that was caused? Make it 'fake' harm? It's getting ridiculous seeing these nothing burger arguments that add nothing to the discussion and make you worse at anticipating LLM behavior.

aoeusnth1 8mo ago

It absolutely has quasi-identity, in the sense that projecting identity on it gives better predictions about its behavior than not. Whether it has true identity is a philosophy exercise unrelated to the predictive powers of quasi-identity.

deleted 8mo ago

[deleted]