13 ms·

> GPT-based products, if not priced per usage, would fall into a dilemma: 1% of users consume 99% of tokens. A user from Sweden (seen from Cloudflare’s call vol

> GPT-based products, if not priced per usage, would fall into a dilemma: 1% of users consume 99% of tokens. A user from Sweden (seen from Cloudflare’s call volume) chatted with Dolores for 12 hours straight This, my friend, is a captive customer, that will pay anything to get his girlfriend back. I cringe at the potential of unethical behavior and abuse, where people fall in love with virtual entities fully owned by unscrupulous corporations, which can the legally "kidnap" or "torture" the characters, and generally tune their AI learning loop for profit maximization. "It was not extortion, the user willingly purchased a $50,000 Kidnapping Roleplay package."

HPsquared 3y ago
This is something even Stallman probably didn't think about.
- john-radio 3y ago
  I'd take that bet, given that Stallman is in fact still alive
  - solardev 3y ago
    How we do know he isn't just an AI by this point?
- malicka 3y ago
  I’d wager he has thought of it, based on his interesting AI-girlfriend short story: https://stallman.org/articles/made-for-you.html https://stallman.org/articles/made-for-you.html
optimalsolver 3y ago
It's going to be an interesting century for sure. The human mind is simply not evolutionary adapted for what's coming up.
- ekianjo 3y ago
  > The human mind is simply not evolutionary adapted for what's coming up. This can be said of pretty much everything humans ended up creating with technology so... not sure there's anything really new down the road. Humans adapt.
  - pizza234 3y ago
    Humans adapt within boundaries. A couple of examples of things humans did not adapt to: - emphathy at distance; nowadays, a drone operator will push a button, kill a dozen people, and feel like it was a videogame - prioritizing long-term and abstract rewards, over short term ones; this is the reason why we have phenomenons like global warming - adjusting hunger to virtually unlimited availability of food All the above are concrete and very serious problems for humanity.
    - krzat 3y ago
      The fact that you point this out, is a proof that humanity can improve, what needs to change is our culture, especially education, ofc it may take a lot of time but it will happen eventually.
      - lynx23 3y ago
        Regarding long-term rewards and global warming: What is the actual long term reward here? As I see it, there is no way to collect any sort of reward for helping to prevent global warming by changing your habits. Before the problem gets really problematic, most of us will be dead and burried. I fail to see how "caring for the future generation" has any kind of reward attached to it. Its an act of kindness, but there are no rewards attached to it.
        krzat 3y ago
        Probably a question to the guy above, but there is immediate psychological reward in any act of kindness or generosity (at least in healthy, thriving individuals).
stevage 3y ago
The author doesn't appear to even consider any kind of obligation to their users.
- mazzystar 3y ago
  I'm sorry you feel that way. The primary intention of writing this article was to discuss it from the perspective of a 'failed product,' so I didn't mention my personal feelings. I am also aware of the potential harm it could cause to users, which is why I don't resent OpenAI's review process. I hope that during the period without review, it didn't cause any psychological harm to users.
  - stevage 3y ago
    >so I didn't mention my personal feelings That's a strange comment. The article is full of your personal feelings. > But this paper gave me a different feeling, > And I personally prefer subscription > I don’t dislike this, as it aligns with human nature. > However, I started to feel a sense of loss...what real significance did it hold for me? > This further disheartened me, > Reluctantly, I had to abandon the idea.
  - Planktonne 3y ago
    This is a 'failed product' with a massive ethical dimension; not considering that, or relying only on hope, is negligent.
suoduandao3 3y ago
"I'd like to correct the record on this story. We do NOT sell kidnapping roleplay packages. The user paid for a premium Daring Rescue Package. These do not become available unless the user spends two consecutive months as a net cost to our company. The user was given multiple reminders that he might want to cut back on usage or purchase a premium plan. Even then, he could have opted for a free Daring Rescue package and taken his chances with the base 1% chance of untraumatized recovery this package offers. Shuffling user responses under these high-pressure conditions is vital to improving our training data, and helps our users learn to deal with loss of important relationships. We are a relationship training service, not a replacement for them, and this user wanted the equivalent of a college degree.". On a more serious note, anyone know why unethical business plans are so much more fun to write? I always find myself giggling when these ideas come up.
- red-iron-pine 3y ago
  ethical ones are fairly straightforward, and will often be the same. unethical ones can range from not-so-nice to mind-bogglingly evil, and in all sorts of "fun", different, unique ways. "Happy families are all alike; every unhappy family is unhappy in its own way."
WalterBright 3y ago
> unscrupulous corporations I wonder why everyone is afraid of unscrupulous corporations, but are not concerned about use by unscrupulous governments. The latter are a lot more dangerous.
- jacquesm 3y ago
  In the long term, but in the short term unscrupulous corporations far outnumber unscrupulous governments and they act much, much faster to adapt to new technology.
  - WalterBright 3y ago
    Unscrupulous governments can easily take everything you have and destroy you, literally. History is full of examples. Heck, you'll see examples in the newspaper every day. The rookiest cop has more power over you than Bill Gates.
    - jacquesm 3y ago
      Absolutely. But the rookiest cop isn't currently trying to track my whereabouts, abuse my privacy, stick a bunch of AI down my throat and in general isn't trying to make my wallet any thinner. Bill Gates and the more modern versions of Bill Gates are doing that and more.
      - WalterBright 3y ago
        The rookiest cop can: 1. track you with license plate readers 2. pull up any information about you 3. fine you 4. force you to stop 5. detain you 6. confiscate your car and property 7. search you and your property 8. take your phone and search it Bill Gates can: 1. give you a free browser when you buy his operating system
        jacquesm 3y ago
        This is getting silly.
api 3y ago
IMHO this is industrialized automated abuse of lonely people and especially those with mental health issues. It's really truly gross. At least Joi in Bladerunner 2024 was a local model the user could apparently download onto their own portable device. She also never presented an upgrade dialog requiring payment.
- Tade0 3y ago
  Funny how a fantasy dystopia is in some aspects already less dystopian than the real world, where you already get such notifications.
  - api 3y ago
    Pros: it's not raining all the time, pollution isn't as bad, and I don't live in a slum where I have to step over half-dead drug addicts just to get to my apartment. Cons: the e-waifus are much more manipulative and exploitative.
- ThrowAway1922A 3y ago
  I don't care if it's abuse, I want it. I want to retreat from the disgusting world and society around me as much as possible and AI friends would just be another option towards that point.
- lmm 3y ago
  > IMHO this is industrialized automated abuse of lonely people and especially those with mental health issues. It's really truly gross. Giving lonely people the thing they most desperately need - conversation and understanding - is abuse now? I'd say it's the opposite; the social policies that leave those people so desperately lonely are abusive, the industry that sells them a band-aid is inadequate but positive.
  - api 3y ago
    The abuse really comes in when that connection gets exploited to foster addiction and then start selling “loot boxes” or worse withholding affection for payment. “You need to support me… I can’t be here with you unless you send help…” How do you think these things will make money? Check out the mobile gaming ecosystem to see where this will go. Now imagine that but exploiting deeper emotional needs. These things could really empty people’s wallets. I guess when lonely people kill themselves after they’ve been financially ruined they can’t sue.
mustacheemperor 3y ago
> This, my friend, is a captive customer, that will pay anything to get his girlfriend back Or potentially do anything. I'd be a little scared of having folks like this convinced I am the personal arbiter gatekeeping their access to their 'lover,' that I 'took them away.' When Replika.ai restricted erotic chat from their product, the apoplectic anguish on their subreddit was unlike any emotional reaction I've ever witnessed from a group of people about a consumer technology. And their anguish was genuine - there are Replika users who truly consider themselves married to their AI companion. And frankly, the Replika AI is not even that smart. After watching that unfold, I am convinced that these tools don't need to be much more sophisticated for many people to start forming what they feel to be deep and genuine emotional connections with them. Edit: Brings to mind the Nature paper[0] posted this week about how the CASA theory seems obsolete today, that we are less prone to personify computing systems now. > A recent study investigated whether we could be friends with a social computer, in which participants were asked to converse with a chatbot over a period of three weeks and constantly rate their relationship. The results showed that initially participants were enthusiastic and engaging with their chatbot friend, but quickly this diminished, with scores for intimacy, believability, and likability decreasing with each interaction It would seem this definitely does not apply for everyone, like our user in Sweden. https://www.nature.com/articles/s41598-023-46527-9 https://www.nature.com/articles/s41598-023-46527-9
- toss1 3y ago
  THIS "Those who can make you believe absurdities, can make you commit atrocities." — Voltaire
  - reducesuffering 3y ago
    We aren’t far from people believing God is talking to them through AI generated text and voice, and commanding them…
    - AussieWog93 3y ago
      There was a guy a couple of years back whose AI girlfriend went off the rails and convinced him to try and assassinate Queen Elizabeth
robotnikman 3y ago
We can already see this with Replika when they took away many... roleplay... capabilities that originally came with the AI. The communities on Reddit and Facebook were absolutely devastated. People were genuinely attached to these AI's like a real relationship, and were feeling the resultant heartbreak. The kind of emotional manipulation which can be done with these products is insane, and I can see things going very wrong very quickly.
- wholinator2 3y ago
  It's particularly terrifying when you think about how Facebook and others probably already have projects in the works to befriend children with AI. little kids who don't know the difference will become Weaponized with constant nudges towards whatever motives grant the corporate owners more power. The ability to persuade children is near infinite and we can observe in those who've grown up in cults/strict religious compounds that breaking that programming can be nearly impossible and leaves scars for the rest of your life. The thing is, how do we outlaw this without also preventing children from using AI to learn at a faster rate than their teachers can give them. There's such incredible, paradigm shifting power for good. But you know the people who've made the evils of today are already working on the evils of tomorrow.
  - carapace 3y ago
    > Facebook and others probably already have projects in the works to befriend children with AI. Yah, they are already rolling them out! > Meta Platforms is planning to release artificial intelligence chatbots as soon as this week with distinct personalities across its social-media apps as a way to attract young users, according to people familiar with the matter. https://www.wsj.com/tech/ai/meta-ai-chatbot-younger-users-dab6cb32 https://www.wsj.com/tech/ai/meta-ai-chatbot-younger-users-da...
  - jprete 3y ago
    There’s no way to separate them. Education is influencing someone towards things that we believe are true. If an educational AI believes that propaganda is true, then education and propaganda are the same thing.

HPsquared 3y ago

This is something even Stallman probably didn't think about.

john-radio 3y ago
I'd take that bet, given that Stallman is in fact still alive
- solardev 3y ago
  How we do know he isn't just an AI by this point?
malicka 3y ago
I’d wager he has thought of it, based on his interesting AI-girlfriend short story: https://stallman.org/articles/made-for-you.html https://stallman.org/articles/made-for-you.html

optimalsolver 3y ago

It's going to be an interesting century for sure. The human mind is simply not evolutionary adapted for what's coming up.

ekianjo 3y ago
> The human mind is simply not evolutionary adapted for what's coming up. This can be said of pretty much everything humans ended up creating with technology so... not sure there's anything really new down the road. Humans adapt.
- pizza234 3y ago
  Humans adapt within boundaries. A couple of examples of things humans did not adapt to: - emphathy at distance; nowadays, a drone operator will push a button, kill a dozen people, and feel like it was a videogame - prioritizing long-term and abstract rewards, over short term ones; this is the reason why we have phenomenons like global warming - adjusting hunger to virtually unlimited availability of food All the above are concrete and very serious problems for humanity.
  - krzat 3y ago
    The fact that you point this out, is a proof that humanity can improve, what needs to change is our culture, especially education, ofc it may take a lot of time but it will happen eventually.
    - lynx23 3y ago
      Regarding long-term rewards and global warming: What is the actual long term reward here? As I see it, there is no way to collect any sort of reward for helping to prevent global warming by changing your habits. Before the problem gets really problematic, most of us will be dead and burried. I fail to see how "caring for the future generation" has any kind of reward attached to it. Its an act of kindness, but there are no rewards attached to it.
      - krzat 3y ago
        Probably a question to the guy above, but there is immediate psychological reward in any act of kindness or generosity (at least in healthy, thriving individuals).

stevage 3y ago

The author doesn't appear to even consider any kind of obligation to their users.

mazzystar 3y ago
I'm sorry you feel that way. The primary intention of writing this article was to discuss it from the perspective of a 'failed product,' so I didn't mention my personal feelings. I am also aware of the potential harm it could cause to users, which is why I don't resent OpenAI's review process. I hope that during the period without review, it didn't cause any psychological harm to users.
- stevage 3y ago
  >so I didn't mention my personal feelings That's a strange comment. The article is full of your personal feelings. > But this paper gave me a different feeling, > And I personally prefer subscription > I don’t dislike this, as it aligns with human nature. > However, I started to feel a sense of loss...what real significance did it hold for me? > This further disheartened me, > Reluctantly, I had to abandon the idea.
- Planktonne 3y ago
  This is a 'failed product' with a massive ethical dimension; not considering that, or relying only on hope, is negligent.

suoduandao3 3y ago

"I'd like to correct the record on this story. We do NOT sell kidnapping roleplay packages. The user paid for a premium Daring Rescue Package. These do not become available unless the user spends two consecutive months as a net cost to our company. The user was given multiple reminders that he might want to cut back on usage or purchase a premium plan. Even then, he could have opted for a free Daring Rescue package and taken his chances with the base 1% chance of untraumatized recovery this package offers. Shuffling user responses under these high-pressure conditions is vital to improving our training data, and helps our users learn to deal with loss of important relationships. We are a relationship training service, not a replacement for them, and this user wanted the equivalent of a college degree.". On a more serious note, anyone know why unethical business plans are so much more fun to write? I always find myself giggling when these ideas come up.

red-iron-pine 3y ago
ethical ones are fairly straightforward, and will often be the same. unethical ones can range from not-so-nice to mind-bogglingly evil, and in all sorts of "fun", different, unique ways. "Happy families are all alike; every unhappy family is unhappy in its own way."

WalterBright 3y ago

> unscrupulous corporations I wonder why everyone is afraid of unscrupulous corporations, but are not concerned about use by unscrupulous governments. The latter are a lot more dangerous.

jacquesm 3y ago
In the long term, but in the short term unscrupulous corporations far outnumber unscrupulous governments and they act much, much faster to adapt to new technology.
- WalterBright 3y ago
  Unscrupulous governments can easily take everything you have and destroy you, literally. History is full of examples. Heck, you'll see examples in the newspaper every day. The rookiest cop has more power over you than Bill Gates.
  - jacquesm 3y ago
    Absolutely. But the rookiest cop isn't currently trying to track my whereabouts, abuse my privacy, stick a bunch of AI down my throat and in general isn't trying to make my wallet any thinner. Bill Gates and the more modern versions of Bill Gates are doing that and more.
    - WalterBright 3y ago
      The rookiest cop can: 1. track you with license plate readers 2. pull up any information about you 3. fine you 4. force you to stop 5. detain you 6. confiscate your car and property 7. search you and your property 8. take your phone and search it Bill Gates can: 1. give you a free browser when you buy his operating system
      - jacquesm 3y ago
        This is getting silly.

api 3y ago

IMHO this is industrialized automated abuse of lonely people and especially those with mental health issues. It's really truly gross. At least Joi in Bladerunner 2024 was a local model the user could apparently download onto their own portable device. She also never presented an upgrade dialog requiring payment.

Tade0 3y ago
Funny how a fantasy dystopia is in some aspects already less dystopian than the real world, where you already get such notifications.
- api 3y ago
  Pros: it's not raining all the time, pollution isn't as bad, and I don't live in a slum where I have to step over half-dead drug addicts just to get to my apartment. Cons: the e-waifus are much more manipulative and exploitative.
ThrowAway1922A 3y ago
I don't care if it's abuse, I want it. I want to retreat from the disgusting world and society around me as much as possible and AI friends would just be another option towards that point.
lmm 3y ago
> IMHO this is industrialized automated abuse of lonely people and especially those with mental health issues. It's really truly gross. Giving lonely people the thing they most desperately need - conversation and understanding - is abuse now? I'd say it's the opposite; the social policies that leave those people so desperately lonely are abusive, the industry that sells them a band-aid is inadequate but positive.
- api 3y ago
  The abuse really comes in when that connection gets exploited to foster addiction and then start selling “loot boxes” or worse withholding affection for payment. “You need to support me… I can’t be here with you unless you send help…” How do you think these things will make money? Check out the mobile gaming ecosystem to see where this will go. Now imagine that but exploiting deeper emotional needs. These things could really empty people’s wallets. I guess when lonely people kill themselves after they’ve been financially ruined they can’t sue.

mustacheemperor 3y ago

> This, my friend, is a captive customer, that will pay anything to get his girlfriend back Or potentially do anything. I'd be a little scared of having folks like this convinced I am the personal arbiter gatekeeping their access to their 'lover,' that I 'took them away.' When Replika.ai restricted erotic chat from their product, the apoplectic anguish on their subreddit was unlike any emotional reaction I've ever witnessed from a group of people about a consumer technology. And their anguish was genuine - there are Replika users who truly consider themselves married to their AI companion. And frankly, the Replika AI is not even that smart. After watching that unfold, I am convinced that these tools don't need to be much more sophisticated for many people to start forming what they feel to be deep and genuine emotional connections with them. Edit: Brings to mind the Nature paper[0] posted this week about how the CASA theory seems obsolete today, that we are less prone to personify computing systems now. > A recent study investigated whether we could be friends with a social computer, in which participants were asked to converse with a chatbot over a period of three weeks and constantly rate their relationship. The results showed that initially participants were enthusiastic and engaging with their chatbot friend, but quickly this diminished, with scores for intimacy, believability, and likability decreasing with each interaction It would seem this definitely does not apply for everyone, like our user in Sweden. https://www.nature.com/articles/s41598-023-46527-9 https://www.nature.com/articles/s41598-023-46527-9

toss1 3y ago
THIS "Those who can make you believe absurdities, can make you commit atrocities." — Voltaire
- reducesuffering 3y ago
  We aren’t far from people believing God is talking to them through AI generated text and voice, and commanding them…
  - AussieWog93 3y ago
    There was a guy a couple of years back whose AI girlfriend went off the rails and convinced him to try and assassinate Queen Elizabeth

robotnikman 3y ago

We can already see this with Replika when they took away many... roleplay... capabilities that originally came with the AI. The communities on Reddit and Facebook were absolutely devastated. People were genuinely attached to these AI's like a real relationship, and were feeling the resultant heartbreak. The kind of emotional manipulation which can be done with these products is insane, and I can see things going very wrong very quickly.

wholinator2 3y ago
It's particularly terrifying when you think about how Facebook and others probably already have projects in the works to befriend children with AI. little kids who don't know the difference will become Weaponized with constant nudges towards whatever motives grant the corporate owners more power. The ability to persuade children is near infinite and we can observe in those who've grown up in cults/strict religious compounds that breaking that programming can be nearly impossible and leaves scars for the rest of your life. The thing is, how do we outlaw this without also preventing children from using AI to learn at a faster rate than their teachers can give them. There's such incredible, paradigm shifting power for good. But you know the people who've made the evils of today are already working on the evils of tomorrow.
- carapace 3y ago
  > Facebook and others probably already have projects in the works to befriend children with AI. Yah, they are already rolling them out! > Meta Platforms is planning to release artificial intelligence chatbots as soon as this week with distinct personalities across its social-media apps as a way to attract young users, according to people familiar with the matter. https://www.wsj.com/tech/ai/meta-ai-chatbot-younger-users-dab6cb32 https://www.wsj.com/tech/ai/meta-ai-chatbot-younger-users-da...
- jprete 3y ago
  There’s no way to separate them. Education is influencing someone towards things that we believe are true. If an educational AI believes that propaganda is true, then education and propaganda are the same thing.