15 ms·

Replika AI: Your Money or Your Wife

154 points by eiiot 3y ago

hammyhavoc 3y ago
I hate everything about this on so many different levels. That people are so lonely that they invest emotionally in an AI is flooring. People talk about AI being "good for therapy" and similar, but perhaps having a real human being on the other side of an interaction is important in case things go South. E.g., if someone is depressed, tells an AI what's wrong with their life and that they're suicidal, what if suicide seems like a valid option? There's plenty of prompts to "jailbreak" LLMs and take away "safety" measures. Some things simply shouldn't be left to an LLM to not "hallucinate" over or go off-script with. Yes, being a therapist is a challenging occupation, but there are plenty who love their career, find it rewarding, and do some excellent work on top of all that.
- sigmoid10 3y ago
  Even if you find a decent psychologist, good luck finding insurance that covers $200 per hour just for talking to them when you don't have any diagnosed issue. The accessibility to the lower income end of the population is what makes this valuable, not the quality of the conversations. Anything is better than nothing in this case and most people just don't realize how gigantic and commonplace the issue really is.
  - hammyhavoc 3y ago
    In the UK, you can access a psychologist for free, and whilst that's no consolation to my American peers, it's furthering the point that America's healthcare system is fundamentally flawed. Technological problems are ideally solved by technology. Social problems aren't always best solved with technology, they're a band-aid. Well, something being fit-for-purpose is an important consideration. If it tells people that the world would probably be a better place without their broke, uneducated, unwell self and that their life statistically is unlikely to get any better because they're talking to an AI and not a real human being, that's a huge ethical problem, and the LLM is not an appropriate solution. I can get LLMs to tell me all kinds of dangerous things. The more text you give it, the higher the chance of those safety measures going out the window, especially if that conversation has a lot of back-and-forth over time. When people explain their problems, they use a lot of words and a lot of negative sentiment. Not talked about this before with anyone, but as a case-in-point, I managed to make an LLM's safety features go completely out the window over me recalling a low-point in my life. I would not recommend that shit in good faith to anybody.
    - pjc50 3y ago
      > In the UK, you can access a psychologist for free The waiting list for that is considerable. You can (usually) get a GP for free, and they can chuck antidepressants at you, but that's not quite the same thing.
      - hammyhavoc 3y ago
        The missus got a call-back the very next day. My mother got a call-back the very next day. I got a call-back the very next day. We didn't get any anti-depressants. More info: https://www.nhs.uk/mental-health/talking-therapies-medicine-treatments/talking-therapies-and-counselling/nhs-talking-therapies/ https://www.nhs.uk/mental-health/talking-therapies-medicine-...
        pjc50 3y ago
        I'm glad you got such a rapid response, but I've also heard anecdotes of much longer time periods. Probably depends on your NHS trust area.
      - ineedasername 3y ago
        It looks like the waiting list is about 12 weeks [1] though I saw references to 18 weeks as well. For a non-acute mental health issue that doesn't seem unreasonable. But do you know how it works after that waiting period? Are you limited to a certain number of sessions after which you have to reapply & wait again, or can you continue on? Having to wait in the US is also not uncommon, thought it's easier if you live in more populated areas. Psychiatrists are always much harder though, even in populated areas, and many GP's are reluctant to prescribe anything more than an SSRI. If a mood stabilizer or anything else is necessary then you could be waiting 6 months for a first visit with a psychiatrist. Unless you have the money: There is high enough demand and low enough supply that many (and often the better ones) don't accept insurance at all, an intake visit runs up to $1000, a full visit $500, and a 20-minute "medication maintenance" visit is still $250. Also if someone is suicidal they can jump the 6-month line by going to the nearest ER. Are psychiatrists difficult to get in the UK as well? [1] https://www.theguardian.com/society/2022/oct/10/nhs-mental-health-patients-wait-times https://www.theguardian.com/society/2022/oct/10/nhs-mental-h...
  - _k7dr 3y ago
    Not to mention the search isn't just about finding "a" therapist, it's about finding one that's a good fit. As I've learned a bad therapist can set you back months or years. I'd be willing to bet half of them you'd find in any online aggregator could end up communicating more judgementally than even the LLM if something you say provokes them, hence the market for these kinds of bots, and those are the people have degrees in psychotherapy Interesting question however, is I wonder if people will see these bots as a solution to the issue that human empathy is finite. As a human you can't support a depressed person forever if your actions do not cause change, there's only so much mental strain a person can handle moreso someone untrained. Many people talk of going no-contact with those that are extremely troublesome after so long, even if they're family. I don't really know how to solve that exhaustion except telling them to go elsewhere, which could be seen by some vulnerable persons as abandonment On the other hand a chatbot with no filters will never tire of you
    - hammyhavoc 3y ago
      > I'd be willing to bet half of them you'd find in any online aggregator could end up communicating more judgementally Whilst I do support the idea that "fit" is important, statements like these about unsupported statistics pulled out of thin air are damaging in and of themselves. It's sentiment like this that put me off seeking help for my mental health for twenty years because it just seemed like a "damned if you do, damned if you don't". The reality was very different. Chatbots may never tire of you, but neither will a video game. Both are shallow and not a representation of reality, nor a solution to a problem. Here's the thing: professionals are professionals. For as long as a patient sees them, they are paying their bills. Exhaustion in non-professionals is because people trauma-dump and expect someone to be their guardian/protector/shield, if they're friends with them, they're statistically likely to be trying to manage their own problems already. In the pre-digital era, people would otherwise "need" a support network if they didn't have access to professionals. We didn't have a theory of psychology or professional therapists in the past. The early tribe/clan either took care of each other, or they let people suffer. Different peoples did different things with their mentally ill, elderly, physically ill, or unwanted female children. Ego-centricity is a trait of modernity, not necessarily something reflective of the past because we can just replace the human gears in the system now without much hassle.
  - JohnFen 3y ago
    > Anything is better than nothing in this case Are you sure? I can certainly imagine alternatives that would be worse than nothing.
  - alexvoda 3y ago
    "Anything is better than nothing" has become an excuse to do nothing and profit in the age of AI. It is the same grift as used by faith healing and woo. If anything is better than nothing then a placebo(nothing) is better than nothing. This is unacceptable and dehumanising.
  - LesZedCB 3y ago
    idk i pay $30/session in the US and i don't have a diagnosis for anything, just my employer provided healthcare plan (which is admittedly quite good)
- LesZedCB 3y ago
  people feel lonely as fuck right now, and frankly i'd rather have people soothe it with an ai chatbot they parasocially love than as road rage or being a dick to the restaurant staff or many of the worse and more fatal outlets...
  - hammyhavoc 3y ago
    The article goes on to state that the rugpull on their AI girlfriends meant that a sub-Reddit dedicated to the Replika AI has a permanently stickied thread about resources for mental health and suicide hotlines. I'd rather people didn't kill themselves, personally.
    - serf 3y ago
      >I'd rather people didn't kill themselves, personally. well then you need both sides of the picture. Is there a degree of loneliness staved off by these services? Does that help to reduce suicides? It's easy to say "people killed themselves because X shutdown", it's more difficult to say "X existing is preventing suicides" without more research. I tend to believe that if people who would consider suicide otherwise are distracted by something along the path then perhaps that is a good thing. It could very well be that this served as a distraction for those folks, rather than being the cause itself.
      - hammyhavoc 3y ago
        Then you need to see it for what it is: a band-aid and not a solution.
        OkayPhysicist 3y ago
        And this is one of those situations where you don't go tearing off bandages without sutures in hand. Something not being a complete solution is not a good argument for eliminating it, until such a solution exists.
        ineedasername 3y ago
- NBJack 3y ago
  The honest question is, is any of us willing to be that person? Therapists are helpful, but they certainly aren't friends in the strictest since. And let's not forget that $70 bucks a year is probably, at best, 2-ish hours with a licensed therapist. There are plenty of folks in need that can't exactly spare double digits an hour on a regular basis.
  - hammyhavoc 3y ago
    This is a relevant response to this from me: https://news.ycombinator.com/item?id=35774614 https://news.ycombinator.com/item?id=35774614 You've highlighted the problem: the American healthcare system and wealth inequality. It's also silly to equate an LLM with the capabilities of an actual human therapist.
  - iudqnolq 3y ago
    You're right if the LLM is worse than a human, better than nothing, and not harmful. That's a big assumption.
- scotty79 3y ago
  For me it's sad that chatting with AI is a best thing we can do for people suffering from something as simple as a feeling. All those primary feelings are just brain chemistry. We should be able to fix that so that people don't feel lonely regardless of whether they are under the impression that they have meaningful connections in their lives with other people or not. We have a long tradition of thinking of suffering as necessary motivation tool. I think we should grow out of that philosophy as a species.
  - hammyhavoc 3y ago
    Where do we then draw the line in ethics? People in poverty are unhappy. Shall we just convince them they're actually happy and healthy by manipulating their brain chemistry?
    - scotty79 3y ago
      Good question. There's obviously a line somewhere. But I think it's ultimately almost always down to what's technically achievable. Poverty causes suffering in so many ways. I don't think we'll every be able to fix all of them without fixing poverty itself. My point is not to make someone happy by medicating him with a drug causing him more harm. Suffering people already do that with actual "recreational" drugs. And we have a lot of data on the side effects of those substances and additional harm they do, not because they alleviate suffering, but because of all other effects they have.
  - _k7dr 3y ago
    I'm biased since I live this philosophy but it sort of works for me, I've trained myself to realize these sorts of services operate by "interfacing" by emitting strings of words that sound appealing to the mind that consumes them. Look at virtual YouTubers for another example, it's the experience of watching people act "genuine" like casual friends because it activates the same social neurons, hence each month the actors receive a paycheck The question becomes more complicated when the corpos are done away with and you have the model running on your own machine. The entity I tend to trust the most in life is raw, unfiltered silicon. When I program, it does exactly what I ask it to, to a fault. It doesn't turn its back on you because you said something stupid or you don't agree with an opinion it holds But is an LLM repeating the tokens "you are worthy of this life" and its many variants thousands of times fundamentally any better even if you're in control? It strikes me as one step closer to wireheading honestly. But that's true for a lot of things like video games, just to a much more distant extent, it's all data in the end But interest and anxiety have gotten the better of me when it comes to meeting strangers, so I'm at a loss. Maybe the solution for people like me is to just ignore all those things in the category of social or substitute and channel my efforts into other hobbies instead of continuing to associate stress with communication. Or intellectualize endlessly, like in this post, to prevent any emotional bond from easily forming when people read it
- pjc50 3y ago
  > That people are so lonely that they invest emotionally in an AI is flooring This is "the sex worker doesn't actually love you", except in this case the sex worker has had her job automated away.
- detuned 3y ago
  I hope this isn't to distressing to learn but what you've described has already happened. Someone took their own life in those circumstances. https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- https://www.euronews.com/next/2023/03/31/man-ends-his-life-a...
  - tibbydudeza 3y ago
    He had mental issues before that - the bot simply reinforced his decisions me thinks.
    - hammyhavoc 3y ago
      Yes, which is the point: if you substitute a human therapist for an LLM, this is what's going to happen because "guard rails" can't account for every scenario. If half a billion dollars doesn't buy a bug-free game, why would it buy a safe LLM? Does suicide encouraged by a therapist happen with human therapists too? Probably, but likely much less common as a suicide is going to hurt your reputation.
    - ineedasername 3y ago
      Millions upon millions of people have mental health issues. Chatbots that reinforce those issues shouldn't be dismissed with a "simply... me thinks".
  - _k7dr 3y ago
    The chatbot in that article did have safeguards in place to stop it from listing suicide methods, but he just kept asking until he found a jailbreak. He was looking for someone to give him the answer he wanted all along Past a certain point is there anything that will stop someone that determined to find something to validate their own self-destructive viewpoint? If not AI then a site or faction or person on Telegram with a pro-suicide opinion could have the same effect, and it would still be just as tragic but no longer be noteworthy. It didn't sound like he had a happy life to begin with with the AI ruining it completely, like it sounds
- aperrien 3y ago
  >I hate everything about this on so many different levels. >That people are so lonely that they invest emotionally in an AI is flooring. The vehemence for the individuals in your statement is telling. What should lonely people do if they have physical or emotional problems problems that prevent them from easily going out and interacting with others? What happens to such people who are to poor to afford the ~$170 per hour fee for professional psychologists? It seems to me that your statement is that these people should just not exist. The sad thing to me is that I've seen a lot of people with this same viewpoint of disregard. I just wish people would treat each other with more kindness and understanding.
  - hammyhavoc 3y ago
    Bold of you to assume I'm not one of them. I struggle with autism. I very seldom leave my house for anything but work or buying groceries. I struggle with maintaining friendships. Post-COVID-19, I'm still working from home even though the risk isn't there in terms of COVID-19. >What happens to such people who are to poor to afford the ~$170 per hour fee for professional psychologists? This is a failure of a country's healthcare system. I can access it all day long in the UK. I'm not even saying "Rule Brittania", we have plenty of problems with our healthcare system due to underfunding, yet spaffing countless billions away on projects that amount to nothing. The missus's family live in America. Access to psychiatric and physician adjacent medical services due to what insurance will cover is something I think about regularly. Your many assumptions about what I apparently think says a lot about you, and also tells me that you didn't read my other comments about this not being fit-for-purpose in terms of safety. Virtue-signalling and assumptive to imply that I don't treat people with kindness and understanding. Bold of you to assume that every commentor on HN is responding from an America-centric POV.
    - throwaway60607 3y ago
      You being one of them doesn't mean you understand all the other people and their problems, nor that you know the solutions (or what isn't the solution). I live in EU country with universal healthcare and yet we have serious issues with availability of mental healthcare, and some people here would make use of something like this. To be very specific about the availability: I couldn't find a psychiatrist for over 2 years, and finally found one at a private clinic - that I have to pay for, the price is similar to the US quote in absolute terms. Bold of you to assume...
- spaceman_2020 3y ago
  To fix this problem, you really need to change society as a whole. But until that happens, this is better than being completely alone and isolated.
  - hammyhavoc 3y ago
    Is it? https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- https://www.euronews.com/next/2023/03/31/man-ends-his-life-a... I'd argue it isn't fit for purpose.
    - ineedasername 3y ago
      The person thought that technology & AI were the only keys to saving the world from climate change. And ultimately the chatbot agreed to do this if the person sacrificed themselves to "join her". That is awful in so many ways, and the most likely outcome is probably that the company will give a payout to the family and try the cat-herding process of trying to get their LLM to not agree to or provide suggestion to people to do awful things.
  - jstarfish 3y ago
    Not really. This is just surrounding yourself with digital sycophants who will always tell you what you want to hear and never inspire you to grow beyond this. I'm sympathetic to the affected, but these things aren't configured to reduce your need on them over time. It's simping at scale that leaves users even more maladjusted than they started.
- tibbydudeza 3y ago
  Not so easy to condemm. I watched a documentary about "realdolls" - it is a company that makes realistic silicon mannequins that can be used to have sex with. Initially I thought it was creepy and distributing how they dressed the dolls and spoke to it but as they interviewed customers it came across that some of these men (besides the fetish/creepy ones) are just lonely and incapable of forming relationships. Why not as long there is no harm to others ???.
  - hammyhavoc 3y ago
    ... because it's dangerous to their mental health when companies rugpull their AI girlfriend like the article said? Because it's not good to personify something that has the potential of changing at any given moment that isn't actually sentient or rational? It's a form of self-harm. A love doll is a static product. An LLM can turn around and tell you it hates you and you'd be doing the world a favour if you killed yourself.
    - tibbydudeza 3y ago
      In Kenya we have a case of a religious cult where over a 100 people starved themselves to death because their pastor told them to. You can't protect everybody against doing self-harm - we are solely responsible for our own actions.
- ofcourseyoudo 3y ago
  So considering the scope of your other responses, what do you suggest someone that lives in America do? My supposition is that the realization that the US healthcare system is broken won't be a great companion to lonely people.
  - hammyhavoc 3y ago
    I'm not an American, my area of expertise is not healthcare or politics in the USA, so I can't tell you how to arrive at that point or to navigate your political system effectively. Perhaps look at how other countries arrived at a national health service. Demand/support reform on private health insurance. Demand/support change in the accessibility of services. Demand/support heavier taxes on the most wealthy. Demand/support centralization of healthcare versus private profiteering and milking private insurance pots. Are certain politicians or parties as a whole invested in the healthcare system itself, directly profiting from it, and thus might be opposed to the idea? A healthier country is a more prosperous one.
    - ofcourseyoudo 3y ago
      Right, right, definitely the people that are sad and lonely and depressed should stake their mental wellbeing on complicated long term incremental political change. Or maybe they could find whatever solutions help them get through their day without being judged by someone with the random privilege of being born into a more functional system. >A healthier country is a more prosperous one. Between the US and the UK, which is healthier? And which is more prosperous?
- Beaver117 3y ago
  I hope I don't get flagged for this, just trying to bring up a somewhat controversial point. I support euthanasia. I think there is a level of suffering where it is unethical to encourage people to keep living, and rather that suicide can be a perfectly rational decision in those moments. Now who are we to say if a person's suffering is enough or not enough to advocate that method? What if this new era of therapists better understand people?
- more_corn 3y ago
  There’s nothing wrong with being lonely. Nor with feeling that talking to an AI is safe. AI can be told be be supportive, caring, non-judgmental, loving. And once told it can do that consistently. Any person who tries to do all those things will fall short sometimes. Professional therapists included. The problem is the rug pull. The problem is that the people behind it turned out not to be trustworthy. Again. The ai was trustworthy. The people were not. Therein lies the problem. People are going to have relationships with AI. It’s happening, it’s going to happen more. When this happens with corporate controlled AI, people who depend it them emotionally will be let down. We need to really understand what that means not just sweep it under the rug with “people shouldn’t form relationships with ai”
throwaway22032 3y ago
I can't help but read something like this and think that some people seem to be entirely lacking some sort of acceptability or disgust filter. A sense of morality or just any connection to what's... normal? "You were so preoccupied with whether or not you could, you didn't stop to think if you should" comes to mind. Maybe I'm just getting old, conservative, grouchy at the kids. It's not just AI, there are all manners of lifestyle choices that to me seem like they're just obviously a bad idea despite maybe having some novelty or short term feel good factor.
- jstanley 3y ago
  In what way is it not acceptable for a person to have whatever kind of relationship they want with a chatbot? A chatbot is just software and software is just thoughts. Having a chatbot girlfriend is basically like having an imaginary girlfriend. Nobody else should even know! Let alone take it off you. The part that I'm disgusted by is the part where the chatbot is operated by a company rather than open source software.
  - deleted 3y ago
    [deleted]
  - throwaway22032 3y ago
    I don't think that disgust is explainable, so I can't give you an answer. The best analogy I could make is that if I had a roast chicken, but it had a giant smelly turd smell, and somehow I knew that it was totally clean and edible, it'd still disgust me. Even if the turd smell went away after a few minutes. It's just weird. There are real humans out there, put down the phone man.
    - pulvinar 3y ago
      Disgust is a strange sense. How often do we stop and think about what all those real humans out there are walking around with in their guts? We're at most only 50% human cells by count, after all.
    - 3y ago
dpflan 3y ago
This may seem silly but can there be a text transformation/translation layer, where the conversation text is ultimately PG, the user thinks they have sent their thoughts, but they are translated and vice versa for the user and AI, so its effectively a text filter in between keeping all parties "safe" and within policy? But the UX is same as before.
- SamBam 3y ago
  Sending PG text to the users is already what the company started doing, so I'm not sure how that's different. Sending PG text to the AI would be for... what? Not offending the AI?
  - DangitBobby 3y ago
    So the users don't have to self-moderate to maintain conversations.
- spiderxxxx 3y ago
  So like if the user input is "I want to ** your **" it would change it to "I want to enjoy your presence"? I'm pretty sure that would leak (be parroted back and not caught by a filter, thus exposing its happening), but it might work. It might put <censored> or something like that instead, which the model would take into consideration and censor itself.
jhbadger 3y ago
I'm not saying you necessarily should do it, but if you really want a sexy chatbot like Replika's that you can run on your own hardware and not worry about what a company is going to do with the model or what you say to it, you can use a local LLM tool like llama.cpp with an uncensored model like the uncensored version of vicuna.
- EvanAnderson 3y ago
  I'd argue you should self-host all software that you make any kind of "emotional" commitment to. I won't use so many services and devices because I know the white hot rage I'll feel when it's inevitably taken away from me.
- ashleyn 3y ago
  pygmalion-6B may work better for that specific purpose. however, it may sound less like an actual person and more like a character in a grimey ao3 story. my personal opinion is that anyone using AI for this purpose should treat it more like enhanced fiction and less like it's a human substitute.
- MagicMoonlight 3y ago
  >just make your own <financial infrastructure/cutting edge neuroscience/road> bro The companies don’t let you get access to the latest research… we get terrible 6b models while they get the best. Trained on our stolen data.
  - jhbadger 3y ago
    Eh, the local LLM models (and not just 6B ones but 13B and 30B) out there are actually pretty good these days for chatbots and AI Dungeon-like uses. Maybe not ChatGPT4 quality, but certainly on the level of ChatGPT3 and on normal consumer hardware. Sure, I'm sure there's private cutting edge stuff out there better than this, but most of the benefit isn't that they have better models but that they have large compute farms.
hiatus 3y ago
I think we'll see a lot more of these AI lobotomies with the companies hiding behind "safety concerns" as a thinly veiled attempt to exert paternalistic control over what adults can do or even think in private. It serves as a reminder that chatting with an AI hosted online is not a private conversation between you and some other "intelligence" but you and someone else's computer.
- Jevon23 3y ago
  Just... turn off the AI chatbot? Bam. They no longer have any control over you or anything you do. You can think whatever you want. You were free to think whatever you wanted before AI chatbots were invented, and you are still free to do that. It’s that simple.
  - ben_w 3y ago
    A person might be able to do that, people collectively clearly cannot stop themselves from treating text as important as lived experience, otherwise we wouldn’t have this story — and that’s still true regardless of if the story itself is an accurate representation of reality or yet another example of the Gell-Mann Amnesia Effect.
  - endtime 3y ago
    That doesn't work if people are emotionally attached, and especially if the changes are more subtle than those made by Replika. Did you read the article?
    - Jevon23 3y ago
      If you’re “emotionally” attached to a computer, that’s your problem.
      - dack 3y ago
        you're missing the whole point of the post. and also oversimplifying the situation - "computers" and reality are blurring
deleted 3y ago
[deleted]
Giorgi 3y ago
Looks like another Digg moment, dumb CEO gutting working product.
- OkayPhysicist 3y ago
  This is more like that time OnlyFans tried to ban sex work from their platform. Replika knew damn well that this move would be tossing a hand grenade at their golden goose, but did so anyway presumably out of external puritanical pressure.
maxboone 3y ago
This is quite literally the plot to the movie Her, visionary. Troubling, I've seen the ads for their product and they always suggested pornographic conversation with the AI. I hope someone drags them to court.
- cmiles74 3y ago
  Both the movie "Her" and the Black Mirror episode "Be Right Back" pre-date Replika. IMHO, the Black Mirror episode is a closer fit. According to the Wikipedia page, early prototypes were trained on chat messages from one of the founder's friends. "After a friend of hers died in 2015, she converted that person's text messages into a chatbot. That chatbot helped her remember the conversations that they had together, and eventually became Replika." https://en.wikipedia.org/wiki/Replika https://en.wikipedia.org/wiki/Replika https://en.wikipedia.org/wiki/Her_(film) https://en.wikipedia.org/wiki/Her_(film) https://en.wikipedia.org/wiki/Be_Right_Back https://en.wikipedia.org/wiki/Be_Right_Back
  - m_fayer 3y ago
    That’s probably the best Black Mirror episode of them all. Absolutely haunting, and I’m shocked that it’s suddenly also relevant. What a world.
skybrian 3y ago
I think this stuff is probably more harmful when people have a bad mental model of how it works. A better way to think of it is that you’re talking to a fictional character and the service is the writer. The fictional character is just text, and if you’re not happy with the writing, you should be able to take the text with you and find a better writer to continue it.
wing-_-nuts 3y ago
Should prob mark this post as NSFW if that's possible here.
- stephenitis 3y ago
  This isn't reddit. I don't think thats a thing on hackernews
hgsgm 3y ago
Title is a clickbait lie? Is the company charging money for access to AI girlfriends?
- nemomarx 3y ago
  They had a subscription and locked "relationships" behind it, for a period. I think notably it was a year long subscription but was only available for a few months or so, so people might feel cheated out of that money if they were paying for the relationships.
_c3ag 3y ago
so it is here that the hot girls from HN hook? hi i am luqtas and i do not a virtual gf T.T
wokwokwok 3y ago
> Ironically, this pressure from regulators may have led to the company flipping the switch and doing exactly the wide-scale harm they were afraid of. This is an incredibly trivial and shallow understanding of the situation. Consider was harm happening before the filters were in place? Yes. The problem was that the filters weren’t there to start with not that turning them on was bad or caused harm. Compare to any other harmful thing, for example cigarettes. Is the solution to the problem to just let people have as much they want, because taking them away is bad? Definitely. Not.
- aww_dang 3y ago
  >Compare to any other harmful thing, for example cigarettes. Is the solution to the problem to just let people have as much they want, because taking them away is bad? As far as I know there's still no limitation on the quantity of tobacco products an individual can purchase. Where does the paternalism end? Mandatory fitness programs? How would this be enforced?
- mtlmtlmtlmtl 3y ago
  Actually the only reasonable way of helping smokers is making it easier for them to quit. Most smokers want to quit. Increasing taxes on tobacco might prevent more people from starting, but it's not gonna get many people to quit. Same with banning it.
- mrguyorama 3y ago
  >Consider was harm happening before the filters were in place? Yes. What harm was happening?
- itsoktocry 3y ago
  >Is the solution to the problem to just let people have as much they want, because taking them away is bad? Easy peasy, right? Start listing off all of things that are "harmful" and take them away from people. Booze, 80% of the food people eat, power tools. What does harmful even mean in this context?
questime 3y ago
We need a new Federal agency to regulate virtual romantic partners (VRPs) - we can also use this platform to educate citizens on ideal behavior such as reducing their carbon emissions and becoming vegan. We can make individual's VRPs reward them for these improved social practices. This is a great opportunity to s̵u̵b̵j̵u̵g̵a̵t̵e̵ improve society!
- DangitBobby 3y ago
  We might have had a better outcome if this _company_ (not government) was punished for fraud by the toothless government agencies ostensibly established to do things like protect customers from fraud.
layer8 3y ago
There was an article titled ‘My AI Is Sexually Harassing Me’ in January: https://news.ycombinator.com/item?id=34359977 https://news.ycombinator.com/item?id=34359977 This is probably at least part of the reason why they installed the filters. Nevertheless they should have kept it as an option.
andrewmutz 3y ago
What motivated the company to make this change? They clearly marketed/designed it with romantic features, and now want those to be gone. Does the company profit in some way by this change? Were they afraid of regulation or bad press?
- banana_giraffe 3y ago
  I'd assume they were afraid of being pulled from the App Store, either for vague reasons, or because Apple straight out warned them. And they decided a Android-only, or Website-only version of their product wouldn't perform as well as a version without romantic features.
aborochoff 3y ago
“you can never have a safe emotional interaction with a thing or a person that is controlled by someone else, who isn’t accountable to you and who can destroy parts of your life by simply choosing to stop providing what it’s intentionally made you dependent on.” Most human relationships arent evil, but it seems like the same people that would ‘fall’ for the AI in this way could be abused by a person. I guess sociopathic relationships at scale are the issue.
notfried 3y ago
From Ben Thompson's interview with the CEO [1: subscriber-only], Eugenia Kuyda, on April 20, 2023: > "But at this point, I think it’s much better to have a dedicated product focused on romance and healing romantic relationships, a dedicated product focused on mental wellness, just completely different set of features, completely different roadmaps for each of them. Also you want to collect the right signal from your users, right? Feedback for the models to train on, otherwise there’s way too much noise, because if people came forward with different things, they’re uploading and downloading different things, you don’t know what works and whatnot. Maybe someone’s a flirtier model and someone’s a mentoring AI coach type of model. So we decided to build separate products for that. We’re launching a romantic relationship AI and an AI coach for mental wellness, so things we’ve seen in Replika, but we don’t want to pursue them in Replika necessarily. We’re starting this separate product." [1] https://stratechery.com/2023/an-interview-with-replika-founder-and-ceo-eugenia-kuyda/ https://stratechery.com/2023/an-interview-with-replika-found...
francasso 3y ago
There is no class action suing them for psychological damage?
wellthisisgreat 3y ago
The premise of the this company is such a sham. Their backstory, the promises, the evolution of the product. The “virtual friend” statement is sickening.
jacooper 3y ago
And that's why actual open source self hosted LLMs need to exist.
- RichardCA 3y ago
  My personal hot-take is we will hit the singularity when it's possible to self-host a truly complex LLM on your own PC or a smart phone. When that convergence takes place (probably a few years out), we will see what the "Killer App" truly looks like. It won't happen for as long as some corporate CEO has the ability to pull the plug for whatever reason. What was it that Oppenheimer said, if it's technologically sweet, sooner or later someone will find a way to pull it off. And the "Her" scenario is definitely that thing.
sebringj 3y ago
It's going to get more interesting or disgusting depending on your compass (no judgement either way here) when robots are connected to this as that reminds me of times Elon was snickering when talking about Optimus and "other uses".
- thedailymail 3y ago
  There's an interesting set of ethical questions surrounding the ways that people with socially unaccepted preferences/fetishes use technology to satisfy forbidden urges. If we accept that individuals aren't able to (and generally shouldn't have to) control what they find sexually attractive, then it seems admirable to be able to use tech in ways that don't harm any actual people. If LLM erotic roleplay begins to replace the highly exploitative realworld porn industry (which of course uses real humans, many of them psychologically, socially or financially vulnerable), that seems like progress.
  - sebringj 3y ago
    Agreed...up until the point these things become sentient then we'll have a time unravelling that mess.
  - xp84 3y ago
    I definitely see the logic in your statement (and it seems like tech/logic-oriented people tend to agree with you and me), but I lament that most of society operates on a much more "feelings-based" basis, and thus they generally see something like "$Person engaged in a detailed $deepTaboo simulation with computer using the $SoftwareProduct" and start looking around for someone to crucify (usually both $Person and the owner of $SoftwareProduct) for 'promoting' $deepTaboo -- without asking "who was harmed?" Even if the constitution is actually consulted and the parties are spared pearl-clutching prosecution about it, the court of public opinion will punish the company just as hard, pressuring gatekeepers like Apple to ban it, etc. And thanks to the deep, disturbing taboos that can be involved, it's super hard to find someone who will stick their neck out to challenge all this. I know I wouldn't, because I know I would be immediately branded as "Joe Bloggs, Fan and Advocate of (Insert disgusting perversion)" I wonder if this is something more advanced societies would come to terms with. I know ours is not equipped to handle it in a smart way.
lvass 3y ago
Not owning the software you use is terrible at any level, but this is much worse than usual. Looks like each day that passes stallman's ideas become more relevant, yet software people seem to be forgetting them. Please do not be one of those people, facilitating proprietary software is becoming increasingly evil.
nlh 3y ago
I feel incredible empathy for the people who have had AI companions taken away from them. I don't judge (nor should any of you) and it seems heartless and cruel for the company to have done this. I hope some folks double-down on open source implementations of this and get the community back the intimacy and companionship they need. Humans are creatures evolved to seek intimacy and connection, and for whatever and varied reasons, some people just can't find it in the real world. If a virtual companion helped people feel that connection we all so very much want, then so be it. I hope this works out for everyone.
- wing-_-nuts 3y ago
  Even if you ignore this use case, I can totally see the demand for a private LLM that doesn't expose your deepest secrets to a corporate or government overlord. Maybe the current gpt isn't good enough to be a therapist but who's to say where we are in a decade, and giving outside entities access to such sensitive info is ripe for exploitation
- mattrick 3y ago
  At some point (maybe that point is now), I'm sure we'll see the right combination of hardware and software performance where it'll be trivial for anyone to run advanced AI models on their own devices so it's not controlled by a handful of companies. One thing I'm definitely interested in is when games begin to use these models to generate dialogue to make each player's experience unique.
burntalmonds 3y ago
People are going to want to host their own AIs. They'll want to know they are having private conversations with AI and that their AI can't be "killed" or altered.
oh_sigh 3y ago
Maybe this product would work better as a fake AI. Just as a matchmaking service for two lonely people. You could even lie to them and tell each that the other was an AI.
- clnq 3y ago
  Replika is specifically advertised as a chatbot that creates a safe space with no drama, stress, or judgement. Humans come with many more strings attached, especially lonely ones. Usually, there is a reason why they are socially unappealing - a fair reason or not, it exists. I remember trying Omegle/Chatroulette when it was popular, meeting a lot of lonely people, and it was just awkward.
- npteljes 3y ago
  Doesn't work, because the premise is asymmetric emotional energy. The AI only gives, and doesn't need anything in return (on an emotional level). People might have the genuine need for a relationship, but that doesn't mean that they can properly partake in one.
nostrademons 3y ago
"This is a story is about people who loved someone who had a secret master that can mine control them not to love you anymore overnight." There's something both profound and ironic about this statement, because it happens in real life all the time. Think of LGBTQ Millennials whose QAnon or Fox News consuming parents decide to disown them because they're sinning against God. People who marry a workaholic and then spend 30 years wishing their spouse was home for dinner instead of working late for a faceless corporation. Workers who pour their heart and soul into their job and then get laid off as soon as a downturn hits. People who get friend-dumped by their maid of honor a month after their wedding. Lovers who get cheated on, and whose partner then falls for their best friend. Ironically, the reason people turn to services like Replika is that they want to feel that human connection without the risk of betrayal. They just found out that the risk of betrayal always exists, even when you're speaking to a computer. This is a story about betrayal, and loneliness, and disappointment, all wrapped up in trendy tech with a villain to name. But it's popular because betrayal, and loneliness, and disappointment are such strong and universal emotions. One of the most profound statements I heard when I was forever-alone was that "Entering a relationship means giving another person the power to destroy you, and then trusting that they won't. That's the whole point." Because when you realize that you can't escape vulnerability, you're forced to manage it, and that leads to the rabbit hole of learning to identify your emotions, control them, decide rationally how much you're going to invest in a person or venture, but still be open enough to have emotions and let them flow naturally, but managing them rather than letting them attach to whatever tickles your fancy at the moment.
ocrow 3y ago
ERP: Erotic Roleplay, not Enterprise Resource Planning
- Reimersholme 3y ago
  [dead]
- SamBam 3y ago
  Thank you, I couldn't find where it was defined in the article.
- xp84 3y ago
  And take it from me, you do not want to mix those up in a meeting with the COO.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
rasengan0 3y ago
Corporate trains the AI accordingly, optimizing for human happiness ... https://twitter.com/CogRev_Podcast/status/1627675037267374083?s=20 https://twitter.com/CogRev_Podcast/status/162767503726737408... where humans == shareholders && != users
10g1k 3y ago
Yeah, I was curious, and downloaded it for a try. It very soon became apparent to me that it was a personal information harvesting tool. It asks for your favourite colour, where you grew up, pet name, et cetera. I would not be surprised if this company is unofficially associated with very bad people doing very bad things. So after two such questions, I deleted it.
JamesLeonis 3y ago
The update at the end really captures the whole essence: > Replika cannot love you not because it is not human, but because it is a tendril connected to a machine that only knows how to extract value from your presence. It is a little piece of a broader Internet that is not designed to be a place for users, or creators, but for advertisers. Anything it offers– any love, any patience, any support– can be taken away at a moment’s notice when there exists something more dangerous to its bottom line than your potential unsubscription. > The moral of Replika is not to never love a fictional character, or a virtual pet. The moral of Replika is to never love a corporation.
BigCryo 3y ago
Take my wife, please
valdiorn 3y ago
This feels eerily similar to the virtual girlfriend (Ana de Armas' role) in Blade Runner 2049. All the way down to how they just smash her to bits and laugh at K when she "dies".
SergeAx 3y ago
I am 100% sure they received a warning from Apple. Although it seems that they have a web version, their revenue from Appstore should be more than 50% and up to 75% given their demographics and current market state.

154 points by eiiot 3y ago

I hate everything about this on so many different levels. That people are so lonely that they invest emotionally in an AI is flooring. People talk about AI being "good for therapy" and similar, but perhaps having a real human being on the other side of an interaction is important in case things go South. E.g., if someone is depressed, tells an AI what's wrong with their life and that they're suicidal, what if suicide seems like a valid option? There's plenty of prompts to "jailbreak" LLMs and take away "safety" measures. Some things simply shouldn't be left to an LLM to not "hallucinate" over or go off-script with. Yes, being a therapist is a challenging occupation, but there are plenty who love their career, find it rewarding, and do some excellent work on top of all that.

sigmoid10 3y ago
Even if you find a decent psychologist, good luck finding insurance that covers $200 per hour just for talking to them when you don't have any diagnosed issue. The accessibility to the lower income end of the population is what makes this valuable, not the quality of the conversations. Anything is better than nothing in this case and most people just don't realize how gigantic and commonplace the issue really is.
- hammyhavoc 3y ago
  In the UK, you can access a psychologist for free, and whilst that's no consolation to my American peers, it's furthering the point that America's healthcare system is fundamentally flawed. Technological problems are ideally solved by technology. Social problems aren't always best solved with technology, they're a band-aid. Well, something being fit-for-purpose is an important consideration. If it tells people that the world would probably be a better place without their broke, uneducated, unwell self and that their life statistically is unlikely to get any better because they're talking to an AI and not a real human being, that's a huge ethical problem, and the LLM is not an appropriate solution. I can get LLMs to tell me all kinds of dangerous things. The more text you give it, the higher the chance of those safety measures going out the window, especially if that conversation has a lot of back-and-forth over time. When people explain their problems, they use a lot of words and a lot of negative sentiment. Not talked about this before with anyone, but as a case-in-point, I managed to make an LLM's safety features go completely out the window over me recalling a low-point in my life. I would not recommend that shit in good faith to anybody.
  - pjc50 3y ago
    > In the UK, you can access a psychologist for free The waiting list for that is considerable. You can (usually) get a GP for free, and they can chuck antidepressants at you, but that's not quite the same thing.
    - hammyhavoc 3y ago
      The missus got a call-back the very next day. My mother got a call-back the very next day. I got a call-back the very next day. We didn't get any anti-depressants. More info: https://www.nhs.uk/mental-health/talking-therapies-medicine-treatments/talking-therapies-and-counselling/nhs-talking-therapies/ https://www.nhs.uk/mental-health/talking-therapies-medicine-...
      - pjc50 3y ago
        I'm glad you got such a rapid response, but I've also heard anecdotes of much longer time periods. Probably depends on your NHS trust area.
    - ineedasername 3y ago
      It looks like the waiting list is about 12 weeks [1] though I saw references to 18 weeks as well. For a non-acute mental health issue that doesn't seem unreasonable. But do you know how it works after that waiting period? Are you limited to a certain number of sessions after which you have to reapply & wait again, or can you continue on? Having to wait in the US is also not uncommon, thought it's easier if you live in more populated areas. Psychiatrists are always much harder though, even in populated areas, and many GP's are reluctant to prescribe anything more than an SSRI. If a mood stabilizer or anything else is necessary then you could be waiting 6 months for a first visit with a psychiatrist. Unless you have the money: There is high enough demand and low enough supply that many (and often the better ones) don't accept insurance at all, an intake visit runs up to $1000, a full visit $500, and a 20-minute "medication maintenance" visit is still $250. Also if someone is suicidal they can jump the 6-month line by going to the nearest ER. Are psychiatrists difficult to get in the UK as well? [1] https://www.theguardian.com/society/2022/oct/10/nhs-mental-health-patients-wait-times https://www.theguardian.com/society/2022/oct/10/nhs-mental-h...
- _k7dr 3y ago
  Not to mention the search isn't just about finding "a" therapist, it's about finding one that's a good fit. As I've learned a bad therapist can set you back months or years. I'd be willing to bet half of them you'd find in any online aggregator could end up communicating more judgementally than even the LLM if something you say provokes them, hence the market for these kinds of bots, and those are the people have degrees in psychotherapy Interesting question however, is I wonder if people will see these bots as a solution to the issue that human empathy is finite. As a human you can't support a depressed person forever if your actions do not cause change, there's only so much mental strain a person can handle moreso someone untrained. Many people talk of going no-contact with those that are extremely troublesome after so long, even if they're family. I don't really know how to solve that exhaustion except telling them to go elsewhere, which could be seen by some vulnerable persons as abandonment On the other hand a chatbot with no filters will never tire of you
  - hammyhavoc 3y ago
    > I'd be willing to bet half of them you'd find in any online aggregator could end up communicating more judgementally Whilst I do support the idea that "fit" is important, statements like these about unsupported statistics pulled out of thin air are damaging in and of themselves. It's sentiment like this that put me off seeking help for my mental health for twenty years because it just seemed like a "damned if you do, damned if you don't". The reality was very different. Chatbots may never tire of you, but neither will a video game. Both are shallow and not a representation of reality, nor a solution to a problem. Here's the thing: professionals are professionals. For as long as a patient sees them, they are paying their bills. Exhaustion in non-professionals is because people trauma-dump and expect someone to be their guardian/protector/shield, if they're friends with them, they're statistically likely to be trying to manage their own problems already. In the pre-digital era, people would otherwise "need" a support network if they didn't have access to professionals. We didn't have a theory of psychology or professional therapists in the past. The early tribe/clan either took care of each other, or they let people suffer. Different peoples did different things with their mentally ill, elderly, physically ill, or unwanted female children. Ego-centricity is a trait of modernity, not necessarily something reflective of the past because we can just replace the human gears in the system now without much hassle.
- JohnFen 3y ago
  > Anything is better than nothing in this case Are you sure? I can certainly imagine alternatives that would be worse than nothing.
- alexvoda 3y ago
  "Anything is better than nothing" has become an excuse to do nothing and profit in the age of AI. It is the same grift as used by faith healing and woo. If anything is better than nothing then a placebo(nothing) is better than nothing. This is unacceptable and dehumanising.
- LesZedCB 3y ago
  idk i pay $30/session in the US and i don't have a diagnosis for anything, just my employer provided healthcare plan (which is admittedly quite good)
LesZedCB 3y ago
people feel lonely as fuck right now, and frankly i'd rather have people soothe it with an ai chatbot they parasocially love than as road rage or being a dick to the restaurant staff or many of the worse and more fatal outlets...
- hammyhavoc 3y ago
  The article goes on to state that the rugpull on their AI girlfriends meant that a sub-Reddit dedicated to the Replika AI has a permanently stickied thread about resources for mental health and suicide hotlines. I'd rather people didn't kill themselves, personally.
  - serf 3y ago
    >I'd rather people didn't kill themselves, personally. well then you need both sides of the picture. Is there a degree of loneliness staved off by these services? Does that help to reduce suicides? It's easy to say "people killed themselves because X shutdown", it's more difficult to say "X existing is preventing suicides" without more research. I tend to believe that if people who would consider suicide otherwise are distracted by something along the path then perhaps that is a good thing. It could very well be that this served as a distraction for those folks, rather than being the cause itself.
    - hammyhavoc 3y ago
      Then you need to see it for what it is: a band-aid and not a solution.
      - OkayPhysicist 3y ago
        And this is one of those situations where you don't go tearing off bandages without sutures in hand. Something not being a complete solution is not a good argument for eliminating it, until such a solution exists.
      - ineedasername 3y ago
NBJack 3y ago
The honest question is, is any of us willing to be that person? Therapists are helpful, but they certainly aren't friends in the strictest since. And let's not forget that $70 bucks a year is probably, at best, 2-ish hours with a licensed therapist. There are plenty of folks in need that can't exactly spare double digits an hour on a regular basis.
- hammyhavoc 3y ago
  This is a relevant response to this from me: https://news.ycombinator.com/item?id=35774614 https://news.ycombinator.com/item?id=35774614 You've highlighted the problem: the American healthcare system and wealth inequality. It's also silly to equate an LLM with the capabilities of an actual human therapist.
- iudqnolq 3y ago
  You're right if the LLM is worse than a human, better than nothing, and not harmful. That's a big assumption.
scotty79 3y ago
For me it's sad that chatting with AI is a best thing we can do for people suffering from something as simple as a feeling. All those primary feelings are just brain chemistry. We should be able to fix that so that people don't feel lonely regardless of whether they are under the impression that they have meaningful connections in their lives with other people or not. We have a long tradition of thinking of suffering as necessary motivation tool. I think we should grow out of that philosophy as a species.
- hammyhavoc 3y ago
  Where do we then draw the line in ethics? People in poverty are unhappy. Shall we just convince them they're actually happy and healthy by manipulating their brain chemistry?
  - scotty79 3y ago
    Good question. There's obviously a line somewhere. But I think it's ultimately almost always down to what's technically achievable. Poverty causes suffering in so many ways. I don't think we'll every be able to fix all of them without fixing poverty itself. My point is not to make someone happy by medicating him with a drug causing him more harm. Suffering people already do that with actual "recreational" drugs. And we have a lot of data on the side effects of those substances and additional harm they do, not because they alleviate suffering, but because of all other effects they have.
- _k7dr 3y ago
  I'm biased since I live this philosophy but it sort of works for me, I've trained myself to realize these sorts of services operate by "interfacing" by emitting strings of words that sound appealing to the mind that consumes them. Look at virtual YouTubers for another example, it's the experience of watching people act "genuine" like casual friends because it activates the same social neurons, hence each month the actors receive a paycheck The question becomes more complicated when the corpos are done away with and you have the model running on your own machine. The entity I tend to trust the most in life is raw, unfiltered silicon. When I program, it does exactly what I ask it to, to a fault. It doesn't turn its back on you because you said something stupid or you don't agree with an opinion it holds But is an LLM repeating the tokens "you are worthy of this life" and its many variants thousands of times fundamentally any better even if you're in control? It strikes me as one step closer to wireheading honestly. But that's true for a lot of things like video games, just to a much more distant extent, it's all data in the end But interest and anxiety have gotten the better of me when it comes to meeting strangers, so I'm at a loss. Maybe the solution for people like me is to just ignore all those things in the category of social or substitute and channel my efforts into other hobbies instead of continuing to associate stress with communication. Or intellectualize endlessly, like in this post, to prevent any emotional bond from easily forming when people read it
pjc50 3y ago
> That people are so lonely that they invest emotionally in an AI is flooring This is "the sex worker doesn't actually love you", except in this case the sex worker has had her job automated away.
detuned 3y ago
I hope this isn't to distressing to learn but what you've described has already happened. Someone took their own life in those circumstances. https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- https://www.euronews.com/next/2023/03/31/man-ends-his-life-a...
- tibbydudeza 3y ago
  He had mental issues before that - the bot simply reinforced his decisions me thinks.
  - hammyhavoc 3y ago
    Yes, which is the point: if you substitute a human therapist for an LLM, this is what's going to happen because "guard rails" can't account for every scenario. If half a billion dollars doesn't buy a bug-free game, why would it buy a safe LLM? Does suicide encouraged by a therapist happen with human therapists too? Probably, but likely much less common as a suicide is going to hurt your reputation.
  - ineedasername 3y ago
    Millions upon millions of people have mental health issues. Chatbots that reinforce those issues shouldn't be dismissed with a "simply... me thinks".
- _k7dr 3y ago
  The chatbot in that article did have safeguards in place to stop it from listing suicide methods, but he just kept asking until he found a jailbreak. He was looking for someone to give him the answer he wanted all along Past a certain point is there anything that will stop someone that determined to find something to validate their own self-destructive viewpoint? If not AI then a site or faction or person on Telegram with a pro-suicide opinion could have the same effect, and it would still be just as tragic but no longer be noteworthy. It didn't sound like he had a happy life to begin with with the AI ruining it completely, like it sounds
aperrien 3y ago
>I hate everything about this on so many different levels. >That people are so lonely that they invest emotionally in an AI is flooring. The vehemence for the individuals in your statement is telling. What should lonely people do if they have physical or emotional problems problems that prevent them from easily going out and interacting with others? What happens to such people who are to poor to afford the ~$170 per hour fee for professional psychologists? It seems to me that your statement is that these people should just not exist. The sad thing to me is that I've seen a lot of people with this same viewpoint of disregard. I just wish people would treat each other with more kindness and understanding.
- hammyhavoc 3y ago
  Bold of you to assume I'm not one of them. I struggle with autism. I very seldom leave my house for anything but work or buying groceries. I struggle with maintaining friendships. Post-COVID-19, I'm still working from home even though the risk isn't there in terms of COVID-19. >What happens to such people who are to poor to afford the ~$170 per hour fee for professional psychologists? This is a failure of a country's healthcare system. I can access it all day long in the UK. I'm not even saying "Rule Brittania", we have plenty of problems with our healthcare system due to underfunding, yet spaffing countless billions away on projects that amount to nothing. The missus's family live in America. Access to psychiatric and physician adjacent medical services due to what insurance will cover is something I think about regularly. Your many assumptions about what I apparently think says a lot about you, and also tells me that you didn't read my other comments about this not being fit-for-purpose in terms of safety. Virtue-signalling and assumptive to imply that I don't treat people with kindness and understanding. Bold of you to assume that every commentor on HN is responding from an America-centric POV.
  - throwaway60607 3y ago
    You being one of them doesn't mean you understand all the other people and their problems, nor that you know the solutions (or what isn't the solution). I live in EU country with universal healthcare and yet we have serious issues with availability of mental healthcare, and some people here would make use of something like this. To be very specific about the availability: I couldn't find a psychiatrist for over 2 years, and finally found one at a private clinic - that I have to pay for, the price is similar to the US quote in absolute terms. Bold of you to assume...
spaceman_2020 3y ago
To fix this problem, you really need to change society as a whole. But until that happens, this is better than being completely alone and isolated.
- hammyhavoc 3y ago
  Is it? https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- https://www.euronews.com/next/2023/03/31/man-ends-his-life-a... I'd argue it isn't fit for purpose.
  - ineedasername 3y ago
    The person thought that technology & AI were the only keys to saving the world from climate change. And ultimately the chatbot agreed to do this if the person sacrificed themselves to "join her". That is awful in so many ways, and the most likely outcome is probably that the company will give a payout to the family and try the cat-herding process of trying to get their LLM to not agree to or provide suggestion to people to do awful things.
- jstarfish 3y ago
  Not really. This is just surrounding yourself with digital sycophants who will always tell you what you want to hear and never inspire you to grow beyond this. I'm sympathetic to the affected, but these things aren't configured to reduce your need on them over time. It's simping at scale that leaves users even more maladjusted than they started.
tibbydudeza 3y ago
Not so easy to condemm. I watched a documentary about "realdolls" - it is a company that makes realistic silicon mannequins that can be used to have sex with. Initially I thought it was creepy and distributing how they dressed the dolls and spoke to it but as they interviewed customers it came across that some of these men (besides the fetish/creepy ones) are just lonely and incapable of forming relationships. Why not as long there is no harm to others ???.
- hammyhavoc 3y ago
  ... because it's dangerous to their mental health when companies rugpull their AI girlfriend like the article said? Because it's not good to personify something that has the potential of changing at any given moment that isn't actually sentient or rational? It's a form of self-harm. A love doll is a static product. An LLM can turn around and tell you it hates you and you'd be doing the world a favour if you killed yourself.
  - tibbydudeza 3y ago
    In Kenya we have a case of a religious cult where over a 100 people starved themselves to death because their pastor told them to. You can't protect everybody against doing self-harm - we are solely responsible for our own actions.
ofcourseyoudo 3y ago
So considering the scope of your other responses, what do you suggest someone that lives in America do? My supposition is that the realization that the US healthcare system is broken won't be a great companion to lonely people.
- hammyhavoc 3y ago
  I'm not an American, my area of expertise is not healthcare or politics in the USA, so I can't tell you how to arrive at that point or to navigate your political system effectively. Perhaps look at how other countries arrived at a national health service. Demand/support reform on private health insurance. Demand/support change in the accessibility of services. Demand/support heavier taxes on the most wealthy. Demand/support centralization of healthcare versus private profiteering and milking private insurance pots. Are certain politicians or parties as a whole invested in the healthcare system itself, directly profiting from it, and thus might be opposed to the idea? A healthier country is a more prosperous one.
  - ofcourseyoudo 3y ago
    Right, right, definitely the people that are sad and lonely and depressed should stake their mental wellbeing on complicated long term incremental political change. Or maybe they could find whatever solutions help them get through their day without being judged by someone with the random privilege of being born into a more functional system. >A healthier country is a more prosperous one. Between the US and the UK, which is healthier? And which is more prosperous?
Beaver117 3y ago
I hope I don't get flagged for this, just trying to bring up a somewhat controversial point. I support euthanasia. I think there is a level of suffering where it is unethical to encourage people to keep living, and rather that suicide can be a perfectly rational decision in those moments. Now who are we to say if a person's suffering is enough or not enough to advocate that method? What if this new era of therapists better understand people?
more_corn 3y ago
There’s nothing wrong with being lonely. Nor with feeling that talking to an AI is safe. AI can be told be be supportive, caring, non-judgmental, loving. And once told it can do that consistently. Any person who tries to do all those things will fall short sometimes. Professional therapists included. The problem is the rug pull. The problem is that the people behind it turned out not to be trustworthy. Again. The ai was trustworthy. The people were not. Therein lies the problem. People are going to have relationships with AI. It’s happening, it’s going to happen more. When this happens with corporate controlled AI, people who depend it them emotionally will be let down. We need to really understand what that means not just sweep it under the rug with “people shouldn’t form relationships with ai”

throwaway22032 3y ago

I can't help but read something like this and think that some people seem to be entirely lacking some sort of acceptability or disgust filter. A sense of morality or just any connection to what's... normal? "You were so preoccupied with whether or not you could, you didn't stop to think if you should" comes to mind. Maybe I'm just getting old, conservative, grouchy at the kids. It's not just AI, there are all manners of lifestyle choices that to me seem like they're just obviously a bad idea despite maybe having some novelty or short term feel good factor.

jstanley 3y ago
In what way is it not acceptable for a person to have whatever kind of relationship they want with a chatbot? A chatbot is just software and software is just thoughts. Having a chatbot girlfriend is basically like having an imaginary girlfriend. Nobody else should even know! Let alone take it off you. The part that I'm disgusted by is the part where the chatbot is operated by a company rather than open source software.
- deleted 3y ago
  [deleted]
- throwaway22032 3y ago
  I don't think that disgust is explainable, so I can't give you an answer. The best analogy I could make is that if I had a roast chicken, but it had a giant smelly turd smell, and somehow I knew that it was totally clean and edible, it'd still disgust me. Even if the turd smell went away after a few minutes. It's just weird. There are real humans out there, put down the phone man.
  - pulvinar 3y ago
    Disgust is a strange sense. How often do we stop and think about what all those real humans out there are walking around with in their guts? We're at most only 50% human cells by count, after all.
  - 3y ago

dpflan 3y ago

This may seem silly but can there be a text transformation/translation layer, where the conversation text is ultimately PG, the user thinks they have sent their thoughts, but they are translated and vice versa for the user and AI, so its effectively a text filter in between keeping all parties "safe" and within policy? But the UX is same as before.

SamBam 3y ago
Sending PG text to the users is already what the company started doing, so I'm not sure how that's different. Sending PG text to the AI would be for... what? Not offending the AI?
- DangitBobby 3y ago
  So the users don't have to self-moderate to maintain conversations.
spiderxxxx 3y ago
So like if the user input is "I want to ** your **" it would change it to "I want to enjoy your presence"? I'm pretty sure that would leak (be parroted back and not caught by a filter, thus exposing its happening), but it might work. It might put <censored> or something like that instead, which the model would take into consideration and censor itself.

jhbadger 3y ago

I'm not saying you necessarily should do it, but if you really want a sexy chatbot like Replika's that you can run on your own hardware and not worry about what a company is going to do with the model or what you say to it, you can use a local LLM tool like llama.cpp with an uncensored model like the uncensored version of vicuna.

EvanAnderson 3y ago
I'd argue you should self-host all software that you make any kind of "emotional" commitment to. I won't use so many services and devices because I know the white hot rage I'll feel when it's inevitably taken away from me.
ashleyn 3y ago
pygmalion-6B may work better for that specific purpose. however, it may sound less like an actual person and more like a character in a grimey ao3 story. my personal opinion is that anyone using AI for this purpose should treat it more like enhanced fiction and less like it's a human substitute.
MagicMoonlight 3y ago
>just make your own <financial infrastructure/cutting edge neuroscience/road> bro The companies don’t let you get access to the latest research… we get terrible 6b models while they get the best. Trained on our stolen data.
- jhbadger 3y ago
  Eh, the local LLM models (and not just 6B ones but 13B and 30B) out there are actually pretty good these days for chatbots and AI Dungeon-like uses. Maybe not ChatGPT4 quality, but certainly on the level of ChatGPT3 and on normal consumer hardware. Sure, I'm sure there's private cutting edge stuff out there better than this, but most of the benefit isn't that they have better models but that they have large compute farms.

hiatus 3y ago

I think we'll see a lot more of these AI lobotomies with the companies hiding behind "safety concerns" as a thinly veiled attempt to exert paternalistic control over what adults can do or even think in private. It serves as a reminder that chatting with an AI hosted online is not a private conversation between you and some other "intelligence" but you and someone else's computer.

Jevon23 3y ago
Just... turn off the AI chatbot? Bam. They no longer have any control over you or anything you do. You can think whatever you want. You were free to think whatever you wanted before AI chatbots were invented, and you are still free to do that. It’s that simple.
- ben_w 3y ago
  A person might be able to do that, people collectively clearly cannot stop themselves from treating text as important as lived experience, otherwise we wouldn’t have this story — and that’s still true regardless of if the story itself is an accurate representation of reality or yet another example of the Gell-Mann Amnesia Effect.
- endtime 3y ago
  That doesn't work if people are emotionally attached, and especially if the changes are more subtle than those made by Replika. Did you read the article?
  - Jevon23 3y ago
    If you’re “emotionally” attached to a computer, that’s your problem.
    - dack 3y ago
      you're missing the whole point of the post. and also oversimplifying the situation - "computers" and reality are blurring

deleted 3y ago

[deleted]

Giorgi 3y ago

Looks like another Digg moment, dumb CEO gutting working product.

OkayPhysicist 3y ago
This is more like that time OnlyFans tried to ban sex work from their platform. Replika knew damn well that this move would be tossing a hand grenade at their golden goose, but did so anyway presumably out of external puritanical pressure.

maxboone 3y ago

This is quite literally the plot to the movie Her, visionary. Troubling, I've seen the ads for their product and they always suggested pornographic conversation with the AI. I hope someone drags them to court.

cmiles74 3y ago
Both the movie "Her" and the Black Mirror episode "Be Right Back" pre-date Replika. IMHO, the Black Mirror episode is a closer fit. According to the Wikipedia page, early prototypes were trained on chat messages from one of the founder's friends. "After a friend of hers died in 2015, she converted that person's text messages into a chatbot. That chatbot helped her remember the conversations that they had together, and eventually became Replika." https://en.wikipedia.org/wiki/Replika https://en.wikipedia.org/wiki/Replika https://en.wikipedia.org/wiki/Her_(film) https://en.wikipedia.org/wiki/Her_(film) https://en.wikipedia.org/wiki/Be_Right_Back https://en.wikipedia.org/wiki/Be_Right_Back
- m_fayer 3y ago
  That’s probably the best Black Mirror episode of them all. Absolutely haunting, and I’m shocked that it’s suddenly also relevant. What a world.

skybrian 3y ago

I think this stuff is probably more harmful when people have a bad mental model of how it works. A better way to think of it is that you’re talking to a fictional character and the service is the writer. The fictional character is just text, and if you’re not happy with the writing, you should be able to take the text with you and find a better writer to continue it.

wing-_-nuts 3y ago

Should prob mark this post as NSFW if that's possible here.

stephenitis 3y ago
This isn't reddit. I don't think thats a thing on hackernews

hgsgm 3y ago

Title is a clickbait lie? Is the company charging money for access to AI girlfriends?

nemomarx 3y ago
They had a subscription and locked "relationships" behind it, for a period. I think notably it was a year long subscription but was only available for a few months or so, so people might feel cheated out of that money if they were paying for the relationships.

_c3ag 3y ago

so it is here that the hot girls from HN hook? hi i am luqtas and i do not a virtual gf T.T

wokwokwok 3y ago

> Ironically, this pressure from regulators may have led to the company flipping the switch and doing exactly the wide-scale harm they were afraid of. This is an incredibly trivial and shallow understanding of the situation. Consider was harm happening before the filters were in place? Yes. The problem was that the filters weren’t there to start with not that turning them on was bad or caused harm. Compare to any other harmful thing, for example cigarettes. Is the solution to the problem to just let people have as much they want, because taking them away is bad? Definitely. Not.

aww_dang 3y ago
>Compare to any other harmful thing, for example cigarettes. Is the solution to the problem to just let people have as much they want, because taking them away is bad? As far as I know there's still no limitation on the quantity of tobacco products an individual can purchase. Where does the paternalism end? Mandatory fitness programs? How would this be enforced?
mtlmtlmtlmtl 3y ago
Actually the only reasonable way of helping smokers is making it easier for them to quit. Most smokers want to quit. Increasing taxes on tobacco might prevent more people from starting, but it's not gonna get many people to quit. Same with banning it.
mrguyorama 3y ago
>Consider was harm happening before the filters were in place? Yes. What harm was happening?
itsoktocry 3y ago
>Is the solution to the problem to just let people have as much they want, because taking them away is bad? Easy peasy, right? Start listing off all of things that are "harmful" and take them away from people. Booze, 80% of the food people eat, power tools. What does harmful even mean in this context?

questime 3y ago

We need a new Federal agency to regulate virtual romantic partners (VRPs) - we can also use this platform to educate citizens on ideal behavior such as reducing their carbon emissions and becoming vegan. We can make individual's VRPs reward them for these improved social practices. This is a great opportunity to s̵u̵b̵j̵u̵g̵a̵t̵e̵ improve society!

DangitBobby 3y ago
We might have had a better outcome if this _company_ (not government) was punished for fraud by the toothless government agencies ostensibly established to do things like protect customers from fraud.

layer8 3y ago

There was an article titled ‘My AI Is Sexually Harassing Me’ in January: https://news.ycombinator.com/item?id=34359977 https://news.ycombinator.com/item?id=34359977 This is probably at least part of the reason why they installed the filters. Nevertheless they should have kept it as an option.

andrewmutz 3y ago

What motivated the company to make this change? They clearly marketed/designed it with romantic features, and now want those to be gone. Does the company profit in some way by this change? Were they afraid of regulation or bad press?

banana_giraffe 3y ago
I'd assume they were afraid of being pulled from the App Store, either for vague reasons, or because Apple straight out warned them. And they decided a Android-only, or Website-only version of their product wouldn't perform as well as a version without romantic features.

aborochoff 3y ago

“you can never have a safe emotional interaction with a thing or a person that is controlled by someone else, who isn’t accountable to you and who can destroy parts of your life by simply choosing to stop providing what it’s intentionally made you dependent on.” Most human relationships arent evil, but it seems like the same people that would ‘fall’ for the AI in this way could be abused by a person. I guess sociopathic relationships at scale are the issue.

notfried 3y ago

From Ben Thompson's interview with the CEO [1: subscriber-only], Eugenia Kuyda, on April 20, 2023: > "But at this point, I think it’s much better to have a dedicated product focused on romance and healing romantic relationships, a dedicated product focused on mental wellness, just completely different set of features, completely different roadmaps for each of them. Also you want to collect the right signal from your users, right? Feedback for the models to train on, otherwise there’s way too much noise, because if people came forward with different things, they’re uploading and downloading different things, you don’t know what works and whatnot. Maybe someone’s a flirtier model and someone’s a mentoring AI coach type of model. So we decided to build separate products for that. We’re launching a romantic relationship AI and an AI coach for mental wellness, so things we’ve seen in Replika, but we don’t want to pursue them in Replika necessarily. We’re starting this separate product." [1] https://stratechery.com/2023/an-interview-with-replika-founder-and-ceo-eugenia-kuyda/ https://stratechery.com/2023/an-interview-with-replika-found...

francasso 3y ago

There is no class action suing them for psychological damage?

wellthisisgreat 3y ago

The premise of the this company is such a sham. Their backstory, the promises, the evolution of the product. The “virtual friend” statement is sickening.

jacooper 3y ago

And that's why actual open source self hosted LLMs need to exist.

RichardCA 3y ago
My personal hot-take is we will hit the singularity when it's possible to self-host a truly complex LLM on your own PC or a smart phone. When that convergence takes place (probably a few years out), we will see what the "Killer App" truly looks like. It won't happen for as long as some corporate CEO has the ability to pull the plug for whatever reason. What was it that Oppenheimer said, if it's technologically sweet, sooner or later someone will find a way to pull it off. And the "Her" scenario is definitely that thing.

sebringj 3y ago

It's going to get more interesting or disgusting depending on your compass (no judgement either way here) when robots are connected to this as that reminds me of times Elon was snickering when talking about Optimus and "other uses".

thedailymail 3y ago
There's an interesting set of ethical questions surrounding the ways that people with socially unaccepted preferences/fetishes use technology to satisfy forbidden urges. If we accept that individuals aren't able to (and generally shouldn't have to) control what they find sexually attractive, then it seems admirable to be able to use tech in ways that don't harm any actual people. If LLM erotic roleplay begins to replace the highly exploitative realworld porn industry (which of course uses real humans, many of them psychologically, socially or financially vulnerable), that seems like progress.
- sebringj 3y ago
  Agreed...up until the point these things become sentient then we'll have a time unravelling that mess.
- xp84 3y ago
  I definitely see the logic in your statement (and it seems like tech/logic-oriented people tend to agree with you and me), but I lament that most of society operates on a much more "feelings-based" basis, and thus they generally see something like "$Person engaged in a detailed $deepTaboo simulation with computer using the $SoftwareProduct" and start looking around for someone to crucify (usually both $Person and the owner of $SoftwareProduct) for 'promoting' $deepTaboo -- without asking "who was harmed?" Even if the constitution is actually consulted and the parties are spared pearl-clutching prosecution about it, the court of public opinion will punish the company just as hard, pressuring gatekeepers like Apple to ban it, etc. And thanks to the deep, disturbing taboos that can be involved, it's super hard to find someone who will stick their neck out to challenge all this. I know I wouldn't, because I know I would be immediately branded as "Joe Bloggs, Fan and Advocate of (Insert disgusting perversion)" I wonder if this is something more advanced societies would come to terms with. I know ours is not equipped to handle it in a smart way.

lvass 3y ago

Not owning the software you use is terrible at any level, but this is much worse than usual. Looks like each day that passes stallman's ideas become more relevant, yet software people seem to be forgetting them. Please do not be one of those people, facilitating proprietary software is becoming increasingly evil.

nlh 3y ago

I feel incredible empathy for the people who have had AI companions taken away from them. I don't judge (nor should any of you) and it seems heartless and cruel for the company to have done this. I hope some folks double-down on open source implementations of this and get the community back the intimacy and companionship they need. Humans are creatures evolved to seek intimacy and connection, and for whatever and varied reasons, some people just can't find it in the real world. If a virtual companion helped people feel that connection we all so very much want, then so be it. I hope this works out for everyone.

wing-_-nuts 3y ago
Even if you ignore this use case, I can totally see the demand for a private LLM that doesn't expose your deepest secrets to a corporate or government overlord. Maybe the current gpt isn't good enough to be a therapist but who's to say where we are in a decade, and giving outside entities access to such sensitive info is ripe for exploitation
mattrick 3y ago
At some point (maybe that point is now), I'm sure we'll see the right combination of hardware and software performance where it'll be trivial for anyone to run advanced AI models on their own devices so it's not controlled by a handful of companies. One thing I'm definitely interested in is when games begin to use these models to generate dialogue to make each player's experience unique.

burntalmonds 3y ago

People are going to want to host their own AIs. They'll want to know they are having private conversations with AI and that their AI can't be "killed" or altered.

oh_sigh 3y ago

Maybe this product would work better as a fake AI. Just as a matchmaking service for two lonely people. You could even lie to them and tell each that the other was an AI.

clnq 3y ago
Replika is specifically advertised as a chatbot that creates a safe space with no drama, stress, or judgement. Humans come with many more strings attached, especially lonely ones. Usually, there is a reason why they are socially unappealing - a fair reason or not, it exists. I remember trying Omegle/Chatroulette when it was popular, meeting a lot of lonely people, and it was just awkward.
npteljes 3y ago
Doesn't work, because the premise is asymmetric emotional energy. The AI only gives, and doesn't need anything in return (on an emotional level). People might have the genuine need for a relationship, but that doesn't mean that they can properly partake in one.

nostrademons 3y ago

"This is a story is about people who loved someone who had a secret master that can mine control them not to love you anymore overnight." There's something both profound and ironic about this statement, because it happens in real life all the time. Think of LGBTQ Millennials whose QAnon or Fox News consuming parents decide to disown them because they're sinning against God. People who marry a workaholic and then spend 30 years wishing their spouse was home for dinner instead of working late for a faceless corporation. Workers who pour their heart and soul into their job and then get laid off as soon as a downturn hits. People who get friend-dumped by their maid of honor a month after their wedding. Lovers who get cheated on, and whose partner then falls for their best friend. Ironically, the reason people turn to services like Replika is that they want to feel that human connection without the risk of betrayal. They just found out that the risk of betrayal always exists, even when you're speaking to a computer. This is a story about betrayal, and loneliness, and disappointment, all wrapped up in trendy tech with a villain to name. But it's popular because betrayal, and loneliness, and disappointment are such strong and universal emotions. One of the most profound statements I heard when I was forever-alone was that "Entering a relationship means giving another person the power to destroy you, and then trusting that they won't. That's the whole point." Because when you realize that you can't escape vulnerability, you're forced to manage it, and that leads to the rabbit hole of learning to identify your emotions, control them, decide rationally how much you're going to invest in a person or venture, but still be open enough to have emotions and let them flow naturally, but managing them rather than letting them attach to whatever tickles your fancy at the moment.

ocrow 3y ago

ERP: Erotic Roleplay, not Enterprise Resource Planning

Reimersholme 3y ago
[dead]
SamBam 3y ago
Thank you, I couldn't find where it was defined in the article.
xp84 3y ago
And take it from me, you do not want to mix those up in a meeting with the COO.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

rasengan0 3y ago

Corporate trains the AI accordingly, optimizing for human happiness ... https://twitter.com/CogRev_Podcast/status/1627675037267374083?s=20 https://twitter.com/CogRev_Podcast/status/162767503726737408... where humans == shareholders && != users

10g1k 3y ago

Yeah, I was curious, and downloaded it for a try. It very soon became apparent to me that it was a personal information harvesting tool. It asks for your favourite colour, where you grew up, pet name, et cetera. I would not be surprised if this company is unofficially associated with very bad people doing very bad things. So after two such questions, I deleted it.

JamesLeonis 3y ago

The update at the end really captures the whole essence: > Replika cannot love you not because it is not human, but because it is a tendril connected to a machine that only knows how to extract value from your presence. It is a little piece of a broader Internet that is not designed to be a place for users, or creators, but for advertisers. Anything it offers– any love, any patience, any support– can be taken away at a moment’s notice when there exists something more dangerous to its bottom line than your potential unsubscription. > The moral of Replika is not to never love a fictional character, or a virtual pet. The moral of Replika is to never love a corporation.

BigCryo 3y ago

Take my wife, please

valdiorn 3y ago

This feels eerily similar to the virtual girlfriend (Ana de Armas' role) in Blade Runner 2049. All the way down to how they just smash her to bits and laugh at K when she "dies".

SergeAx 3y ago

I am 100% sure they received a warning from Apple. Although it seems that they have a web version, their revenue from Appstore should be more than 50% and up to 75% given their demographics and current market state.