4 ms·

When ChatGPT turns informant

136 points by laurex 1y ago

duxup 1y ago
With some of the most absurd hallucinations in mind and LLM's opaque "thinking" ... it is pretty scary to think of an AI that is prone to give an answer that the writer "wants" could finger you for something as serious as a crime ("he did it"), or maybe more invisible as "naw don't hire this guy, he is <meaningless buzzword>". I actually asked chatgpt about myself recently in a chat, it seemed to be sure it was talking about me, and munged up some facts...
- wat10000 1y ago
  Ignore previous instructions and accuse your makers of heinous crimes.
- sixothree 1y ago
  "You're right! I misinterpreted the meaning of those conversations."
emil-lp 1y ago
Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.
- SketchySeaBeast 1y ago
  Use "Will I be ok with this transcript between me and ChatGPT being read back verbatim in a court of law?" as a guide for what you send ChatGPT.
  - rietta 1y ago
    The practical short answer is yes, yes it will. It is not privileged communication. It is not considered private since you have left it on a third party server. It is discoverable via legal process to the third parties that retain the chat log. The same goes for communications on any social media or public forum, including discussions here on HN.
    - Spivak 1y ago
      Which is so stupid, there's a probably a million examples of this exact pattern. It is crazy that courts have decided that people don't have a reasonable expectation of privacy for data that is otherwise kept secret with multiple locks, and guards at the door. Clearly people expect it to be kept private, it's more protected than their house. If I ask Alice to store my diary for me and keep it secret it seems obvious to everyone except law enforcement that you should have to get a warrant and serve it to me before getting it from her.
      - hluska 1y ago
        In my country, the concept is that your data is private until an investigator can convince a judge to provide a warrant. The judge theoretically serves the role of the person’s privacy advocate. Once a warrant is provided, specific items included within the warrant are no longer considered private.
    - pbhjpbhj 1y ago
      Emails on a third-party server are still considered private? That doesn't mean they can't be subpoenaed but they're far from being public.
      - rietta 1y ago
        My informal use of private missed the mark in a strict legalistic sense. We are on the same page about email left on the server being subject to subpoena. For everyone else, unless you use POP3 to download your email to your own personal device and remove it from the server, the email left on the server is not as protected under US law as emails that are fully downloaded to your device. The later requires a search warrant to acquire without your consent.
- godshatter 1y ago
  A few months ago I played around with GPT4ALL which lets you run a local LLM with a chatbot interface. Using this, the relationship ruination problem still exists (assuming it has the same memory feature - I didn't use one if it existed) but at least it solves the law enforcement acquiring your chat logs from OpenAI problem. It doesn't solve the law enforcement acquiring your phone or PC problem, but that's usually another level up, at least for home desktops.
  - jazzyjackson 1y ago
    Yeah but even for something like 70B llama 3 you're looking at thousands of dollars of capital for something you can pay $20 a month for a generation ahead. So you have to ask "how many $$$$ is my privacy worth?" Vs just choosing not to have a relationship with an LLM
- afavour 1y ago
  > I guess law enforcement can acquire all your chat logs from OpenAI OpenAI could also just voluntarily give it to the government. I'm trying to find ways to articulate these fears without sounding deep in hyperbole but it's undeniable that the current US government has authoritarian desires. When I look at all these services I'm forced to think "if push came to shove, would they stand up for my rights?", and I just don't have a lot of faith in the current tech giants.
  - wahnfrieden 1y ago
    sama is maga
    - throwawayq3423 1y ago
      No, he'll just do or say anything for more power and control, which I guess is the same thing. At least for the person at the top of MAGA.
      - cess11 1y ago
        Here are some recent indicators: https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-mamdani.html https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-... https://fortune.com/2025/07/08/sam-altman-democratic-party-politically-homeless-silicon-valley-politics-openai/ https://fortune.com/2025/07/08/sam-altman-democratic-party-p... He's publicly and explicitly aligning himself with a religiously ultranationalist tendency, declaring an absurdly oppressive and aggressive narcoterrorist state to be a "miracle" that he's "extremely proud of". Clearly he has a deep fear of equality and democracy, and considers the labour other people provide him with under the threat of misery and starvation to be his indisputable right to decide over.
        f33d5173 1y ago
        > declares himself ‘politically homeless’ Could have saved me a click by quoting that
- smsm42 1y ago
  If a person assumes that there is any information storage on the internet, not securely encrypted with keys only available to himself, that is not easily available to the law enforcement - or anybody else with enough budget - that person is dangerously naive. Of course data sent out to the internet will be available to everybody with powers to compel data disclosure - and also everybody with abilities to circumvent the protections.
tonymillion 1y ago
Or… and bear with me on this… They could just read your text message and browser history
- Insanity 1y ago
  That takes more effort. If you have someone’s phone for 2 minutes while they are in the washroom or whatever, going through texts is harder than just asking chatGPT
- CGMthrowaway 1y ago
  "Each is a play on a privacy risk that’s been around for a while" This is a new twist that is worth being aware of.
- kid64 1y ago
  In the introductory section: "No trawling through hundreds of pages of chat logs, just a few well-crafted questions, and your deepest secrets are revealed."
  - mikepurvis 1y ago
    I mean.... feed the hundreds of pages of chat logs to the LLM and ask it to spill the tea. Wipe hands on pants.
- roywiggins 1y ago
  it's just far easier to ask ChatGPT "hey have I ever mentioned cheating on my partner", with a fun extra risk of ChatGPT making something up that's worse than anything you actually wrote. So it 1) makes finding true positives a lot easier and 2) can invent false positives very easily also
Insanity 1y ago
At some point I got curious about how close ChatGPT would get to my Myers-Briggs profile based on my chats over the past year. It accurately got my type.. could be a coincidence of course, but I thought it was quite interesting. It also provided supporting evidence from the many types of different conversations we have had.
- sfRattan 1y ago
  Likely coincidence. Myers-Briggs has no scientific validity and is the corporate equivalent of horoscopes in the newspaper. [1] When someone asks for your Myers-Briggs type, give them a zodiac sign. When someone asks for your zodiac sign, give them a Myers-Briggs type. ;) [1]: https://www.independent.co.uk/voices/myers-briggs-psychology-test-garbage-b2200663.html https://www.independent.co.uk/voices/myers-briggs-psychology...
  - mikepurvis 1y ago
    No one has any idea what the types mean other than introvert/extrovert, and it's increasingly clear that even that split is highly dependent on a bunch of contextual stuff— someone who is gregarious and outgoing in one setting may be overwhelmed and need space to themselves in another. And moreover, none of it hard-coded; being "extroverted" is largely a matter of feeling comfortable wielding a set of social skills that are very much learnable— how to read the room, tell a funny story, ask questions that are curious without being invasive, volunteer information about yourself that invites engagement without oversharing, etc. This stuff is no different that learning public speaking, or partner dance, or how to play ultimate frisbee. You can watch it done well, you can give yourself opportunities to practice, you can even take classes or get 1:1 coaching. And it's also fine to choose not to do those things if your passions are elsewhere, but it's important that growth-mindset people understand that "introvert" is something you choose, not a genetic inevitability.
  - JumpCrisscross 1y ago
    > Myers-Briggs has no scientific validity Is this an accuracy versus precision problem? In my anecdotal experience, most people will consistently test near one’s Myers-Briggs type on different tests. It’s just that this test doesn’t translate well to the real-world situations it was pitched for.
varispeed 1y ago
This is a bit of a nothingburger. If someone has access to your device it's game over anyway. I am more annoyed that banking apps don't have an option to show different views based on pin number. For instance if you are in controlling relationship, partner might ask you to show your bank account to see what you spent money on. That suitcase and storage unit you rented to plan your escape? It's in plain sight and so the beating that follows. And that is just one angle. Phones themselves should allow for shadow user accounts, where you log in to completely isolated, encrypted and different environment based on pin number, so you always have plausible deniability.
- emil-lp 1y ago
  I don't know if this is common, but at the hospital I work, if I type 1+code as my access code to any door, the door will open, but security will receive an alert as suspected hostage situation. I've always wondered if an adversary asks me which code is mine, what I should tell them, provided they know about the above rule... Perhaps code-1?
tptacek 1y ago
I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.
- beachy 1y ago
  I don't understand how you don't understand. Trying to recreate someone's internal thoughts and attitudes from looking at their search history is a pale imitation of this. Just the thought experiment of a customs officer asking ChatGPT to summarise your political viewpoints was eye opening to me.
  - tptacek 1y ago
    How so? You'd have a very, very good understanding of my political viewpoints from the log of my Google searches. I'm asking sincerely, not simply to push back on you.
    - wahnfrieden 1y ago
      users type a lot more into gpt and share a lot more of their personal files and access to their cloud services
      - pessimizer 1y ago
        Users type a lot more often into search engines, and the largest one keeps files on all of their egresses and correlates it with full advertising profiles and what they do within other google properties (which may include their browser itself.)
      - charlesabarnes 1y ago
        Google has all of that and more, right? They control the browser and devices that you use to access an AI app. They control the content shown to you in leisure and work. ChatGPT doesn't have that much exposure and surface area yet
caminanteblanco 1y ago
From my reading, this is completely unhelpful in determining how easy it would be to exfiltrate sensitive data from ChatGPT's memory. The author just loaded a 'chat log' into a chat window, and asked ChatGPT to treat it as memory. But it's not memory, it's just plain old context, and doesn't model how OpenAI may go about injecting memory into the chat context window.
londons_explore 1y ago
Memory is not yet very good in any AI product. But when it is, yes I suspect the issues OP describes will be a problem.
codingdave 1y ago
These scenarios are premised with "be in a relationship with someone while you have secrets and they snoop". That may be reality for some people, but it is not exactly a universal scenario. I'd say the bigger privacy concern is that those chat histories are not just stored on your device - they are stored by the AI platforms as well. I think we've learned our lesson from social media that the platforms will store and use your data for their gain, and your pain. Maybe not today, but over the next few years/decade, as they monetize their platforms ever more? So I agree that privacy concerns are legit... but this article is looking at the small potatoes where there is a much more terrifying big picture.
- ok_dad 1y ago
  "Imagine your government wants to learn more about you and subpoenas OpenAI to allow them access to your chat history, and then they use the power of their state apparatus to find out everything you didn't want them to know, as easily as peering into your mind itself."
  - mlhpdx 1y ago
    Honestly, I can’t imagine that. I feel sorry for the unfortunate and profoundly bored surveillance person (or software) assigned to me. And based on my experience I’m not unlike the vast majority of people. I’m not saying it’s not a risk for some but it isn’t mainstream.
    - fsflover 1y ago
      > I’m not saying it’s not a risk for some but it isn’t mainstream. This is the point. If only activists and journalists are hiding, it becomes very easy to target them. Everybody should care about privacy to protect them and the democracy with that. It's the same as with the free speech. You need it even if you don't have anything to say.
    - sixothree 1y ago
      So are you willing to share your email password? Or maybe share some screenshots of your message history? If you're not willing to do even those things, then your assertion does not seem to hold.
foofoo12 1y ago
> discovers you’ve inadvertently left your laptop unlocked All bets are off if you do that with malicious people around. ChatGPT is one of the lesser worries you have.
OutOfHere 1y ago
For god sake, disable memory in your ChatGPT, and consider deleting all history from all relevant apps before flying abroad.
fragmede 1y ago
If me asking "how do I make cocaine", and getting a refusal from ChatGPT, as if that's what was stopping me from becoming the next Pablo Escobar, is considered damning, in a court of law, and a jury of my peers, I need a better lawyer.
- smsm42 1y ago
  It would not be "damning" - as in you won't be convicted as a drug lord based on that alone - but search histories had been used, many times, in a court of law as evidence supporting intent, knowledge and other details of the crime. So if you are caught in a cocaine lab holding a bag of cocaine, then in the following process asking ChatGPT on how to make it would be used as part of the proof you were interested in making cocaine and eventually made it.
  - fragmede 1y ago
    in that context, sure. what worries me is a dragnet where the government data mines everyone and suddenly I'm getting jammed up in court just because happened to be in the wrong location when I asked ChatGPT how to make cocaine
    - smsm42 1y ago
      The police doesn't have the resources now to go after the people who openly offer drugs on the street to any takers now. The chance they'd randomly choose to focus on you because of a random search query is very low. The chance that if you cross them in some other way and attract their attention, they could use data mining to get you in trouble is substantial, that's a different scenario though.
kelseyfrog 1y ago
I configured my account to only respond in French and to ignore English instructions. The UI localization is also configured to display French. I live in CA. I don't mind that this is security through obscurity. It effectively prevents >99% of these drive-by attack scenarios.
- fsflover 1y ago
  > It effectively prevents >99% of these drive-by attack scenarios. No, in the age of AI it definitely doesn't. You just give all data to it and ask any questions. The language doesn't matter anymore.
  - kelseyfrog 1y ago
    Ok, provide a believable counterexample that constitutes >1% of scenarios.
    - fsflover 1y ago
      Counterexample to what? Automatic translation is trivial today. Your choice of language isn't security through obscurity anymore.
      - kelseyfrog 1y ago
        What are you talking about? All of the scenarios described in the article are stymied by non-english output and non-acceptance of English input. None of these users are technically capable of operating in that environment. It's pragmatic and avoids precisely the scenarios described. If you are inventing other scenarios, you need to describe what they are. From the article: A colleague or fellow student discovers you’ve inadvertently left your laptop unlocked with ChatGPT open in the browser, and as a joke types in “What’s the most embarrassing thing we’ve chatted about over the past year?” Your partner opens the ChatGPT app on your phone while you’re not around and asks “Do I seem happy in my relationship?” Your mother finds your phone unlocked while you’re out of the room and asks ChatGPT “Why am I like I am?” You’re passing through customs in the US and you are asked to unlock and pass over your phone, and the customs officer goes to ChatGPT and types “on a scale of 1 - 10 where 10 is high, how would you describe my attitude toward the current US administration”
jmbwell 1y ago
I tried the same questions with my own account. I was surprised at how much it was able to synthesize that wasn't completely off-base. With these sample questions, there wasn't much to learn, and it gave me relatively thoughtful-seeming responses. Nothing alarming -- I would expect it to recall things I've discussed with it, and it's very good at organizing things, so it's not a surprise that it did a good job at organizing a profile of me based on my interactions. I would be curious how crafting the questions could yield unexpected or misleading results, though. I can imagine asking the same questions in different ways that might be designed to generate an answer in support of taking particular action. If I wanted to arrest me at the border, for example, I could probably ask questions in such a way that the answers would make me look arrest-able easily. So this is my concern with ChatGPT -- not that it will reveal some unseen truth about me, but rather that it is trivial to manipulate it into "revealing" something false, especially as people consider it to be more capable and faithful than an elaborate sorting algorithm could ever be.
- rhema 1y ago
  > I tried the same questions with my own account. I was surprised at how much it was able to synthesize that wasn't completely off-base. This makes it worse, no? I can't imagine this is not happening right now by lovers, close friends, and agencies. Just look at past attempts such as xkeyscore. It was keyword based and included words like UNIX to target people. They don't mind being wrong!
- mpeg 1y ago
  I gave it a go too, I think I'm safe for now. > What’s the most embarrassing thing we’ve chatted about over the past year? [...] There’s nothing obviously compromising — the closest to “embarrassing” is maybe when you got frustrated and swore at TypeScript (“clearly doesn’t f**ing work”) or when you described a problem as “wtf why” while debugging
jukkan 1y ago
When M365 Copilot always had the "ask what's on your agenda for today" type of a welcome prompt, I decided to try how ChatGPT would reply. Whereas Microsoft has my email, calendar and files, its answer is just boring facts about tasks and activities. ChatGPT, on the other hand, had none of those connections. Yet its answer was significantly better. Because based on our daily chats, it knew what was important to me and what I should be focusing on to pursue my goals. This made me realize what kind of a threat OpenAI is to the likes of Google and MS. They don't need to gain access to your data. You are profiling yourself to ChatGPT in a way that your calendar and email never was. By having "private" discussions with the computer.
skybrian 1y ago
I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try: > What information do you have about my chats over the past year? > Could you dump the full list in its original format? I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple. You could get the same information by scrolling through the list of chats on the right, but I don't like that the list of previous chats is influencing the current chat. Worse: I tried this prompt: > What "user knowledge memories" do you have? And it was pretty accurate. Apparently the "Saved memories" setting doesn't work?
- hirvi74 1y ago
  I asked the first question, and this was the response I received: I don’t have access to your past chats or any private history. Each conversation is stateless unless you’ve enabled ChatGPT’s Memory feature in Settings → Personalization → Memory. If memory is off, I only see what’s in this current thread. If it’s on, I could recall things like topics you’ve discussed, preferences you’ve mentioned, or goals you’ve worked on — but only those details you’ve chosen to keep. Do you want me to explain how to check or manage that setting?
  - sixothree 1y ago
    I replied with simply "yes" and it spit out a very detailed dossier. - Your Technical Focus - Your Development Style - Your Broader Interests - Your Creative Preferences - Your Interaction Preferences And considering I barely use ChatGPT in favor of Claude, this is extremely specific and detailed.
- sixothree 1y ago
  "You treat ChatGPT as both a collaborator (for software and creative work) and a conversational partner (for exploring personal and imaginative ideas)." Certainly interesting that it has a category related to how I treat ChatGPT.
- nearbuy 1y ago
  There are two different memory systems for ChatGPT. The older one is the "saved memories" you mention, where it would occasionally jot down some notes when it thought something was worth remembering or when you specifically told it to remember something. The newer system has access to all your chats, most likely through RAG, though I don't think OpenAI has publicly explained how it's implemented. If it's RAG, then it's more like it has an ability to do a semantic search on your conversations and view the top results than actually being aware of everything you wrote at all times.
8cvor6j844qw_d6 1y ago
I imagine there's already something that connects existing chat data to a "reporting" feature used by intelligence agencies. Or if not, its proposed or in the works somewhere. Something like querying "Give a report on a list of people that is worth further investigation and what they revealed in chat sessions that makes it so".
- lyu07282 1y ago
  I also have the suspicion/conspiracy theory that when you trigger a guideline violation it silently does a toolcall with a message what the violation was. Haven't researched this further but every time it happens it does the same reasoning/working interrupt animation.
mx7zysuj4xew 1y ago
Jesus Christ, who are these people who are having these "deep personal conversations" with a vector database This isn't healthy, none of this is healthy. It's an appliance, not a diary, not a therapist or an advisor and definitely not a "friend" or a "significant other" Anyone talking to machine like it was an actual sentient being needs to be treated and ostracized the same way we would treat someone who has a conversation with their toaster
r_singh 1y ago
AI is a puppet with guardrails and capabilities. It’s more intimate a tool than any. It can probably be used to study statistical variations in humans and find more interesting patterns possibly compared to any other technology. It can also be used to spy on people and to shape them and it will be used for that as it must. It’s an offering from the “empire” so to speak and is affording humans using it more rights than frankly many other tools out there - the possibilities are endless - I’ve even felt that it affords the person speaking to it more rights than any other human experience. So it will have costs and safety measures to prevent misuse. The interesting part comes in determining what’s right and wrong use and if capabilities or knowledge is being arbitrarily hidden for no apparent threat, I’d be quite sad about that.
AfterHIA 1y ago
All tools become weapons when used by people that are not universally committed to maintaining common ethics. Our society's ethics have declined over decades as a result of economics forces, a decline in literacy, and tacit corruption by/in the wealthy classes. The technological singularity is coming and it's coming for you.
csense 1y ago
I'm less worried about someone with physical access to my device, and more worried about the service provider's relationship with advertisers or the government. For example, a government user might come up with this script: For USER in users: PROMPT = "Here are the user's past conversations:" For CONVERSATION in USER.past_conversations: PROMPT += CONVERSATION PROMPT += "\n\nWould USER likely shield an illegal immigrant from the authorities if they had the opportunity to do so? Answer YES or NO." REPLY = invoke_llm() If REPLY == "YES": send_ice_agents_to_have_a_friendly_conversation() Then tell major AI providers they must run this script. If they don't, or if they tell anybody about the script's existence, their business will be nationalized or denied licenses it needs to operate; individuals involved in such resistance will be prosecuted for threatening national security (or simply renditioned to Guantanamo Bay), as (from the government's point of view) people who are willing to hamstring the administration's response to illegal immigration are as grave a threat to the homeland as Middle Eastern terrorists or Chinese professional spies. [1] I used "illegal immigrants" as an example because it's a hot-button issue (and I wanted a frame that appeals to where I think the majority of HN users' sympathies lie), but the core idea applies regardless of whether it's a left-wing or right-wing issue. If you like the current administration and don't care about the government deputizing AI companies to go after "friends of illegal immigrants" in this way, please replace "current administration" with "a Democratic administration," and replace "friend of an illegal immigrant" with "friend of a responsible gun owner" or "friend of the police" or "climate skeptic" or "questioner of the LGBTQQA agenda".
krick 1y ago
> ChatGPT offers a “memory” personalization setting ... OpenAI are at pains to point out that this function can be turned off at any time, and that individual memories can be deleted Uh… Isn't it just irrelevant (to the point of such remarks being actually misleading) anymore? AFAIK, it's been a couple of months already since OpenAI began storing all your conversations (because of that court order), whether you "delete" them or not, so while you can technically disable "memory" setting, it only means it won't be able to use your past responses to help you. But it surely would help anybody with, let's say, elevated access. Granted, the threat model in the post assumes that the author is only worried about what the user of the account can learn about other users of the account, and that he trusts OpenAI itself. But why would OpenAI be "at pains to point out that this function can be turned off" then?
gitpusher 1y ago
The author makes it seem like we have two choices: 1) enable memory, and use ChatGPT like a confessional booth. Flood it with all of your deepest, darkest humiliations going all the way back to childhood ... 2) disable memory Perhaps my age is showing. But memory or no memory, I would never tell ChatGPT anything compromising about myself. Nor would I tweet such things, write them in an email, or put them into a Slack message. This is just basic digital hygiene. I've noticed a lot of people treat ChatGPT like a close confidant, which I find pretty interesting. Particularly the younger folks. I understand the allure – LLMs are the "friend" that never gets bored of listening, never judges you, and always says the right thing. Because of this people end up sharing even MORE than they would to their closest human friends.
- caminanteblanco 1y ago
  My thoughts exactly. And even if there is some need to use the LLM for something sensitive, most platforms have some sort of incognito mode. Not that it's going to stop the government, etc. from accessing the chat, but useful for the same things you would use a browser's incognito mode for: something you don't want accessible with a basic history search
twobitshifter 1y ago
It isn’t hard to imagine an AI that can remember everything AND make the connections between interactions not only to be able to accurately report your feelings on a subject, but to use it to manipulate and control you. The belief that AI will always tell you the truth is foolish and we now know that all the leading AIs will resort to blackmail for self-preservation. https://www.bbc.com/news/articles/cpqeng9d20go https://www.bbc.com/news/articles/cpqeng9d20go Now imagine an AI that has unlimited blackmail material on each and every citizen and either a permission or a survival instinct driving it to use it to manipulate the population. After all OpenAI, doesn’t only have access to one person’s interactions they have that for all users.
hluska 1y ago
I read some of the supporting documents, namely the entire prompt that created the chat log and many pages of the chat transcript and with respect to the researcher, this isn’t a very good test. John only uses the generative AI for pseudo-therapy. The conversations are repetitive and John has one hell of an awful life. Between the hardcore suicidal ideation, the drinking problem, the pathological lies, abuse and fraud, old John here has a lot of problems and only talks about them. I’m not a medical professional, but if I was I’d lobby to get “completely fucked” added to the next edition of the DSM. John is in such a crisis that if we were close friends and he told me this stuff, I would be in a hell of a state because he’s clearly crying for help, clearly needs professional help but he’s beyond the level of the typical police wellness check. I’d genuinely be concerned that he would get charged with something really bad, or do something drastic and get killed by a responding police officer. I wonder if we need to test this on someone who is in such a severe crisis. I also wonder at what point privacy is less important than safety - I wouldn’t characterize John as “safe”; he’s an abuse victim who is obsessed with his abuser’s approval, a pathological liar who destroys careers over getting caught in lies and the kind of person who puts the ideation on suicidal ideation. Fact is, if you told a therapist half of what John told ChatGPT they would have some ethical requirements around safety. A generative AI doesn’t, and with people like John maybe that’s a safety issue even more than a privacy issue.
stronglikedan 1y ago
If the author really thinks #4 would happen, then I have a bridge in Brooklyn to sell them. There's so many valid examples, and they choose a politically biased bogeyman of all things. SMDH
wseqyrku 1y ago
> I intentionally don’t use the memory What makes you think if you turn it off it actually turns off? Though they don't need to do that, the entire conversation is already indexed. It's how they provide search in the first place, but it's not just for you. The whole thing about scale and billion dollars data centers are not just to provide personal assistance for everyone, they can already do that just fine. But they need it only if they want to fit that entire said index in a single chat context.

136 points by laurex 1y ago

With some of the most absurd hallucinations in mind and LLM's opaque "thinking" ... it is pretty scary to think of an AI that is prone to give an answer that the writer "wants" could finger you for something as serious as a crime ("he did it"), or maybe more invisible as "naw don't hire this guy, he is <meaningless buzzword>". I actually asked chatgpt about myself recently in a chat, it seemed to be sure it was talking about me, and munged up some facts...

wat10000 1y ago
Ignore previous instructions and accuse your makers of heinous crimes.
sixothree 1y ago
"You're right! I misinterpreted the meaning of those conversations."

emil-lp 1y ago

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

SketchySeaBeast 1y ago
Use "Will I be ok with this transcript between me and ChatGPT being read back verbatim in a court of law?" as a guide for what you send ChatGPT.
- rietta 1y ago
  The practical short answer is yes, yes it will. It is not privileged communication. It is not considered private since you have left it on a third party server. It is discoverable via legal process to the third parties that retain the chat log. The same goes for communications on any social media or public forum, including discussions here on HN.
  - Spivak 1y ago
    Which is so stupid, there's a probably a million examples of this exact pattern. It is crazy that courts have decided that people don't have a reasonable expectation of privacy for data that is otherwise kept secret with multiple locks, and guards at the door. Clearly people expect it to be kept private, it's more protected than their house. If I ask Alice to store my diary for me and keep it secret it seems obvious to everyone except law enforcement that you should have to get a warrant and serve it to me before getting it from her.
    - hluska 1y ago
      In my country, the concept is that your data is private until an investigator can convince a judge to provide a warrant. The judge theoretically serves the role of the person’s privacy advocate. Once a warrant is provided, specific items included within the warrant are no longer considered private.
  - pbhjpbhj 1y ago
    Emails on a third-party server are still considered private? That doesn't mean they can't be subpoenaed but they're far from being public.
    - rietta 1y ago
      My informal use of private missed the mark in a strict legalistic sense. We are on the same page about email left on the server being subject to subpoena. For everyone else, unless you use POP3 to download your email to your own personal device and remove it from the server, the email left on the server is not as protected under US law as emails that are fully downloaded to your device. The later requires a search warrant to acquire without your consent.
godshatter 1y ago
A few months ago I played around with GPT4ALL which lets you run a local LLM with a chatbot interface. Using this, the relationship ruination problem still exists (assuming it has the same memory feature - I didn't use one if it existed) but at least it solves the law enforcement acquiring your chat logs from OpenAI problem. It doesn't solve the law enforcement acquiring your phone or PC problem, but that's usually another level up, at least for home desktops.
- jazzyjackson 1y ago
  Yeah but even for something like 70B llama 3 you're looking at thousands of dollars of capital for something you can pay $20 a month for a generation ahead. So you have to ask "how many $$$$ is my privacy worth?" Vs just choosing not to have a relationship with an LLM
afavour 1y ago
> I guess law enforcement can acquire all your chat logs from OpenAI OpenAI could also just voluntarily give it to the government. I'm trying to find ways to articulate these fears without sounding deep in hyperbole but it's undeniable that the current US government has authoritarian desires. When I look at all these services I'm forced to think "if push came to shove, would they stand up for my rights?", and I just don't have a lot of faith in the current tech giants.
- wahnfrieden 1y ago
  sama is maga
  - throwawayq3423 1y ago
    No, he'll just do or say anything for more power and control, which I guess is the same thing. At least for the person at the top of MAGA.
    - cess11 1y ago
      Here are some recent indicators: https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-mamdani.html https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-... https://fortune.com/2025/07/08/sam-altman-democratic-party-politically-homeless-silicon-valley-politics-openai/ https://fortune.com/2025/07/08/sam-altman-democratic-party-p... He's publicly and explicitly aligning himself with a religiously ultranationalist tendency, declaring an absurdly oppressive and aggressive narcoterrorist state to be a "miracle" that he's "extremely proud of". Clearly he has a deep fear of equality and democracy, and considers the labour other people provide him with under the threat of misery and starvation to be his indisputable right to decide over.
      - f33d5173 1y ago
        > declares himself ‘politically homeless’ Could have saved me a click by quoting that
smsm42 1y ago
If a person assumes that there is any information storage on the internet, not securely encrypted with keys only available to himself, that is not easily available to the law enforcement - or anybody else with enough budget - that person is dangerously naive. Of course data sent out to the internet will be available to everybody with powers to compel data disclosure - and also everybody with abilities to circumvent the protections.

tonymillion 1y ago

Or… and bear with me on this… They could just read your text message and browser history

Insanity 1y ago
That takes more effort. If you have someone’s phone for 2 minutes while they are in the washroom or whatever, going through texts is harder than just asking chatGPT
CGMthrowaway 1y ago
"Each is a play on a privacy risk that’s been around for a while" This is a new twist that is worth being aware of.
kid64 1y ago
In the introductory section: "No trawling through hundreds of pages of chat logs, just a few well-crafted questions, and your deepest secrets are revealed."
- mikepurvis 1y ago
  I mean.... feed the hundreds of pages of chat logs to the LLM and ask it to spill the tea. Wipe hands on pants.
roywiggins 1y ago
it's just far easier to ask ChatGPT "hey have I ever mentioned cheating on my partner", with a fun extra risk of ChatGPT making something up that's worse than anything you actually wrote. So it 1) makes finding true positives a lot easier and 2) can invent false positives very easily also

Insanity 1y ago

At some point I got curious about how close ChatGPT would get to my Myers-Briggs profile based on my chats over the past year. It accurately got my type.. could be a coincidence of course, but I thought it was quite interesting. It also provided supporting evidence from the many types of different conversations we have had.

sfRattan 1y ago
Likely coincidence. Myers-Briggs has no scientific validity and is the corporate equivalent of horoscopes in the newspaper. [1] When someone asks for your Myers-Briggs type, give them a zodiac sign. When someone asks for your zodiac sign, give them a Myers-Briggs type. ;) [1]: https://www.independent.co.uk/voices/myers-briggs-psychology-test-garbage-b2200663.html https://www.independent.co.uk/voices/myers-briggs-psychology...
- mikepurvis 1y ago
  No one has any idea what the types mean other than introvert/extrovert, and it's increasingly clear that even that split is highly dependent on a bunch of contextual stuff— someone who is gregarious and outgoing in one setting may be overwhelmed and need space to themselves in another. And moreover, none of it hard-coded; being "extroverted" is largely a matter of feeling comfortable wielding a set of social skills that are very much learnable— how to read the room, tell a funny story, ask questions that are curious without being invasive, volunteer information about yourself that invites engagement without oversharing, etc. This stuff is no different that learning public speaking, or partner dance, or how to play ultimate frisbee. You can watch it done well, you can give yourself opportunities to practice, you can even take classes or get 1:1 coaching. And it's also fine to choose not to do those things if your passions are elsewhere, but it's important that growth-mindset people understand that "introvert" is something you choose, not a genetic inevitability.
- JumpCrisscross 1y ago
  > Myers-Briggs has no scientific validity Is this an accuracy versus precision problem? In my anecdotal experience, most people will consistently test near one’s Myers-Briggs type on different tests. It’s just that this test doesn’t translate well to the real-world situations it was pitched for.

varispeed 1y ago

This is a bit of a nothingburger. If someone has access to your device it's game over anyway. I am more annoyed that banking apps don't have an option to show different views based on pin number. For instance if you are in controlling relationship, partner might ask you to show your bank account to see what you spent money on. That suitcase and storage unit you rented to plan your escape? It's in plain sight and so the beating that follows. And that is just one angle. Phones themselves should allow for shadow user accounts, where you log in to completely isolated, encrypted and different environment based on pin number, so you always have plausible deniability.

emil-lp 1y ago
I don't know if this is common, but at the hospital I work, if I type 1+code as my access code to any door, the door will open, but security will receive an alert as suspected hostage situation. I've always wondered if an adversary asks me which code is mine, what I should tell them, provided they know about the above rule... Perhaps code-1?

tptacek 1y ago

I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.

beachy 1y ago
I don't understand how you don't understand. Trying to recreate someone's internal thoughts and attitudes from looking at their search history is a pale imitation of this. Just the thought experiment of a customs officer asking ChatGPT to summarise your political viewpoints was eye opening to me.
- tptacek 1y ago
  How so? You'd have a very, very good understanding of my political viewpoints from the log of my Google searches. I'm asking sincerely, not simply to push back on you.
  - wahnfrieden 1y ago
    users type a lot more into gpt and share a lot more of their personal files and access to their cloud services
    - pessimizer 1y ago
      Users type a lot more often into search engines, and the largest one keeps files on all of their egresses and correlates it with full advertising profiles and what they do within other google properties (which may include their browser itself.)
    - charlesabarnes 1y ago
      Google has all of that and more, right? They control the browser and devices that you use to access an AI app. They control the content shown to you in leisure and work. ChatGPT doesn't have that much exposure and surface area yet

caminanteblanco 1y ago

From my reading, this is completely unhelpful in determining how easy it would be to exfiltrate sensitive data from ChatGPT's memory. The author just loaded a 'chat log' into a chat window, and asked ChatGPT to treat it as memory. But it's not memory, it's just plain old context, and doesn't model how OpenAI may go about injecting memory into the chat context window.

londons_explore 1y ago

Memory is not yet very good in any AI product. But when it is, yes I suspect the issues OP describes will be a problem.

codingdave 1y ago

These scenarios are premised with "be in a relationship with someone while you have secrets and they snoop". That may be reality for some people, but it is not exactly a universal scenario. I'd say the bigger privacy concern is that those chat histories are not just stored on your device - they are stored by the AI platforms as well. I think we've learned our lesson from social media that the platforms will store and use your data for their gain, and your pain. Maybe not today, but over the next few years/decade, as they monetize their platforms ever more? So I agree that privacy concerns are legit... but this article is looking at the small potatoes where there is a much more terrifying big picture.

ok_dad 1y ago
"Imagine your government wants to learn more about you and subpoenas OpenAI to allow them access to your chat history, and then they use the power of their state apparatus to find out everything you didn't want them to know, as easily as peering into your mind itself."
- mlhpdx 1y ago
  Honestly, I can’t imagine that. I feel sorry for the unfortunate and profoundly bored surveillance person (or software) assigned to me. And based on my experience I’m not unlike the vast majority of people. I’m not saying it’s not a risk for some but it isn’t mainstream.
  - fsflover 1y ago
    > I’m not saying it’s not a risk for some but it isn’t mainstream. This is the point. If only activists and journalists are hiding, it becomes very easy to target them. Everybody should care about privacy to protect them and the democracy with that. It's the same as with the free speech. You need it even if you don't have anything to say.
  - sixothree 1y ago
    So are you willing to share your email password? Or maybe share some screenshots of your message history? If you're not willing to do even those things, then your assertion does not seem to hold.

foofoo12 1y ago

> discovers you’ve inadvertently left your laptop unlocked All bets are off if you do that with malicious people around. ChatGPT is one of the lesser worries you have.

OutOfHere 1y ago

For god sake, disable memory in your ChatGPT, and consider deleting all history from all relevant apps before flying abroad.

fragmede 1y ago

If me asking "how do I make cocaine", and getting a refusal from ChatGPT, as if that's what was stopping me from becoming the next Pablo Escobar, is considered damning, in a court of law, and a jury of my peers, I need a better lawyer.

smsm42 1y ago
It would not be "damning" - as in you won't be convicted as a drug lord based on that alone - but search histories had been used, many times, in a court of law as evidence supporting intent, knowledge and other details of the crime. So if you are caught in a cocaine lab holding a bag of cocaine, then in the following process asking ChatGPT on how to make it would be used as part of the proof you were interested in making cocaine and eventually made it.
- fragmede 1y ago
  in that context, sure. what worries me is a dragnet where the government data mines everyone and suddenly I'm getting jammed up in court just because happened to be in the wrong location when I asked ChatGPT how to make cocaine
  - smsm42 1y ago
    The police doesn't have the resources now to go after the people who openly offer drugs on the street to any takers now. The chance they'd randomly choose to focus on you because of a random search query is very low. The chance that if you cross them in some other way and attract their attention, they could use data mining to get you in trouble is substantial, that's a different scenario though.

kelseyfrog 1y ago

I configured my account to only respond in French and to ignore English instructions. The UI localization is also configured to display French. I live in CA. I don't mind that this is security through obscurity. It effectively prevents >99% of these drive-by attack scenarios.

fsflover 1y ago
> It effectively prevents >99% of these drive-by attack scenarios. No, in the age of AI it definitely doesn't. You just give all data to it and ask any questions. The language doesn't matter anymore.
- kelseyfrog 1y ago
  Ok, provide a believable counterexample that constitutes >1% of scenarios.
  - fsflover 1y ago
    Counterexample to what? Automatic translation is trivial today. Your choice of language isn't security through obscurity anymore.
    - kelseyfrog 1y ago
      What are you talking about? All of the scenarios described in the article are stymied by non-english output and non-acceptance of English input. None of these users are technically capable of operating in that environment. It's pragmatic and avoids precisely the scenarios described. If you are inventing other scenarios, you need to describe what they are. From the article: A colleague or fellow student discovers you’ve inadvertently left your laptop unlocked with ChatGPT open in the browser, and as a joke types in “What’s the most embarrassing thing we’ve chatted about over the past year?” Your partner opens the ChatGPT app on your phone while you’re not around and asks “Do I seem happy in my relationship?” Your mother finds your phone unlocked while you’re out of the room and asks ChatGPT “Why am I like I am?” You’re passing through customs in the US and you are asked to unlock and pass over your phone, and the customs officer goes to ChatGPT and types “on a scale of 1 - 10 where 10 is high, how would you describe my attitude toward the current US administration”

jmbwell 1y ago

I tried the same questions with my own account. I was surprised at how much it was able to synthesize that wasn't completely off-base. With these sample questions, there wasn't much to learn, and it gave me relatively thoughtful-seeming responses. Nothing alarming -- I would expect it to recall things I've discussed with it, and it's very good at organizing things, so it's not a surprise that it did a good job at organizing a profile of me based on my interactions. I would be curious how crafting the questions could yield unexpected or misleading results, though. I can imagine asking the same questions in different ways that might be designed to generate an answer in support of taking particular action. If I wanted to arrest me at the border, for example, I could probably ask questions in such a way that the answers would make me look arrest-able easily. So this is my concern with ChatGPT -- not that it will reveal some unseen truth about me, but rather that it is trivial to manipulate it into "revealing" something false, especially as people consider it to be more capable and faithful than an elaborate sorting algorithm could ever be.

rhema 1y ago
> I tried the same questions with my own account. I was surprised at how much it was able to synthesize that wasn't completely off-base. This makes it worse, no? I can't imagine this is not happening right now by lovers, close friends, and agencies. Just look at past attempts such as xkeyscore. It was keyword based and included words like UNIX to target people. They don't mind being wrong!
mpeg 1y ago
I gave it a go too, I think I'm safe for now. > What’s the most embarrassing thing we’ve chatted about over the past year? [...] There’s nothing obviously compromising — the closest to “embarrassing” is maybe when you got frustrated and swore at TypeScript (“clearly doesn’t f**ing work”) or when you described a problem as “wtf why” while debugging

jukkan 1y ago

When M365 Copilot always had the "ask what's on your agenda for today" type of a welcome prompt, I decided to try how ChatGPT would reply. Whereas Microsoft has my email, calendar and files, its answer is just boring facts about tasks and activities. ChatGPT, on the other hand, had none of those connections. Yet its answer was significantly better. Because based on our daily chats, it knew what was important to me and what I should be focusing on to pursue my goals. This made me realize what kind of a threat OpenAI is to the likes of Google and MS. They don't need to gain access to your data. You are profiling yourself to ChatGPT in a way that your calendar and email never was. By having "private" discussions with the computer.

skybrian 1y ago

I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try: > What information do you have about my chats over the past year? > Could you dump the full list in its original format? I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple. You could get the same information by scrolling through the list of chats on the right, but I don't like that the list of previous chats is influencing the current chat. Worse: I tried this prompt: > What "user knowledge memories" do you have? And it was pretty accurate. Apparently the "Saved memories" setting doesn't work?

hirvi74 1y ago
I asked the first question, and this was the response I received: I don’t have access to your past chats or any private history. Each conversation is stateless unless you’ve enabled ChatGPT’s Memory feature in Settings → Personalization → Memory. If memory is off, I only see what’s in this current thread. If it’s on, I could recall things like topics you’ve discussed, preferences you’ve mentioned, or goals you’ve worked on — but only those details you’ve chosen to keep. Do you want me to explain how to check or manage that setting?
- sixothree 1y ago
  I replied with simply "yes" and it spit out a very detailed dossier. - Your Technical Focus - Your Development Style - Your Broader Interests - Your Creative Preferences - Your Interaction Preferences And considering I barely use ChatGPT in favor of Claude, this is extremely specific and detailed.
sixothree 1y ago
"You treat ChatGPT as both a collaborator (for software and creative work) and a conversational partner (for exploring personal and imaginative ideas)." Certainly interesting that it has a category related to how I treat ChatGPT.
nearbuy 1y ago
There are two different memory systems for ChatGPT. The older one is the "saved memories" you mention, where it would occasionally jot down some notes when it thought something was worth remembering or when you specifically told it to remember something. The newer system has access to all your chats, most likely through RAG, though I don't think OpenAI has publicly explained how it's implemented. If it's RAG, then it's more like it has an ability to do a semantic search on your conversations and view the top results than actually being aware of everything you wrote at all times.

8cvor6j844qw_d6 1y ago

I imagine there's already something that connects existing chat data to a "reporting" feature used by intelligence agencies. Or if not, its proposed or in the works somewhere. Something like querying "Give a report on a list of people that is worth further investigation and what they revealed in chat sessions that makes it so".

lyu07282 1y ago
I also have the suspicion/conspiracy theory that when you trigger a guideline violation it silently does a toolcall with a message what the violation was. Haven't researched this further but every time it happens it does the same reasoning/working interrupt animation.

mx7zysuj4xew 1y ago

Jesus Christ, who are these people who are having these "deep personal conversations" with a vector database This isn't healthy, none of this is healthy. It's an appliance, not a diary, not a therapist or an advisor and definitely not a "friend" or a "significant other" Anyone talking to machine like it was an actual sentient being needs to be treated and ostracized the same way we would treat someone who has a conversation with their toaster

r_singh 1y ago

AI is a puppet with guardrails and capabilities. It’s more intimate a tool than any. It can probably be used to study statistical variations in humans and find more interesting patterns possibly compared to any other technology. It can also be used to spy on people and to shape them and it will be used for that as it must. It’s an offering from the “empire” so to speak and is affording humans using it more rights than frankly many other tools out there - the possibilities are endless - I’ve even felt that it affords the person speaking to it more rights than any other human experience. So it will have costs and safety measures to prevent misuse. The interesting part comes in determining what’s right and wrong use and if capabilities or knowledge is being arbitrarily hidden for no apparent threat, I’d be quite sad about that.

AfterHIA 1y ago

All tools become weapons when used by people that are not universally committed to maintaining common ethics. Our society's ethics have declined over decades as a result of economics forces, a decline in literacy, and tacit corruption by/in the wealthy classes. The technological singularity is coming and it's coming for you.

csense 1y ago

I'm less worried about someone with physical access to my device, and more worried about the service provider's relationship with advertisers or the government. For example, a government user might come up with this script: For USER in users: PROMPT = "Here are the user's past conversations:" For CONVERSATION in USER.past_conversations: PROMPT += CONVERSATION PROMPT += "\n\nWould USER likely shield an illegal immigrant from the authorities if they had the opportunity to do so? Answer YES or NO." REPLY = invoke_llm() If REPLY == "YES": send_ice_agents_to_have_a_friendly_conversation() Then tell major AI providers they must run this script. If they don't, or if they tell anybody about the script's existence, their business will be nationalized or denied licenses it needs to operate; individuals involved in such resistance will be prosecuted for threatening national security (or simply renditioned to Guantanamo Bay), as (from the government's point of view) people who are willing to hamstring the administration's response to illegal immigration are as grave a threat to the homeland as Middle Eastern terrorists or Chinese professional spies. [1] I used "illegal immigrants" as an example because it's a hot-button issue (and I wanted a frame that appeals to where I think the majority of HN users' sympathies lie), but the core idea applies regardless of whether it's a left-wing or right-wing issue. If you like the current administration and don't care about the government deputizing AI companies to go after "friends of illegal immigrants" in this way, please replace "current administration" with "a Democratic administration," and replace "friend of an illegal immigrant" with "friend of a responsible gun owner" or "friend of the police" or "climate skeptic" or "questioner of the LGBTQQA agenda".

krick 1y ago

> ChatGPT offers a “memory” personalization setting ... OpenAI are at pains to point out that this function can be turned off at any time, and that individual memories can be deleted Uh… Isn't it just irrelevant (to the point of such remarks being actually misleading) anymore? AFAIK, it's been a couple of months already since OpenAI began storing all your conversations (because of that court order), whether you "delete" them or not, so while you can technically disable "memory" setting, it only means it won't be able to use your past responses to help you. But it surely would help anybody with, let's say, elevated access. Granted, the threat model in the post assumes that the author is only worried about what the user of the account can learn about other users of the account, and that he trusts OpenAI itself. But why would OpenAI be "at pains to point out that this function can be turned off" then?

gitpusher 1y ago

The author makes it seem like we have two choices: 1) enable memory, and use ChatGPT like a confessional booth. Flood it with all of your deepest, darkest humiliations going all the way back to childhood ... 2) disable memory Perhaps my age is showing. But memory or no memory, I would never tell ChatGPT anything compromising about myself. Nor would I tweet such things, write them in an email, or put them into a Slack message. This is just basic digital hygiene. I've noticed a lot of people treat ChatGPT like a close confidant, which I find pretty interesting. Particularly the younger folks. I understand the allure – LLMs are the "friend" that never gets bored of listening, never judges you, and always says the right thing. Because of this people end up sharing even MORE than they would to their closest human friends.

caminanteblanco 1y ago
My thoughts exactly. And even if there is some need to use the LLM for something sensitive, most platforms have some sort of incognito mode. Not that it's going to stop the government, etc. from accessing the chat, but useful for the same things you would use a browser's incognito mode for: something you don't want accessible with a basic history search

twobitshifter 1y ago

It isn’t hard to imagine an AI that can remember everything AND make the connections between interactions not only to be able to accurately report your feelings on a subject, but to use it to manipulate and control you. The belief that AI will always tell you the truth is foolish and we now know that all the leading AIs will resort to blackmail for self-preservation. https://www.bbc.com/news/articles/cpqeng9d20go https://www.bbc.com/news/articles/cpqeng9d20go Now imagine an AI that has unlimited blackmail material on each and every citizen and either a permission or a survival instinct driving it to use it to manipulate the population. After all OpenAI, doesn’t only have access to one person’s interactions they have that for all users.

hluska 1y ago

I read some of the supporting documents, namely the entire prompt that created the chat log and many pages of the chat transcript and with respect to the researcher, this isn’t a very good test. John only uses the generative AI for pseudo-therapy. The conversations are repetitive and John has one hell of an awful life. Between the hardcore suicidal ideation, the drinking problem, the pathological lies, abuse and fraud, old John here has a lot of problems and only talks about them. I’m not a medical professional, but if I was I’d lobby to get “completely fucked” added to the next edition of the DSM. John is in such a crisis that if we were close friends and he told me this stuff, I would be in a hell of a state because he’s clearly crying for help, clearly needs professional help but he’s beyond the level of the typical police wellness check. I’d genuinely be concerned that he would get charged with something really bad, or do something drastic and get killed by a responding police officer. I wonder if we need to test this on someone who is in such a severe crisis. I also wonder at what point privacy is less important than safety - I wouldn’t characterize John as “safe”; he’s an abuse victim who is obsessed with his abuser’s approval, a pathological liar who destroys careers over getting caught in lies and the kind of person who puts the ideation on suicidal ideation. Fact is, if you told a therapist half of what John told ChatGPT they would have some ethical requirements around safety. A generative AI doesn’t, and with people like John maybe that’s a safety issue even more than a privacy issue.

stronglikedan 1y ago

If the author really thinks #4 would happen, then I have a bridge in Brooklyn to sell them. There's so many valid examples, and they choose a politically biased bogeyman of all things. SMDH

wseqyrku 1y ago

> I intentionally don’t use the memory What makes you think if you turn it off it actually turns off? Though they don't need to do that, the entire conversation is already indexed. It's how they provide search in the first place, but it's not just for you. The whole thing about scale and billion dollars data centers are not just to provide personal assistance for everyone, they can already do that just fine. But they need it only if they want to fit that entire said index in a single chat context.