9 ms·
A Chinese official’s use of ChatGPT revealed an intimidation operation
- upupupandaway 8mo agoI was in Shanghai recently and while casually testing one of their AI chat bots I typed "What do you think of the situation in Taiwan?". It started discussing like a Western bot would - "it's complicated, etc. etc." and around 5s it abruptly stopped and regurgitated the same line the CCP uses "... it's an unalienable part of China etc. etc.". After printing the line, a popup opened and my camera was activated. The app wanted me to submit my information, presumably to decide what to do with me next time I enter China. 1) All the lights and modern buildings cannot hide that China is a creepy authoritarian state underneath. 2) Given the bot started printing the Western consensus first, I bet $10 it was trained by distilling ChatGPT or Gemini.
- raven12345 8mo agoCan you tell what AI chat bots are you using? as i know all chat in China just block answer, no apps will activate camera and ask for information
- titaniumtown 8mo ago> After printing the line, a popup opened and my camera was activated. The app wanted me to submit my information, presumably to decide what to do with me next time I enter China. Was this on your personal device? I'm just wondering how it activated your camera. I would love more details!
- 9999px 8mo ago[flagged]
- kelseyfrog 8mo ago[flagged]
- parl_match 8mo agoAn increasing use of AI is to gather user feedback. The Chatbot UI detected an error state, and then loaded a feedback vendor, who then popped the camera open for their interactive feedback session I've run into this a few times, now. So what OP is saying is plausible, I just don't appreciate their added and probably incorrect conclusion that it's because the government of China wants to do something to them
- fasbiner 8mo agoWhat are you talking about? Why are you using imprecise language like "popped the camera open?" You've run into a site you view on chrome/firefox/safari accessing your camera without granting access a few times now? Can you give us an example of a site that does this so we can reproduce? Or could you retract your statement and clarify that you did grant camera permissions for that site previously? Otherwise, you're saying very casually there's a huge bug and security issue that no one else has detected but you personally have seem multiple times. I've run into people on the internet misremembering things or not understanding how the browser works more times than I've run into browsers allowing access to system devices like the camera without a permission prompt.
- parl_match 8mo ago> Or could you retract your statement and clarify that you did grant camera permissions for that site previously? I never said anything about granting permissions. I can respond to your other points, in turn, but first I would like you to confirm that I am who you think you are responding to :) I am not OP. In case you don't think I'm OP, then, well I was being imprecise. Yes, it requires browser/app/manifest permissions. Your paranoid and aggressive tone implies you're not giving me any benefit of the doubt, as I speak informally in a casual web forum discussion about understanding what happened.
- fasbiner 7mo agoSorry, I think of HN as a technical forum where people prize precision. I think that when you informally bolster lies of omission with your own imprecise language agreeing with it, that harms the quality of discussion and creates pointless confusion. I think you get less benefit of the doubt if you do that routinely. By your own admission you do that routinely since you use this as a place to speak informally in a casual web discussion where you are often imprecise or lie by omission.
- anvuong 8mo agoYeah that part is either just bullshit or OP gave the bot access to his camera previously, which is just dumb.
- sixseven 8mo agoThe story smells like bullshit to me.
- crossroadsguy 8mo agoOP might have meant that it asked for camera permission. Or OP might have set camera permission to auto allow but then that is also stupid (more so).
- Justitia 8mo agoI would not be surprised if they just activated the camera
- 3rodents 8mo agoIf this were true, why didn’t the chatbot immediately recognize that the word “Taiwan” should trigger the response? Detecting the word “Taiwan” has been possible since before most of us were born. China has more restrictions on what you can say than the U.S. but what you are describing is not reality. Some westerner asking Deepseek about Taiwan is completely uninteresting. Just as the government do not chase people over VPN usage. China doesn’t try to hide that they are an authoritarian state. They don’t need to. Most people in China are no less happy with their government than westerners are with their governments. Governments reflect culture. And as for foreigners, our view of China is far worse than it actually is, China doesn’t need to hide anything, people who visit China will come away with a more positive view of the country than those who do not visit.
- deleted 8mo ago[deleted]
- CWuestefeld 8mo agoThis is manifestly false. My wife grew up in Shanghai, and you'll have to go quite some distance to find someone more critical of the PRC and CCP than she is. And it's with good reason. She grew up during the cultural revolution, and was largely raised by her grandmother because literally every other person in her extended family was in prison or work camp, not because of anything they had actually done wrong, but for political reasons because the whole family was blacklisted. And that's not just the old days. Her father died as a direct result of Chinese Covid policy. During the pandemic her cousins still in the country would ask her (on Skype) "is X true?", and largely their perception of what was going on was false. She would exfiltrate encrypted news reports to them - until those started getting blocked. Her dad's estate still has affairs that need to be resolved, but we've decided not to return to China until Xi is gone, as it's just not safe. It doesn't get much airplay, but there are currently a couple of hundred Americans who are being illegally detained in China right now. It's not worth the risk. My first trip to China was about 30 years ago, shortly after we got married. And back then, I would have said that you were right. Honestly, it felt like for the average person in their day-to-day-lives, the Chinese were less under the governmental thumb than we are. People from the countryside would bring their produce into the city to sell, or cook dumplings and buns to sell on the side of the street - stuff that in America we'd have to get permits for. It seemed that the oligarchy had an understanding with the people: let us control the big picture, and we'll look the other way for the little things. But Chinese politics is a pendulum swinging very widely. From Tienanmen Square and Tank Man, it had swung quite a bit the other way. But today, it's come back 180-degrees. Xi is really trying for a Cultural Revolution 2.0. These impressions largely match what I hear from other Chinese immigrants - except for Party members, who tend not to want to talk about it at all. I'm afraid that you've been listening to too much propaganda.
- hmmmmmmmmmmmmmm 8mo ago[flagged]
- chausen 8mo ago[flagged]
- jajuuka 8mo agoI love Hacker News fiction. Wild stuff. haha
- imwillofficial 8mo agoThis risk is far overstated. I was talking crap about china from the great wall.
- expedition32 8mo agoYou can't yell Free Palestine or the BBC will mute you... Personally as a Dutch person it is amusing as all hell hoe goddamn triggered everyone gets about Israel- truly mindblowing.
- carabiner 8mo ago[flagged]
- deaux 8mo agoDid everyone clap and Albert Einstein hand you a crisp $10 bill? You should use that to make the bet you mentioned! (The first half is obviously true, the second part isn't)
- holoduke 8mo agoWhich chat bot?
- heavyset_go 8mo agoI ran an anonymized Facebook account for years with thousands of followers that mainly sticks to news and politics. Once I started criticizing Libs of TikTok, the propaganda arm for this administration, and getting traction with users, my account was locked and now I have to scan my face and ID if I want to use it again. You have to toe the party line here, too.
- ZeroAurora 8mo agoNice bad story. Make up one better next time.
- stevenhuang 7mo agoThat you think this is made up shows how little you know about what the Chinese netizen deals with day to day. It is very real and I am not surprised at all something exactly like what op said has happened.
- ZeroAurora 7mo agoWhat if I say I AM a Chinese netizen, right here in China Mainland talking to you? What if I say no application on my phone ever turn my camera on without my prior approval? What if I confidently say the data privacy situation in China is not in any way worse than USA? You say I'm censored by gov. Yeah, and so do YOU. We are quite the same, so don't laugh at each other.
- stevenhuang 7mo agoIf you ever allowed the app access to your camera then next time it will be able to access your camera. No one is suggesting the app need have circumvented standard android permissions. I'm saying it is not surprising the app would try to open the camera on violation of rules. You might be surprised to know that I do not disagree with what you said. When it comes to data privacy your information is safe from other private companies but not the government. > You say I'm censored by gov. When it comes to political inquiry? Without question this is more sensitive in China. My point is that in China apps are coerced to reveal information about their users to a degree where actively trying to take a picture would not be surprising, or much of an escalation. Not that it would be needed anyways since Internet usage is tied to ID.
- glenstein 8mo ago>Given the bot started printing the Western consensus first, I bet $10 it was trained by distilling ChatGPT or Gemini. To your point I've seen something similar with Deepseek, generic answers start printing and then, in plain sight, removed and replaced with a non committal message along the lines of "I don't have access to that information."
- mrdependable 8mo agoWow, our surveillance helped take down their surveillance. Yay, I guess?
- AceJohnny2 8mo ago"Our glorious oversight vs their barbaric surveillance" (I kid, mostly. While the US certainly isn't pure, its scale of surveillance intrusion is light compared to China)
- pcthrowaway 8mo ago> While the US certainly isn't pure, its scale of surveillance intrusion is light compared to China I assume that for someone to believe this, they either have to believe the U.S. has poorer surveillance capability than China, or, more likely, they consider U.S. surveillance unintrusive and Chinese surveillance intrusive.
- kvuj 8mo ago> ... or, more likely, they consider U.S. surveillance unintrusive and Chinese surveillance intrusive. Of course. What's the point of surveillance if you're not going to use it to enforce dogma? I think you can reasonably evaluate a country's surveillance by looking at the pettiness of the arrests & censorship they make. See this chinese tech reviewer[1] being bullied by the government for putting a spotlight on chinese phone makers cheating about benchmarks. I'm not sure the US is at this point yet... [1] https://old.reddit.com/r/hardware/comments/1rfw6oj/hardware_reviewer_geekerwan_possibly_censored_by/ https://old.reddit.com/r/hardware/comments/1rfw6oj/hardware_...
- deaux 8mo ago> See this chinese tech reviewer[1] being bullied by the government for putting a spotlight on chinese phone makers cheating about benchmarks. I'm not sure the US is at this point yet... Plenty of signs point towards this being a case where it wasn't the government, so it's not a good example. Instead it looks to have been the tech companies filing takedowns and threatening lawsuits. Especially when it comes to China, this is a big difference. You should've pointed towards more clear-cut examples like Naomi Wu and Peng Shuai. But those cases are less unthinkable in the US, and it should be uncontroversial to say that it's what's been worked towards.
- dddddaviddddd 8mo agoMore interesting than the fact that ChatGPT was used, was seeing all the specific examples of the types of work that this individual was doing.
- DANmode 8mo agoWhere can they be found?
- layer8 8mo agoI wonder what exactly the trigger conditions are that lead to the chats of an account being human-reviewed by OpenAI.
- simlevesque 8mo agoI'm pretty sure they can just prompt any convo in the background and ask "is this conversation sensitive ?" and the model can answer without this being added to the context of the convo.
- spwa4 8mo ago"Is this someone important enough to spy on?" One hopes the CIA/Secret service would be willing to provide the human to do the reviewing but sadly I've worked for European telco's and I know better.
- dlev_pika 8mo agoSounds like Anthropic is fighting this exact battle, and DOD is arguing they don’t want to do that lol
- rrr_oh_man 8mo ago> sadly I've worked for European telco's and I know better Can you elaborate?
- LightBug1 8mo agoThis is -the- question.
- coliveira 8mo agoSo, it seems they're openly admitting that OpenAI is a surveillance mechanism used at the discretion of the US gov.
- morkalork 8mo agoSuch a hot topic these days: https://www.cbc.ca/radio/frontburner/chatgpt-and-the-tumbler-ridge-shooter-9.7106186 https://www.cbc.ca/radio/frontburner/chatgpt-and-the-tumbler...
- layer8 8mo agoThis is the report on which the CNN article is based (which it doesn’t link to): https://cdn.openai.com/pdf/df438d70-e3fe-4a6c-a403-ff632def8f79/disrupting-malicious-uses-of-ai.pdf https://cdn.openai.com/pdf/df438d70-e3fe-4a6c-a403-ff632def8...
- gitpusher 8mo ago> “This is what Chinese modern transnational repression looks like,” Ben Nimmo, principal investigator at OpenAI, told reporters ahead of the report’s release. “It’s not just digital. It’s not just about trolling. It’s industrialized. [...] There's something poetic about OpenAI being asked to comment on mis-use of their slop generator, and their answer is composed entirely of AI slop.
- zdragnar 8mo agoThe more of it they and others put out, the more normalized and acceptable it becomes. The next generations will even think in slop.
- direwolf20 7mo agoskibidi
- 2OEH8eoCRo0 8mo ago> “It’s not just digital. It’s not just about trolling. It’s industrialized. It’s about trying to hit critics of the CCP [Chinese Communist Party] with everything, everywhere, all at once.”
- CrzyLngPwd 8mo agoPushing aside the fact that OpenAI is just a tool of the US regime. Will OpenAI release the same for other government officials from any other states? I can't wait to see Starmer's chats with ChatGPT. Anyway, all of this smells like 1934, "accusing them of what we are already doing"
- ImPostingOnHN 8mo agothere are multiple states mentioned in the report, so yes
- hungryhobo 8mo agoMostly non aligned countries with the US
- deleted 8mo ago[deleted]
- direwolf20 8mo agoFor them, not of them. ChatGPT releases information on the US's enemies to the US. Will it release information on France's enemies to France?
- ImPostingOnHN 8mo agoWhy do you ask?
- guelo 8mo agoI'm assuming they would not disclose such campaigns by the US government. I can't imagine the amount of government secrets, trade secrets, business plans, personal secrets, etc that people divulge on there.
- kykat 8mo agoThe amount of information about everything that people are giving OpenAI is astronomical, information that was previously kept closely guarded is now just freely flowing through foreign servers. Truly a paradise for american intelligence. Would have expected that the chinese officials be briefed on not using us tech companies, but opsec is hard to teach, and even harder to always follow.
- simmerup 8mo agoBut the american silicon valley nerds pinky swear not to look! How can you not trust them.
- bucketdeveloper 8mo agoDid they though? I never got to the end of the Terms & Conditions myself.
- direwolf20 8mo agoThe T&C's promise they will look.
- dlev_pika 8mo agoCrazy to me that Chinese officials use ChatGPT to discuss sensitive operations lmao
- tehjoker 8mo agoi kinda get the impression this was from 2023 and also it is not clear what this dissident did, hard to evaluate whether i should care without knowing that
- romulussilvia 8mo agoI remember a while back when a few cars with CCP decals driving around SoCal to intimidate some dissidents!
- leonflexo 8mo ago> Chinese operators allegedly disguised themselves as US immigration officials
- lxgr 8mo agoThis seems to be the source report: https://openai.com/index/disrupting-malicious-ai-uses/ https://openai.com/index/disrupting-malicious-ai-uses/ (since it would of course kill CNN, like almost all media outlets, to link to a non-affiliated primary source...) Does this level of detail seem strange to anybody else? Shining such a strong light on OpenAI's moderation/manual review efforts seems like it would draw unwanted attention to the fact that ChatGPT conversations are anything but private, and seems somewhat at odds with their recent outrage about the subpoena for user chats in the NYT case. Manual reviews of sensitive data are ok as long as their own employees are the reviewers, I suppose?
- coliveira 8mo agoYes, it is either a lie or an admission that OpenAI is a global surveillance mechanism.
- andai 8mo agoAlas! My vision of One Fed Per Child hath come to pass!
- jajuuka 8mo agoThis feels very planted. Wouldn't be surprised if this some attempt to look patriotic with the DoW turning up the heat against Anthropic.
- erhuve 8mo agogiven that OpenAI just nabbed the contract, I'd say that's spot on
- ticulatedspline 8mo agothat creepy feeling of "being watched" has mostly kept me from taking advantage of any SOTA models, i only dabble in a few local ones. The level of detail does not seem surprising. they're both charged with maintaining a facade of privacy while eliminating any and all miss-use. Certainly they heavily analyze basically everything given to them. And generally as a society we've been ok with basically zero privacy as long as the data we send stays inside the company we sent it too. Google reads all your emails? Sure thing, read away, just don't send them to the popo. Apple knows when you're ovulating? no problem, just don't tell Amazon. etc
- cc-d 8mo ago[flagged]
- zoklet-enjoyer 8mo agoVery creepy on the part of Open AI. Glad I don't use chatgpt
- _ache_ 8mo agoIn France we have a report on Chinese officials abusing diplomatic rights to oppress Chinese critics (from Chineses expatriated people). The disproportion between how this people express they opposition and how Chinese officials track them is HUGE. This very much feel unnecessary. It was here: https://www.france.tv/france-2/envoye-special/5971095-la-chine-fait-elle-la-police-en-france.html https://www.france.tv/france-2/envoye-special/5971095-la-chi... https://www.youtube.com/watch?v=k-prS7BlLpI https://www.youtube.com/watch?v=k-prS7BlLpI
- gigel82 8mo agoHoly dystopian f*k. So not only does ChatGPT record all interactions, it actually leaks them to the press when they see fit? If you still needed a reason to look into self hosted models, it'd be tough to find a better one than this.
- newzino 8mo ago[flagged]
- irenetusuq 8mo ago[dead]
- ipcress_file 8mo agoWhy did they ban the user rather than informing American intelligence and continuing to monitor the user? They just gave up a source that could have provided info for years.
- plagiarist 8mo agoIf I were doing this sort of thing, I would make certain to ban accounts that were too obvious while leaving ones that are subtle enough, so that the other side has less reason to suspect I am tracking their inputs and feeding them disinformation.
- ipcress_file 8mo agoGood point. Perhaps that's what's happened here.
- est 8mo agoI think one of the reasons why AI companies are valued this high is one can actually inspect what user inputs & outputs are. It's basically an OSINT siphon. In this Chinese case, the tokens are leaked at least twice. ChatGPT offers no direct access to the Chinese, they have to use some kind of Openrouter-like service, but the data where also in clear-text during transmission.
- nameconflicts 8mo agoChina has hundreds of Fortune Global 500 companies and ranks second in GDP. But these have nothing to do with ordinary people.
- csense 8mo ago> intimidating Chinese dissidents abroad, including by impersonating US immigration officials I hope those victims of immigration impersonation don't have family within China's borders. AI-enabled impersonation and intimidation are far from the worst of China's crimes [1] against its overseas critics. China likes to make you an offer you can't refuse [2] [3]: You're saying stuff the Chinese government doesn't like, but you live outside its borders and the secret police can't get at you? You need to come to China and be jailed (or worse). If you don't, your family will be the ones who are jailed (or worse). Or you can unalive yourself, and save the glorious Chinese Communist Party the expense of a bullet. [1] China would say "the government punishes a criminal's family" is not a crime, it's a perfectly legal implementation of government policy under Chinese law. I respond that the death camps were perfectly legal implementation of government policy under Nazi law, but were still crimes against humanity -- China's actions fall in this category of crimes. As I understand it: Western societies have a very individualistic view of responsibility. If you didn't commit a crime, you're innocent. Punishing the innocent family members of a criminal is morally abominable. In the Chinese Communist Party's view, criminal responsibility is collectivist. By their definition, the family members of a criminal share responsibility for the crime regardless of their participation in the criminal acts. "Innocent family members of a criminal" is a logically inconsistent concept in their world view. The family of a criminal is guilty by definition -- being related to a criminal is itself a crime. This is sickening to me. [2] https://en.wikipedia.org/wiki/Operation_Fox_Hunt https://en.wikipedia.org/wiki/Operation_Fox_Hunt [3] https://www.abc.net.au/news/2020-07-08/fbi-chief-says-china-uses-threats-to-coerce-overseas-critics/12433208 https://www.abc.net.au/news/2020-07-08/fbi-chief-says-china-...
- chenzhekl 8mo agoThis tells us that we should never share sensitive information with GPT, even if you’ve set it not to use your data for training. Nothing can stop OpenAI from misusing your data.
- titaniumrain 8mo agolol everyone claims deepseek and all chinese companies are collecting private information and ban them in western companies. but it is okay being spied by openai :)))
- edg5000 8mo agoI like DeepSeek because of their pricing, although I'm still evaluating. I wonder if I'll need a VPN in the future to access it though (from EU). Cheap is good, cheap prevails. The official DeepSeek API is routed through AWS load balancing btw.
- RobotToaster 8mo agoWell someone's getting fired, hopefully not literally.
- geon 7mo ago> The Chinese law enforcement official used ChatGPT like a diary But why? It’s like using photoshop for spreadsheets.