9 ms·
Microsoft is spying on users of its AI tools
- bearjaws 3y agoEZ rage bait. Everyone of these platforms is scraping your inputs to train their next model. How is it not obvious they would also detect abuse?
- throwaway22032 3y agoWhen you send data to someone else they have that data, act accordingly. Anything else is just stupid. Didn't we all learn on the playground that a secret isn't secret any more once you tell a friend?
- DinoCoder99 3y ago[flagged]
- poszlem 3y agoThe illusion of a borderless world has faded. The comfortable notion of "global citizenship" was a fleeting artifact of American dominance. As new realities emerge, the vital concept of state security demands far greater vigilance than in decades past. This is not jingoism, it's a clear-eyed assessment of our world. In this new reality, we can no longer afford the naivete of sharing our most valuable assets with nations who oppose us.
- int_19h 3y agoYet corporations still embrace this illusion when it comes to, say, operating in those nations. It's a borderless world for them, just not for us.
- 15457345234 3y agoWarmongers will never cease to monger war, I guess. What you've written is massively jingoistic; yes certain forces seem oddly desperate to talk a war into existence - I guess all those solar panels coming out of CN are disrupting certain money flows on a permanent-looking basis - but the absence of any actual, idk, military buildup will be harder to explain away. Unless you just deepfake the war. Which is probably what will happen.
- DinoCoder99 3y agoWhy are you conflating the concept of "nation" with that of a "state"? These are two words with different meanings. This just strikes me as meaningless friction to cause domestic arms production.
- rhdunn 3y agoYou can geolocate a user/request based on the IP address. This is common in servers. It allows a server to know which country a request is coming from. I suspect that this is what OpenAI are doing, as Bing Translate and Google Translate do similar things.
- deleted 3y ago[deleted]
- BlueTemplar 3y ago2nd : Because Microsoft is still part of PRISM ? https://en.wikipedia.org/wiki/PRISM https://en.wikipedia.org/wiki/PRISM P.S.: As a reminder, metadata is typically more informative than data.
- jsnell 3y agoOriginal discussion: https://news.ycombinator.com/item?id=39368859 https://news.ycombinator.com/item?id=39368859
- sandworm101 3y ago>> The only way Microsoft or OpenAI would know this would be to spy on chatbot sessions. I’m sure the terms of service—if I bothered to read them—gives them that permission. And of course it’s no surprise that Microsoft and OpenAI (and, presumably, everyone else) are spying on our usage of AI, but this confirms it. What? The chatbot is not your therapist. The chatbot is Microsoft. They are the same entity. Everything shared with the chatbot is shared with microsoft, just as every google search is shared with google. Related legal determination re chatbots: https://www.washingtonpost.com/travel/2024/02/18/air-canada-airline-chatbot-ruling/ https://www.washingtonpost.com/travel/2024/02/18/air-canada-...
- deleted 3y ago[deleted]
- Zelphyr 3y agoI don't trust Microsoft as far as I can throw them collectively but it doesn't come as a surprise to me that they are doing this any more than it would if I were to hear that OpenAI, Google, etc... are engaging in this behavior. That's not to excuse it, however. I think it's already time they all implemented a privacy stance where they are only allowed to view chat conversations if the user gives them explicit and revokable permission.
- sammyteee 3y agoIn other news, water is wet. All joking aside, and even as the article points out re: the T+Cs, is anybody surprised?
- michaelt 3y agoMany cloud providers maintain the pretence that their employees don't have access to customer data. For example, they like to maintain that if you run an amazon competitor hosted on AWS, amazon insiders can't look at sales numbers in your hosted database. Or if you run a google competitor with e-mail hosted on gmail, that google insiders can't view your e-mails. And so on. Personally I've always found these promises rather hard to believe, but a lot of people have a great deal of trust in them.
- deleted 3y ago[deleted]
- nullindividual 3y agoI worked for a large cloud provider and that is correct, employees did not have access to customer data. The encryption key in this product is split between a SQL database and the front end servers with blob data being stored outside of SQL (customer data). There was no way for a single employee to get access to both portions of the key to decrypt customer data. Using AWS when you're a competitor of Amazon seems like poor business decision making.
- marcosdumay 3y agoWho isn't a competitor of Amazon? Or, better, who isn't a competitor and is also not at risk of becoming one?
- nullindividual 3y agoIf the answer is "everyone" is a competitor of Amazon, no one should be using AWS except for Amazon.
- surrTurr 3y agoIf you call this "spying", you should read the TOS of the OpenAI API again. If you, as a "hacker", still use their API, it's your own fault: > OpenAI may securely retain API inputs and outputs for up to 30 days to provide the services and to identify abuse. After 30 days, API inputs and outputs are removed from our systems, unless we are legally required to retain them. You can also request zero data retention (ZDR) for eligible endpoints if you have a qualifying use-case. For details on data handling, visit our Platform Docs page. Source: https://openai.com/enterprise-privacy https://openai.com/enterprise-privacy
- binarymax 3y agoAzure's TOS are different. You can also opt-out of data retention: https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy https://learn.microsoft.com/en-us/legal/cognitive-services/o...
- gpjanik 3y agoI'd assume Azure's TOS does not allow foreign secret services to use it for its operations. I remember TOS of iTunes mentioned you can't use it to build nuclear bombs. In that sense, there's no way that what Microsoft did was illegal.
- blackoil 3y agoIt also lists "30-days retention window" for Asynchronous Abuse Monitoring.
- binarymax 3y agoYes, but you can opt-out of the data retention and abuse monitoring if you go through a vetting process and are deemed low risk.
- ftkftk 3y agoThat is correct. The process is manual and definitely has humans in the loop. It is also required for every individual subscription in your account.
- api 3y agoI've seen numerous stories recently that boil down to people being surprised that cloud hosted products "spy" on you. If you're sending the data to someone else's computer and it's not encrypted, you're.... sending the data to someone else's computer. Do people not understand this?
- __MatrixMan__ 3y agoMy wife teaches English. She uses Google docs revision history to spy on her students writing (and occasionally to recover it in the event of a disaster). So if it's plagiarized, she can see the moment where they pasted the plagiarized version. They apparently have no idea. So yeah, I think that people legit don't understand. The gap between tech literate and hacker is problematically large.
- api 3y agoI increasingly feel like a member of some kind of high priest or scribe class in a feudal society, and I don't like it. I don't like where this is going. It'll either end in a dark cyberpunk total surveillance and mass manipulation state or a kind of Butlerian Jihad where people reject networked cybernetic systems en masse. Maybe the first happens followed by the second.
- __MatrixMan__ 3y agoAgreed. I think we need to be a bit more conscious of whose agenda we're willing to further with our work. I'd rather retire of unremarkable means in a functioning society than be wealthy in a dystopia of my own making.
- codeptualize 3y agoYes.. they are pretty clear that they monitor for abuse. 30 days. Not sure how this is a surprise.
- wredue 3y agoAs nothingburgers go, this is pretty tame and expected. You use a platform that entirely runs on a hosted back end, you’re going to have everything logged and saved. Compare that to, say, Nvidia, who sends every single titlebar value, as well as every single click you make to their servers (this data sends even if you opt out, as it is classified as “necessary”). Probably the most invasive spyware you have installed today.
- ambichook 3y agojfc, is this from geforce experience? if so i knew it was bad but this is something else entirely
- 23B1 3y agoYou should know that if you're interacting with an AI tool through various social channels – facebook, twitter, instagram, texting – those chats are not only being actively read by the employees of that company, but anyone who is responsible for the 'marketing' of that company, e.g. outside vendors. Certainly these firms are bound by contract, but the stuff I've seen people ask these chatbots – health information, sexual information, highly personal stuff you might only share with your shrink... well, it's being read by the 22 year old marketing intern as well. Unencrypted, in the clear, copy & paste'able, and attributable to your username/ID
- 2OEH8eoCRo0 3y agoNo shit. The internet is a network of other people's computers. Was the internet designed to be private?
- bilvar 3y agoThe neighbourhood is a network of other people's homes. Let's allow everybody in ours then.
- 2OEH8eoCRo0 3y agoWhat does the public street represent in your analogy?
- gpjanik 3y agoI love how the user made "Microsoft is spying on users of its AI tools" from "Microsoft found out that hostile governments of authoritarian countries use their AI tools to conduct illegal activities" and forgot that you legit have to click "Yes, I agree" under the terms and conditions that state they indeed do track all this stuff. Next up, "Major television broadcasters spy on soccer players during Champions League games".
- hef19898 3y agoCan I use those tools without clicking on "Yes"?
- bilekas 3y agoCan you drive your own car without a key ? You don't have a 'right' to use the services. So the answer to your question is "It depends if the service lets you, so 9/10 cases, no. Rightly so" Edit : I can see a lot of people don't like this. But like it or not, right or wrong, that is how it is. Expecting anything else is just fanciful.
- arp242 3y agoInformed consent is key. Do they clearly inform you of this when using these tools or is it hidden in a dense Privacy Policy somewhere with language so vague and woolly most people hardly understand it, which you "accepted" when you first installed Windows? I don't have a Windows machine and didn't check, but I'm guessing it's a lot closer to the second than the first. In principle I don't think anything is "wrong" with keeping a record of AI chat sessions, but you do need to know about it so you can modify your behaviour as you see fit (i.e. ask different questions). If there's a camera pointed at you then some might decide that the nose is best left unpicked and those balls will have to remain unscratched. This is really the problem with a lot of these things: not that they're doing these things as such, but that it's so hidden and obfuscated that you need to spend an hour or more trying to understand it, and even then you only know what they "may" do, not what they "actually" do – a lot of these privacy policies are so vague and full of qualified language that they "may" do almost anything. And even in recording conversations there's nuance. Is this recorded with your full account detail, or is that information stripped? Are all conversations recorded or only some? Etc. etc.
- kmeisthax 3y agoThe whole excuse OpenAI gave for going proprietary was specifically that they could spy on you and use that to reactively adjust GPT's alignment training. For what it's worth, the US military is also evaluating LLMs in the same way the Chinese and Russian hackers are. I imagine - or at least hope - they aren't using OpenAI's hosted ChatGPT and are getting copies of models they can run on SIPRNET or JWICS.
- capital_guy 3y agoAnyone using these tools is asking for all their data to be hoovered up. Locally powered AI is the only way to use it if you care about this kind of thing.
- nox101 3y agoCurious what the difference is between that and a search engine.
- VyseofArcadia 3y agoCould be shortened to just "Microsoft Is Spying on Users". If you use a modern MS product, you are sending them lots of data, full stop.
- berniedurfee 3y agoOr more generally, “Everyone is spying on users, that’s how the internet works.”
- Zuiii 3y agoMore like "Most Big-Tech and VC-funded companies spy on users. That's how exploiting customers for profit works." Many of the privacy apps and services I use explicitly set out to collect as little information as possible. It's sad that people have been conditioned to think spying is appropriate or that it has to be this way for the internet to "work".
- VyseofArcadia 3y agoThat's pretty defeatest. I do my best to avoid the spying.
- bilekas 3y agoI do not like M$' behavior in general but this is almost certainly not spying. The OP seems to think that because the users region was recorded its spying on sessions. With a self righteous "I didn't read the TOS but This just confirms it". Microsoft would obviously be recording region metrics, maybe even keyword/request trends. None of this information is identifiable and makes complete sense to be monitoring for improvement on a new product area.
- cooper_ganglia 3y ago> I’m sure the terms of service—if I bothered to read them—gives them that permission. I can’t tell if this article is parody or not, and that’s concerning, haha
- fnordpiglet 3y agoA more precise yet general headline: “Microsoft is spying on users” It’s nearly impossible to install their products without opting into layers of spyware and adware cloaked in dark patterns and requiring extraordinary measures to back out of hidden spyware installs done without permission. Microsoft has always been a fairly malignant business, built in shenanigans of various sorts, and as the market perfects new shenanigan vectors they’re right there pushing the envelope.
- vosper 3y agoWhat are some examples outside of the OpenAI one in Schneier's blog post?
- illusive4080 3y agoWindows 11
- fnordpiglet 3y agoOffice 365
- Zenul_Abidin 3y agoOutlook
- throwanem 3y agoVS Code.
- enopod_ 3y agoEdge
- fnordpiglet 3y ago
- _sword 3y agoThe White House’s AI regulatory order requires big cloud providers to monitor for and report foreign clients (ie Chinese) for using their services to train AI fyi. Monitoring for malicious activity isn’t nearly as invasive as what the government is requiring
- mistrial9 3y agooddly, top ranking AI research in the world right now comes from Stanford USA+Wuhan PRC .. this is trivially verified. Intelligent commentary will need to differentiate between public cooperation layers, non-public cooperation layers, and non-public adversarial layers .. because, they are all active.. all of those.
- _sword 3y agoThe White House doesn’t differentiate! If you’re foreign and training bigger models on US cloud infra, you’re reported https://www.whitehouse.gov/briefing-room/statements-releases/2024/01/29/fact-sheet-biden-harris-administration-announces-key-ai-actions-following-president-bidens-landmark-executive-order/ https://www.whitehouse.gov/briefing-room/statements-releases...
- nonethewiser 3y agoSounds like the government is requiring monitoring of malicious activity.
- numair 3y agoWhile we are on this topic — can someone, anyone, tell me how to disable the AI autocompletion suggestions in Outlook for iOS? Getting ready to dump Microsoft and Exchange to get away from that utter mess. Perhaps some of the people who are commenting here don’t realize that there isn’t much of a way to opt out of these data “analysis” activities.
- probably_satan 3y agoWhy did Microsoft build their AI lab in China, then? Sounds like Microsoft sold the US citizens out to China and are now trying to play the victim card while adding Russia to the blame. Say what you want about conspiracy theories but China and Bill Gates seem to be afraid of women to the point where they have chosen to try to eradicate them from society. Maybe the rumors are true about Bill Gates' attempt to make little girls infertil via vaccination.
- BandButcher 3y agoImo bad story piece to cover up the recent Microsoft hacks done by "state actors". This has nothing to do with MS spying on user ai tool usage. Admit it MS, you got hacked, service accounts and executive emails were leaked, along with who knows what else, and you "caught them" after damage was done.
- add-sub-mul-div 3y agoWhy waste time on performative outrage about how services that require a login have our data and use it in ways we'll never know? You have to accept it as a given and move on in order to start asking the right questions. Like, are we letting ourselves be herded towards a future where it's normalized that all search must be done by authenticated users, and "classic search" is effectively deprecated ten years from now?
- bee_rider 3y agoI could see somebody being confused and not understanding that, say, a copilot tool in a local IDE, or some of their AI in the start menu stuff, is actually using online services. But it is pretty obvious that when you write something into a website that’s… going on the internet, right?
- gigel82 3y agoI mean, probably. But there's no smoking gun here, just wild speculation. Someone should dig deeper and ask Microsoft / OpenAI to disclose exactly how they obtained the "threat intelligence". Here's the full report for reference, but they don't answer how the data was collected or correlated to the specific "threat actors": https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/ https://www.microsoft.com/en-us/security/blog/2024/02/14/sta...
- cynicalsecurity 3y agoFinally Microsoft is doing some good things. Spying on Russian, Iranian etc spies is what you are supposed to do.
- notavalleyman 3y agoThere are real spies in this story, who are committing truly evil acts in the world, and using technology to further their aims of spreading malware and destroying international space conventions. And yet the poster finds Microsoft the bad guys in this story? This is a story about our international enemies using advanced western tech to develop malware, with which to attack us. It's right to be outraged, but not at Microsoft
- deleted 3y ago[deleted]
- bastard_op 3y agoDo they have a product that does NOT spy on users at all? Pretty much everything they do is either cloud connected inherently giving them visibility into what you do, or they feed constant telemetry back of what you do in one form or another in the os, office apps, whatever you happen to use they make, on whatever platform you use it on, including non-windows systems. Really do you expect anything more of Microsoft at this point in history?
- Urgo 3y agoGoogle Bard.. Gemini makes it explicitly clear that they do this right from the homepage: "Your conversations are processed by human reviewers to improve the technologies powering Gemini Apps. Don’t enter anything you wouldn’t want reviewed or used." "How conversations improve Gemini Just by having a conversation with Gemini, you’re making Google services better, including the machine-learning models that power Gemini. As part of that improvement, trained reviewers need to process your conversations. So when using Gemini, don’t enter anything you wouldn’t want a reviewer to see or Google to use. Your Google Workspace content, like from Gmail or Drive, is not reviewed or used to improve Gemini. You can turn Gemini Apps Activity off If you don’t want future conversations reviewed or used to improve machine-learning models, turn off Gemini Apps Activity"
- ilc 3y agoI'd love to get some clarity on this from Google on if they actually support not getting spied on or not? It is only kinda 1/2 clear here.
- living_room_pc 3y agoI always assumed so, and I never feed it anything sensitive.
- autoexec 3y agoI assume it's true for anyone offering AI as a service. Anytime you hand your data over to someone else they will use it for whatever benefits them. No one should be handing them sensitive data.
- orev 3y agoI don’t think this is a surprise to anyone here (I just assume anything done on any of these services is being “used to improve it” i.e. “spying”). But if Schneier is talking about it, maybe it will get more attention.
- interludead 3y agoThere is no such a thing as privacy nowadays
- kelnos 3y agoThe title/headline here is a bit flamebaity. Of course Microsoft is going to retain copies of AI chatbot sessions for a bit, and look for signs of abuse. Even not having read their ToS, it's pretty safe to assume that's going to happen. I just don't see the harm. There are plenty of examples of privacy overreach on the internet, and Microsoft is guilty of some of them, but this particular thing seems fine to me. And in this case, I think MS is doing the right thing! They shouldn't allow people to use these tools if they're going to use them to help them build malware. Certainly this sort of abuse-tracking should be narrow in scope, but it seems like that's exactly what it is, at least in this instance. If you don't want a third-party reading your interactions with an LLM, build/train your own. Eventually we may decide (legally, hopefully) that our interactions with AI tools should have strong privacy protections, but that has not happened yet, and companies are going to apply their usual privacy rules to these things: that is, not much privacy granted, at all. I generally have a lot of respect for Schneier, but this sort of breathless, sensational "reporting" isn't doing privacy advocacy any favors.
- lm28469 3y agoWhat's new? The vast majority of online companies are glorified data miners
- atum47 3y agoNot to be an as*ole but is anyone surprised by this?
- racl101 3y agoSSSHHHHHHHHHHHHHHOCKER.
- canadiantim 3y agoHow much do people trust that VSCode isn't spyware?
- skybrian 3y agoOpenAI is welcome to train on the questions I’ve been asking about how to do type-level programming in TypeScript. In other circumstances, I might ask such questions in a public forum like StackOverflow, or a discussion forum on GitHub, but ChatGPT is more convenient. I suppose other people might ask more private questions, but I have trouble coming up with examples of private things that I’d want to ask an AI chatbot about. And there are plenty of things to ask about that aren’t personal.
- copperx 3y agoThat sounds like the old "I don't do bad things therefore I don't need privacy" pro-spying argument.
- skybrian 3y agoI do need privacy sometimes, but not for this. Others may have different use cases. This is true of everyone. Do you need privacy all the time? Presumably you're okay with your Hacker News comments being public.
- 123sereusername 3y ago[dead]
- lenerdenator 3y agoDuh? AI is built on data. The more you spy, the more data you get to train AI, the more you train AI, the better/more valuable it gets and the more people want to use it to give you data. It's a cycle.
- phillipcarter 3y agoThis is a weird post. Yes, they monitor for abuse of their TOS. No surprise there. But also, this is really the only way to actually improve the underlying tech. How do you know patterns of abuse? Normal use? Weird but non-abusive use? By collecting data on that usage and feeding it back into development.
- yjftsjthsd-h 3y ago> But also, this is really the only way to actually improve the underlying tech. How do you know patterns of abuse? Normal use? Weird but non-abusive use? By collecting data on that usage and feeding it back into development. You can collect that data internally or let users opt in; spying on users is absolutely not the only way to improve.
- phillipcarter 3y agoIt's not spying on users. And no, it's really the only way to do it effectively. When you make it opt in or internal only, you get far too much selection bias to make the product good. That's just how it works with this tech right now.
- dev1ycan 3y agoMicrosoft is very literally an arm of the US government, I think people should already realize that. Google is terrible, but Google refused (they say so we don't know how true this is), to help the US government on spying, and Microsoft stepped in. Bill Gates gets to sit with Xi Jinping and get called a "friend" and MS has exceptions in China unlike other companies. Microsoft wants to buy ActivisionBlizzard, gets brought to the senate on the antitrust case, and the senators start barking against Sony instead, lol. Microsoft is very literally all the US accusses Chinese companies of being for the Chinese government, now, so far it doesn't seem like MS has done that much with our data, but who knows what the future holds.
- coldtea 3y ago>Google is terrible, but Google refused Isn't "we refused" what they would have said they did even if they haven't refused?
- dev1ycan 3y agoYes, I know, that's why I stated I can't confirm if that's really the truth. Again I'll cite the whole ordeal with Apple claiming they are the privacy company and how they "refused" the government access to a persons iphone back in the day as an example, I wonder how they refused when the patriot act is a thing? they refused the government? lol
- weare138 3y agoIn collaboration with OpenAI, we are sharing threat intelligence showing detected state affiliated adversaries—tracked as Forest Blizzard, Emerald Sleet, Crimson Sandstorm, Charcoal Typhoon, and Salmon Typhoon Does anyone else think this new APT naming scheme is terrible or is it just me?
- tzm 3y agospying === user engagement?
- RIMR 3y agoFrom the comments: >OpenAI are “spying” on ChatGPT sessions the same way that email providers “spy” on your email to filter spam. This sounds like a variation on “Google is reading your email”. Yeah, but Google is actually reading your emails. They build a profile of everything you buy from emailed receipts, and use that data to target ads at you. It's definitely not about improving the user experience, like a Spam filter accomplishes...
- drivingmenuts 3y agoIf it’s running on their serve and it’s part of the license, why shouldn’t they? A real competitor or wanna-be should have the sense not to do that, so it seems justified that they would keep an eye for how their product is used.
- wut-wut 3y agoWell Duh.
- 1vuio0pswjnm7 3y ago"I'm sure the terms of service-if I bothered to read them-gives them that permission. And of course it's no surprise that Microsoft and OpenAI (and, presumably, everyone else) are spying on our usage of AI, but this confirms it." Not sure why anyone would think that terms that allow surveillance are not functionally equivalent to evidence of surveillance. But it seems like this thinking is what lets people trust the aptly-named "privacy policies" used by so-called "tech" companies to make users more comfortable with surveillance. If a so-called "tech" company makes statements in a "privacy policy", something like, "We do not do X", and there is no way to verify whether they do X or not,^1 what purpose does the privacy policy serve. If in fact someone acquires evidence that the company is doing X, it is already too late. X was not prevented. Whereas, if a so-called "tech" company includes provisions in its terms of service that allow it to do X, then what functional difference does it make whether the company does X or not. They could do X at any time. Anyone who agrees to the terms has given them permission to do X. In general, companies do not ask for the explicit right to do X in an agreement if there is zero possibility they will do X. Why do these so-called "tech" companies need a separate document, a "privacy policy", to make users believe the company will not do X. Why not include a prohibition against X in the terms of service. To understand what the company does, read the terms of service. If the terms allow the company to do X, then assume the company does X. It is foolish to assume otherwise. 1. If there was a way to verify it, then the company would not need to make such assurances.
- southernplaces7 3y agoWell color me shocked and surprised that Microsoft would ever do such a thing! A basic rule of thumb is to assume that virtually all commercial apps or digital tools (and many physical "smart" tools and products) pervasively, parasitically gather as much data about every aspect of your activity as they possibly can. This is the modern world's running business model: being constantly spied on by an ever growing range of objects and systems around you for possible monetization reasons at all times, even if you paid for them. Governments make noises about opposing this but many of them tacitly love it because it spares them all the legal and logistical bullshit of justifying doing the same themselves. Why bother after all? Now you can just buy or subpoena, or straight up illegally force (looking at you Snowden leaks) all of it from all the private entities that are given free rein to do what your "sadly archaic" laws still don't let you do. Most absurd of all, that these same private entities make minimal effort at keeping their own monitoring a secret. The majority of their users literally don't give a damn and in fact freely hand over permission in a nicely, legally justified bundle. We essentially live in a Stasi, KGB or Gestapo functionary's dream of near panopticon.
- begueradj 3y agoNot one single software produced by Microsoft, starting from its operating system, which is not a spyware