16 ms·

Let's talk about AI and end-to-end encryption

269 points by chmaynard 2y ago

username223 2y ago
[flagged]
- dingnuts 2y ago
  fucking hell are we only now having this discussion? when GitHub Copilot first released, a few years ago, my first reaction was: that's a keylogger! it sends everything in your editor to Microsoft, isn't anyone worried about that at all? No, no one was. I despair
  - AyyEye 2y ago
    "Keylogger" and "spyware" have become so normalized that they have disappeared from the public lexicon.
    - jazzyjackson 2y ago
      With apologies, 1960s: “I BETTER NOT SAY THAT OR THE GOVERNMENT WILL WIRETAP MY HOUSE” 2020s: “HEY WIRETAP, DO YOU HAVE A RECIPE FOR PANCAKES?”
- airstrike 2y ago
  > tl;dr -- Most AI stuff won't run at an acceptable speed on your computer ...today. There's no real reason we can't get acceptable speeds in the near future.
  - deathanatos 2y ago
    Yes, but also no: surface area is a proxy for compute power, because it's a proxy to heat dissipation. Phones are also only passively air-cooled — which is like the absolute worst cooling mechanism, too¹. So yes, processors do get better, but watts are watts, and there's a hard upper limit to how many watts a phone can consume, and you have to get better within that limit. Or … IDK, maybe you don't. Recent Android phones are now capable of prompting the user to actively cool them off because the ~bloat~ software is consuming too much compute/power. What basically translates to "help I'm dying" is an amusing message, but also a depressing state of affairs. ¹compare to a laptop, or worse for the phone, a desktop. The comparative cooling surface area is multiple times larger. And both have fans, desktops can be liquid cooled — it's not mere chance that they're faster, and phone form factor literally presents challenges. Even with an infinite battery, or a USB capable, you can't dissipate 60 W into a human hand.
    - airstrike 2y ago
      I agree but I was replying to the comment about computers specifically, meaning desktop and laptop form factors rather than mobile phones, especially when I say "near future"
- deleted 2y ago
  [deleted]
crackalamoo 2y ago
See also CrypTen, Meta's library for privacy preserving machine learning: https://github.com/facebookresearch/CrypTen https://github.com/facebookresearch/CrypTen. This isn't fully homomorphic encryption, but it is multi-party computation (MPC), which hides the inputs from the company owning the model. But while not revealing user input, it would still reveal the outputs of the model to the company. And yeah, as the article mentions, unfortunately this kind of thing (MPC or fully-homomorphic encryption) probably won't be feasible for the most powerful ML models.
EGreg 2y ago
I heard that homomorphic encryption can actually preserve all the operations in neural networks, since they are differentiable. Is this true? What is the slowdown in practice?
- crackalamoo 2y ago
  This is true in principle, yes. In practice, the way this usually works is by converting inputs to bits and bytes, and then computing the result as a digital circuit (AND, OR, XOR). Doing this encrypted is very slow: without hardware acceleration or special tricks, running the circuit is 1 million times slower than unencrypted, or about 1ms for a single gate. (https://www.jeremykun.com/2024/05/04/fhe-overview/ https://www.jeremykun.com/2024/05/04/fhe-overview/) When you think about all the individual logic gates involved in just a matrix multiplication, and scale it up to a diffusion model or large transformer, it gets infeasible very quickly.
  - j2kun 2y ago
    There are FHE schemes that do better than binary gates (cf. CKKS) but they have other problems in that they require polynomial approximations for all the activation functions. Still they are much better than the binary-FHE schemes for stuff like neural networks, and most hardware accelerators in the pipeline right now are targeting CKKS and similar for this reason. For some numbers, a ResNet-20 inference can be done in CKKS in like 5 minutes on CPU. With custom changes to the architecture you can get less than one minute, and in my view HW acceleration will improve that by another factor of 10-100 at least, so I'd expect 1s inference of these (still small) networks within the next year or two. LLMs, however, are still going to be unreasonably slow for a long time.
jFriedensreich 2y ago
I think this has also a silver lining. The E2E encryption movement especially for messenger apps was largely also used to silently lock users out of their own data and effectively prevent user agency to use their own data to move apps, write automations or archive, this is not just true for whatsapp (the data export feature does not fully work since its launch and was just made to appease some EU law that did not properly check if the button works until the end.) Also signal does not have a way to do this. Maybe with ai coming into the game companies finally decide to provide access to data, I just hope it's in a transparent way with user opt in and user control.
- do_not_redeem 2y ago
  1. Is data encrypted in transit? 2. Can the user access their data at rest? Those two things are entirely orthogonal. I don't think you can extrapolate a trend from a few apps having bugs in their export code. Google Takeout is also notoriously buggy and they don't use E2E encryption. A more likely explanation is companies of all kinds don't care that much about export functionality, due to the incentives involved.
  - jFriedensreich 2y ago
    you CAN extrapolate from nearly all e2e encrypted apps not giving a way to use the data. And there is a big difference between buggy google export features or facebook actively making export unusable to lock in users.
- walrus01 2y ago
  Signal does not have a way to manually export your private keys and chat history, but the process of "moving" your signal account to a new phone is quite straightforward. You put both devices on the same wifi/LAN layer 2 broadcast segment, start the transfer process in the app, input the verification codes displayed on the screen from both devices, and it sends everything over. This moves the private key in a way that does not result in all of your contacts receiving the scary "this person's key has changed" message.
_wire_ 2y ago
[flagged]
bobbiechen 2y ago
There is always going to be a gap between local-first processing and what can be achieved in a full-sized datacenter/cloud. That leads to the risks mentioned in the article. I wrote about Apple's Private Cloud Compute last year; for the foreseeable future, I still think server-side Confidential Computing is the most practical way to do processing without huge privacy risks: https://www.anjuna.io/blog/apple-is-using-secure-enclaves-to-ensure-ai-privacy-why-arent-you https://www.anjuna.io/blog/apple-is-using-secure-enclaves-to...
tonygiorgio 2y ago
> Although PCC is currently unique to Apple, we can hope that other privacy-focused services will soon crib the idea. IMHO, Apple's PCC is a step in the right direction in terms of general AI privacy nightmares where they are at today. It's not a perfect system, since it's not fully transparent and auditable, and I do not like their new opt-out photo scanning feature running on PCC, but there really is a lot to be inspired by it. My startup is going down this path ourselves, building on top of AWS Nitro and Nvidia Confidential Compute to provide end to end encryption from the AI user to the model running on the enclave side of an H100. It's not very widely known that you can do this with H100s but I really want to see this more in the next few years.
- mnahkies 2y ago
  I didn't actually realize that AWS supported this, I thought Azure was the only one offering it (https://azure.microsoft.com/en-us/blog/azure-confidential-computing-with-nvidia-gpus-for-trustworthy-ai/ https://azure.microsoft.com/en-us/blog/azure-confidential-co...) Are you speaking of this functionality? https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/ https://developer.nvidia.com/blog/confidential-computing-on-... (and am I just failing to find the relevant AWS docs?)
  - tonygiorgio 2y ago
    Yes, you're correct on both, though I think Google Cloud recently started supporting it as well. AWS will likely have GPU enclave support with Trainium 2 soon (AFAIK, that feature is not publicly offered yet but could be wrong). We work with Edgeless Systems who manages the GPU enclave on Azure that we speak to from our AWS Nitro instance. While not ideal, the power of enclaves and the attestation verification process, we at least know that we're not leaking privacy by going with a third party GPU enclave provider.
    - calum-bird 2y ago
      Do you have any resources you can share on managing the enclave’s public key? We found documentation to be sparse.
bee_rider 2y ago
The author helpfully emphasized the interesting question at the end > This future worries me because it doesn’t really matter what technical choices we make around privacy. It does not matter if your model is running locally, or if it uses trusted cloud hardware — once a sufficiently-powerful general-purpose agent has been deployed on your phone, the only question that remains is who is given access to talk to it. Will it be only you? Or will we prioritize the government’s interest in monitoring its citizens over various fuddy-duddy notions of individual privacy. I do think there are interesting policy questions there. I mean it could hypothetically be mandated that the government must be given access to the agent (in the sense that we and these companies exist in jurisdictions that can pass arbitrary laws; let’s skip the boring and locale specific discussion of whether you think your local government would pass such a law). But, on a technical level—it seems like it ought to be possible to run an agent locally, on a system with full disk encryption, and not allow anyone who doesn’t have access to the system to talk with it, right? So on a technical level I don’t see how this is any different from where we were previously. I mean you could also run a bunch of regex’s from the 80’s to find whether or not somebody has, whatever, communist pamphlets on their computers. There’s always been a question of whether the government should be able to demand access to your computer. I guess it is good to keep in mind that if they are demanding access to an AI agent that ran on your computer, they are basically asking for a lossy record of your entire hard drive.
- _boffin_ 2y ago
  Unreasonable search?
  - bee_rider 2y ago
    > (in the sense that we and these companies exist in jurisdictions that can pass arbitrary laws; let’s skip the boring and locale specific discussion of whether you think your local government would pass such a law) Anyway the idea of what’s a reasonable search in the US has been whittled away to almost nothing, right? “The dog smelled weed on your hard drive.” - A cop, probably.
  - chgs 2y ago
walrus01 2y ago
It's a good thing that encrypted data at rest on your local device is inaccessible to cloud based "AI" tools. The problem is that your average person will blithely click "yes/accept/proceed/continue/I consent" on pop up dialogs in a GUI and agree to just about any Terms of Service, including decrypting your data before it's sent to some "cloud" based service. I see "AI" tools being used even more in the future to permanently tie people to monthly recurring billing services for things like icloud, microsoft's personal grade of office365, google workspace, etc. You'll pay $15 a month forever, and the amount of your data and dependency on the cloud based provider will mean that you have no viable path to ever stop paying it without significant disruption to your life.
Animats 2y ago
> Who does your AI agent actually work for? Yes. I made that point a few weeks ago. The legal concept of principal and agent applies. Running all content through an AI in the cloud to check for crimethink[1] is becoming a reality. Currently proposed: - "Child Sexual Abuse Material", which is a growing category that now includes AI-generated images in the US and may soon extend to Japanese animation. - Threats against important individuals. This may be extended to include what used to be considered political speech in the US. - Threats against the government. Already illegal in many countries. Bear in mind that Trump likes to accuse people of "treason" for things other than making war against the United States. - "Grooming" of minors, which is vague enough to cover most interactions. - Discussing drugs, sex, guns, gay activity, etc. Variously prohibited in some countries. - Organizing protests or labor unions. Prohibited in China and already searched for. Note that talking around the issue or jargon won't evade censorship. LLMs can deal with that. Run some ebonics or leetspeak through an LLM and ask it to translate it to standard English. Translation will succeed. The LLM has probably seen more of that dialect than most people. "If you want a vision of the future, imagine a boot stepping on a face, forever" - Orwell [1] https://www.orwell.org/dictionary/ https://www.orwell.org/dictionary/
- iugtmkbdfil834 2y ago
  A cynic in me is amused at the yet unknown corporation being placed under investigation due to a trigger phrase in one of the meetings transcribed incorrectly. Your point is worth reiterating.
  - Terr_ 2y ago
    Or poisoned-data that sets up a trap, so that a system will later confabulate false innocence or guilt when certain topics or targets come up.
- crooked-v 2y ago
  "Grooming" in particular is the angle that Republicans want to use to illegalize any kind of gender-nonconforming behavior, up to and including desired states like "women wearing pants is crossdressing, and doing so around children is a felony".
nashashmi 2y ago
The most depressing realization in all of this is that the vast treasure trove of data that we used to have in the cloud thinking it was not scannable even for criminal activity has now become a vector where we shall have thought police coming down upon us for simple ideas of dissent.
- AlexandrB 2y ago
  A lot of people tried to sound the alarm. It's not "the cloud", it's "other people's computers". And given that other people own these machines, their interests - whether commercial or ideological - will always come first.
  - tokioyoyo 2y ago
    To be fair, most people understand that risk. It’s just it is very convenient in a lot of scenarios and some businesses might have not even started without it. Privacy is not that big of a concern for a big chunk of people. And they’re basically voting with their wallets.
    - like_any_other 2y ago
      "Voting with their wallets", where all the choices are picked by entities hostile to consumer privacy and autonomy, and then they mislead you about those choices. People who bought LG (and now most other, now "smart") TVs did not in any meaningful way "vote" to be spied on and support DRM - simply all the TVs in a store would spy and show ads, and not disclose any of it at time of sale.
      - tokioyoyo 2y ago
        Well, it shows how it’s not the biggest issue for an average customer, as it, in theory, can make the TV cheaper. If people cared about absence of ads that much, all social media would not be ad-ridden since people would stop using it.
        deleted 2y ago
blueblimp 2y ago
> Yet this approach is obviously much better than what’s being done at companies like OpenAI, where the data is processed by servers that employees (presumably) can log into and access. No need for presumption here: OpenAI is quite transparent about the fact that they retain data for 30 days and have employees and third-party contractors look at it. https://platform.openai.com/docs/models/how-we-use-your-data https://platform.openai.com/docs/models/how-we-use-your-data > To help identify abuse, API data may be retained for up to 30 days, after which it will be deleted (unless otherwise required by law). https://openai.com/enterprise-privacy/ https://openai.com/enterprise-privacy/ > Our access to API business data stored on our systems is limited to (1) authorized employees that require access for engineering support, investigating potential platform abuse, and legal compliance and (2) specialized third-party contractors who are bound by confidentiality and security obligations, solely to review for abuse and misuse.
- deleted 2y ago
  [deleted]
- chefandy 2y ago
  I have to say — I’m kind of amazed that anyone would expect privacy out of chat bot companies and products. You’re literally having a “conversation” with the servers of companies that built their entire product line using other people’s professional and personal output whether they approved, or even knew about it or not. Less a “it’s better to ask for forgiveness than permission” sort of thing than a “we’d rather just not ask and be pretty cagey about it if they ask, and then if they prove it, tell them they tacitly agreed to it by not hiding it from us even though they had no way to know we were looking at it“ sort of thing. Frankly I’m astonished that open ai, specifically, promises as much as they do in their privacy policy. Based on their alleged bait-and-switch tactics quietly swapping out models or reducing compute for paying customers after the initial “gee wiz look at that” press cycle, I can’t imagine those privacy policies will have much longevity when the company gets a more stable footing… and whooops looks like they figured out how to extract the training data from the models! And it’s different data since we extracted it from the model so the old privacy policy doesn’t apply! Haha sorry, that’s business and we’re building a techno utopian society here, so you should feel honored to be included! You think Altman wouldn’t sell that in a heartbeat to try and fund some big moonshot product if they get clobbered in the marketplace? Never mind the sketchy girlfriend-in-an-app-class chatbots. Don’t get me wrong — I absolutely think the privacy SHOULD be there, but I’m just shocked that anyone would assume it was. Maybe I’m being overly cynical? These days when I think I might be, in the end, it seems I wasn’t being cynical enough.
lowbatt 2y ago
Maybe a little off topic, but is there a way for a distributed app to connect to one of the LLM companies (OpenAI, etc.) without the unencrypted data hitting an in-between proxy server? An app I'm building uses LLMs to process messages. I don’t want the unencrypted message to hit my server - and ideally I wouldn’t have the ability to decrypt it. But I can’t communicate directly from client -> LLM Service without leaking the API key.
- simonw 2y ago
  "But I can’t communicate directly from client -> LLM Service without leaking the API key." There is a way you can do that right now: the OpenAI WebRTC API introduced the idea of an "ephemeral key": https://platform.openai.com/docs/guides/realtime-webrtc https://platform.openai.com/docs/guides/realtime-webrtc This provides a way for your server to create a limited-time API key for a user which their browser can then use to talk to OpenAI's API directly without proxying through you. I love this idea, but I want it for way more than just the WebRTC API, and I'd like it for other API providers too. My ideal version would be a way to create an ephemeral API key that's only allowed to talk to a specific model with a specific pre-baked system prompt (and maybe tool configuration and suchlike) and that only works for a limited time and has a limited token budget.
  - lowbatt 2y ago
    interesting, will check that out. thanks!
- michaelmrose 2y ago
  Will such processing be cheap enough to be done by a box that plugs into a customers router to handle such? Would they buy them? Notably not just for this use case but others
- whyage 2y ago
  Check out https://www.opaque.co/ https://www.opaque.co/
ozgune 2y ago
> Apple even says it will publish its software images (though unfortunately not the source code) so that security researchers can check them over for bugs. I think Apple recently changed their stance on this. Now, they say that "source code for certain security-critical PCC components are available under a limited-use license." Of course, would have loved it if the whole thing was open source. ;) https://github.com/apple/security-pcc/ https://github.com/apple/security-pcc/ > The goal of this system is to make it hard for both attackers and Apple employees to exfiltrate data from these devices. I think Apple is claiming more than that. They are saying 1/ they don't keep any user data (data only gets processed during inference), 2/ no privileged runtime access, so their support engineers can't see user data, and 3/ they make binaries and parts of the source code available to security researchers to validate 1/ and 2/. You can find Apple PCC's five requirements here: https://security.apple.com/documentation/private-cloud-compute/corerequirements https://security.apple.com/documentation/private-cloud-compu... Note: Not affiliated with Apple. We read through the PCC security guide to see what an equivalent solution would look like in open source. If anyone is interested in this topic, please hit me up at ozgun @ ubicloud . com.
- saagarjha 2y ago
  Some of the core elements of the boot process are not source available, unfortunately.
natch 2y ago
From Apple's document on Advanced Data Protection: >With Advanced Data Protection enabled, Apple doesn't have the encryption keys needed to help you recover your end-to-end encrypted data. Apple doesn't have the keys. Somebody else might. Somebody other than you. Also, I think they meant to say decryption keys, although they're probably just dumbing down terminology for the masses. >If you ever lose access to your account, you’ll need to use one of your account recovery methods "You'll need to use." Not "there is no way except to use." >Note: Your account recovery methods are never shared with or known to Apple. "shared with or known to Apple." Not "shared with or known to anyone else." The encryption is there, I believe that. I just don't know how many copies of the keys there are. If the only key is with me, it would be super easy for Apple to just say that. I believe that they have said that in the past, but the wording has now changed to this hyper-specific "Apple does not have the key" stuff.
- deleted 2y ago
  [deleted]
- musicale 2y ago
  As you suggest, the wording should be clarified to say that the key is never copied, is only stored on your device, is not accessible to others, etc.
  - natch 2y ago
    Maybe they are unable to make that clarification, if it would be false.
  - systoll 2y ago
    It does say > It’s protected with the new key which is controlled solely by the user’s trusted devices I think main thing they’re avoiding is an explicit guarantee that the key cannot be retrieved from your phone by a third party.
rglover 2y ago
> We are about to face many hard questions about these systems, including some difficult questions about whether they will actually be working for us at all. And how. I'd lean towards no. Where we're headed feels like XKEYSCORE on steroids. I'd love to take the positive, optimistic bent on this, but when you look at where we've been combined with the behavior of the people in charge of these systems (to be clear, not the researchers or engineers, but c-suite), hope of a neutral, privacy-first future seems limited.
- deleted 2y ago
  [deleted]
- ActorNightly 2y ago
  Given how politics and companies evolved, I actually trust those people in charge of XKEYSCORE systems more than ever. They may wear suits, but those people usually come from some military background, and have a sense of duty towards defending US, from threats both foreign and domestic, and historically have not really abused their powers no matter what the administration is. XKEYSCORE for example, wasn't really about hacking people, it was just about collecting mass metadata and building profiles, well within the legal system, and the blame should be on the companies that didn't provide privacy tools, because any big government could have build the same system. Meanwhile, the anti anti-establishment Republican Party since 2016 who cried about big tech turned out to be the biggest pro-establishment fans, giving Elmo an office in a white house and Zucc bending a knee to avoid prosecution. With these new systems, Id rather have smart people who only work in US defensive forces because of a sense of duty (considering they could get paid much more in the private sector) in charge.
  - schmidtleonard 2y ago
    > well within the legal system It's not a search if we don't find anything, and it's not a seizure if we charge the money with the crime. These are court approved arguments, so they must be correct interpretations. Point is: modern bureaucrats have proven that they are absolutely willing to abuse power, even in the best of times when there is no real domestic political strife.
jrm4 2y ago
"The goal of encryption is to ensure that only two parties, the receiver and sender, are aware of the contents of your data. Thus, AI training on your data breaks this, because it's another party. You now don't have encryption." Thanks for coming to my blah blah blah
- dialup_sounds 2y ago
  The article has nothing to do with model training.
peppertree 2y ago
Is embeddings enough to preserve privacy? If I run the encoder/decoder on device and only communicate with server in embeddings?
- nostradumbasp 2y ago
  For a short time maybe, but that kind of activity might look like something interesting enough for someone to target you and remove all privacy from your entire existence.
- hallh 2y ago
  No, the original text can largely be recovered from embeddings[0] if you know which embedding model was used. [0] https://arxiv.org/abs/2310.06816 https://arxiv.org/abs/2310.06816
klik99 2y ago
> You might even convince yourself that these questions are “privacy preserving,” since no human police officer would ever rummage through your papers, and law enforcement would only learn the answer if you were (probably) doing something illegal. Something I've started to see happen but never mentioned is the effect automated detection has on systems: As detection becomes more automated (previously authored algorithms, now with large AI models), there's less cash available for individual case workers, and more trust at the managerial level on automatic detection. This leads to false positives turning into major frustrations since it's hard to get in touch with a person to resolve the issue. When dealing with businesses it's frustrating, but as these get more used in law enforcement, this could be life ruining. For instance - I got flagged as illegal reviews on Amazon years ago and spent months trying to make my case to a human. Every year or so I try to raise the issue again to leave reviews, but it gets nowhere. Imagine this happening for a serious criminal issue, with the years long back log on some courts, this could ruin someones life. More automatic detection can work (and honestly, it's inevitable) but it's got to acknowledge that false positives will happen and allocate enough people to resolve those issues. As it stands right now, these detection systems get built and immediately human case workers get laid off, there's this assumption that detection systems REPLACE humans, but it should be that they augment and focus human case workers so you can do more with less - the human aspect needs to be included in the budgeting. But the incentives aren't there, and the people making the decisions aren't the ones working the actual cases so they aren't confronted with the problem. For them, the question is why save $1m when you could save $2m? With large AI models making it easier and more effective to build automated detection I expect this problem to get significantly worse over the next years.
- smallmancontrov 2y ago
  The UK Post Office scandal is bone-chilling. Update this to a world where every corner of your life is controlled by a platform monopoly that doesn't even provide the most bare-bones customer service and yeah, this is going to get a lot worse before it gets better.
  - Vampiero 2y ago
    And that's the early game. Imagine when AI will be monitoring all internet traffic and arresting people for thoughtcrime. What wasn't feasible to do before is now quite in reach and the consequences are dire. Though of course it won't happen overnight. First they will let AI encroach every available space (backed by enthusiastic techbros). THEN, once it's established, boom. Authoritarian police state dystopia times 1000. And it's not like they need evidence to bin you. They just need inference. People who share your psychological profile will act and speak and behave in a similar way to you, so you can be put in the same category. When enough people in that category are tagged as criminals, you will be too. All because you couldn't be arsed to write some boilerplate
deathanatos 2y ago
TFA makes some rather basic errors. First, > Prior to 2011, most cloud-connected devices simply uploaded their data in plaintext. > Around 2011 our approach to data storage began to evolve. […] began to roll out default end-to-end encryption […] This technology changed the way that keys are managed, to ensure that servers would never see the plaintext content of your messages. "changed the way that keys are managed" is at a confused contradiction with "uploaded their data in plaintext". If you're going from TLS → E2EE, then yeah, "changed the way keys are managed" miiight make sense, though that's not how I'd phrase it. Then later, > On the one hand they can (1) send plaintext off to a server, in the process resurrecting many of the earlier vulnerabilities that end-to-end encryption sought to close. Or else (2) they can limit their processing to whatever can be performed on the device itself. We're still confusing "transmit plaintext" with plaintext being available to the server; the clear option of "use TLS" is omitted. It doesn't really undermine the argument — the server would still have access to the data, and could thus maliciously train AI on it — but it is surprising for a "cryptographer". > For example, imagine that Apple keeps its promise to deliver messages securely, but then your (Apple) phone goes ahead and uploads (the plaintext) message content to a different set of servers where Apple really can decrypt it. Apple is absolutely using end-to-end encryption in the dullest technical sense… yet is the statement above really accurate? Is Apple keeping its broader promise that it “can’t decrypt the data”? No, no reasonable person would believe that (though I am sure that if the scenario ever came to be, Apple, or whoever, would likely argue "yes") since it would utterly scuttle the term "E2EE". If you say "Our product supports X", and then have to caveat away 100% of what makes X X, then it's just grift, plain and simple. (Now, whether grift sees regulatory action … well.) > Now imagine that some other member of the group — not you, but one of your idiot friends — decides to turn on some service that uploads (your) received plaintext messages to WhatsApp. > In general, what we’re asking here is a question about informed consent. I would sort of agree, but corporations will expose the consent here to the "friend", and then argue that because the friend consented to your data being uploaded, it is fine. An argument for privacy regulations. (I don't think you have to go through all this … work. Just upload the user's data. They'll complain, for a bit, but the market has already consolidated into at least an ologopoly, users have shown that, for the most part, they're going to keep using the product rather than leave, or else I'll be ending this comment with a free "2025 will be the Year of the Linux Desktop". What's gonna happen, regulation to ensure a free market remains free¹? Please. Cf. MS Recall, currently in the "complain" phase, but give it time, and we'll reach the "we heard your concerns, and we value your input and take your feedback with the utmost respect ram it down their throats" stage.) (¹free as in "dictated by the laws of supply & demand", not laissez-faire which is where the US will be headed for the next 4.) (and … 2011? I'd've said 2013 is when we found out the 4A meant way less than we thought it did, leading to the rise in massive adoption of TLS. Less so E2EE.)
lifeisstillgood 2y ago
So if I understand it 1. E2E encryption does work 2. But phones can send plaintext back to the cloud to get help doing AI things 3. And we tend not to know because it’s all “assisstance” But the solution like anything is pricing. I mean yet again (uber, Airbnb) billions of dollars of VC money is used as subsidy so my photos can get OCR’d. If phones said “hey for a dollar fifty I can work out what the road sign says behind your dogs head in 32 photos your mum sent you last week” I think we woukd see a different threat landscape This is - again - unsustainable cash spending distorting markets and common sense. If the market was “we can OCR and analyse these insurance claims” the. Things like privacy and encryption would be first class requirements and harder to sell and build. By spending a billion they can sell services to people without regulators to ask awkward questions and then they hope step 3. Profit. I short not even AI can spot patterns in encrypted data, it’s only when plaintext gets sent around in the hope of profit do we see a threat. That’s seems a simple fix if not an easy one
fragmede 2y ago
The article hinges on a bad assertion that > Apple can’t rely on every device possessing enough power to perform inference locally. This means inference will be outsourced to a remote cloud machine. If you go look at Apple's site https://www.apple.com/apple-intelligence/ https://www.apple.com/apple-intelligence/ and scroll down, you get: Apple Intelligence is compatible with these devices. iPhone 16 A18 iPhone 16 Plus A18 iPhone 16 Pro Max A18 Pro iPhone 16 Pro A18 Pro iPhone 15 Pro Max A17 Pro iPhone 15 Pro A17 Pro iPad Pro M1 and later iPad Air M1 and later iPad mini A17 Pro MacBook Air M1 and later MacBook Pro M1 and later iMac M1 and later Mac mini M1 and later Mac Studio M1 Max and later Mac Pro M2 Ultra If you don't have one of those devices, Apple did the obvious thing and disabled features on devices that don't have the hardware to do it. While Apple has this whole private server architecture, they're not sending iMessages off device for summarization, that's happening on device.
flossposse 2y ago
Green, (the author), makes an important point: > a technical guarantee is different from a user promise. [...] End-to-end encrypted messaging systems are intended to deliver data securely. They don’t dictate what happens to it next. Then Green seems to immediately forget the point they just made, and proceed to talk about PCC as if it were something other than just another technical guarantee. PCC only helps to increase confidence that the software running on the server is the software Apple intended to be there. It doesn't give me any guarantees about where else my data might be transferred from there, or whether Apple will only use it for purposes I'm okay with. PCC makes Apple less vulnerable to hacks, but doesn't make them any more transparent or accountable. In fact, to the extent that some hackers hack for pro-social purposes like exposing corporate abuse, increased security also serves as a better shield against accountability. Of course, I'm not suggesting that we should do away with security to achieve transparency. I am, however, suggesting that transparency, moreso than security, is the major unaddressed problem here. I'd even go so far as to say that the woeful state of security is enabled in no small part by lack of transparency. If we want AI to serve society, then we must reverse the extreme information imbalance we currently inhabit wherein every detail of each person's life is exposed to the service provider, but the service provider is a complete black-box to the user. You want good corporate actors? Don't let them operate invisibly. You want ethical tech? Don't let it operate invisibly. (Edit: formatting)
p4bl0 2y ago
This article should be accompanied by Five things privacy experts know about AI [1]. They pair really well and should probably link to each other. [1] https://news.ycombinator.com/item?id=42695628 https://news.ycombinator.com/item?id=42695628
- fourthark 2y ago
  (which discusses privacy issues on the training side)
ajb 2y ago
The real threat here is going to be when AI expands from being applied to accelerate the work of individuals, to being applied to the control of organisations. And it will be tempting to do that. We all know the limitations of managers, management hierarchies, metrics, OKRs etc. It's easy to think of a CEO deciding that all the communications between their employees should just be fed into an AI that they can query . (Ironically that would be easier to enforce if everyone was remote). It's quite possible that it would enable more effective organisations, as the CEO and upper level management can have a better idea of what is really happening. But it will reduce the already tenuous belief of the powerful that their ordinary staff are real human beings. And it will inevitably leak out from private organisations, as the executive class see no reason why they shouldn't have the same tools when running the country as when running a corporation. Advocates of mass surveillance like to point out that no human now needs to listen to your calls. But the real danger was never the guy in a drab suit transcribing your conversions from reel-to-reel tape. It was always the chief who could call for the dossier on anyone they were finding inconvenient, and have it looked at with an eye to making you never inconvenience them again. The full consequences of mass surveillance have not played out simply because no one had the means to process that much ad-hoc unstructured data. Now they do.
- TeMPOraL 2y ago
  > It's easy to think of a CEO deciding that all the communications between their employees should just be fed into an AI that they can query . (Ironically that would be easier to enforce if everyone was remote). This is already happening, whether the CEOs want it or not - when there's a legal issue requiring discovery, e-discovery software may be used to pull in all digital communications that can be accessed, and feed it all to AI for, among other things, sentiment analysis. Applications of GenAI for legal work, in general, is a hot topic in legal circles now.
1vuio0pswjnm7 2y ago
"You've probably also noticed Silicon Valley's enthusiasm to find applications for this tech." A "solution" looking for a "problem". Like cryptocurrency. Like the "metaverse". And so on. When the "solution" brings new problems SillyCon Valley then proclaims they have a solution to the problem they created. "Even if these firms don't quite know how AI will be useful to their customers, they've already decided that models are the future." More often than not the "customers" are advertisers. Other computer users are just ad targets.
- TeMPOraL 2y ago
  GenAI (and LLMs in particular) is a "solution looking for a problem" in the sense petroleum or electricity was, not in the sense cryptocurrency is. I.e. you take one look at it and it's 100% obvious it's a general technology that's going to be useful in some way for all kinds of work people do, and be doing in the near future, and itself will make possible things and endeavors that weren't possible before. The devil is always in the details, but assuming that GenAI might be important for your business is correct no matter what business you're in, so it's worth it to pay attention and test it sooner rather than later.
- 1vuio0pswjnm7 2y ago
  "Artificial intelligence (AI), in its broadest sense, is intelligence exhibited by machines, particularly computer systems." "A cryptocurrency, crypto-currency, or colloquially, crypto, is a digital currency designed to work through a computer network that is not reliant on any central authority, such as a government or bank, to uphold or maintain it." "The metaverse is a loosely defined term referring to virtual worlds in which users represented by avatars interact,^[1] usually in 3D and focused on social and economic connection." "Electricity is the set of physical phenomena associated with the presence and motion of electric charge." "Petroleum is a naturally occurring yellowish-black liquid mixture." (Definitions taken from one of ChatGPTS's primary training sources.) A careful reader is expected to notice the first three have some charactersitics in common, specifically ones that the last two do not.
reader9274 2y ago
For some reason I felt like this article had a lot of promise to start, but turned into fluff and I learned nothing. It covered very basic ideas of both AI and encryption, which I didn't expect from such a highly regarded expert.
dathinab 2y ago
> They’ll order your food and find the best deals on shopping, swipe your dating profile, negotiate with your lenders, and generally anticipate your every want or need. Why should they do so? I mean seriously. There is more money to make in telling you that the AI will buy you the beast deal but instead buy premeditated (i.e. bought) "okay" looking deals instead. Similar dating apps and the related ecosystems has a long history of scamy behavior in all kind of ways as they want to keep you using the app. And people with money have always found ways to highlight themself more. I.e. there is more money to make in "swiping for you" in way which looks on the surface honest but isn't. etc. etc. There is basically always more money to make in systematically deceiving people as long as you do it well enough so that people don't notice, or they don't have a real/realistic choice, i.e. are forced by the circumstances. So the moment you take the human and human conscience/moral out of the loop and also have no transparency there is a 0% of this ending well if regulators don't preclude abuse. And AI is pretty effective at removing transparency and the humanity out of the loop. With how things currently look, especially in the US, they (edit: they==regulators) are more likely to do the opposite (edit: i.e. remove consumer protections).
- CatWChainsaw 2y ago
  It's like nobody actually learned anything from the Honey scandal!
xp84 2y ago
“who does the ai work for” I think even ignoring the more scary government/cops questions, this gets to a key problem. Because for 25 years we’ve taught 2 generations of digital natives as well as everybody else that everything Internet should be “free as in beer” paid for by trashy advertising, and that consumers’ only cash cost should be hardware and ISP/data plan. Therefore the answer to “who’s going to foot the bill for that AI” “must” be Big Adtech meaning the AI 100% will work for them and often directly against users’ interests (just as the YouTube algorithm mindlessly but intentionally prefers to radicalize a user or drive them to obsession on any number of topics vs. having them arrive at a healthy level of satisfaction and then sign off and go outside). In my opinion a lot of these problems we see in the scary law enforcement scenarios would be easier to solve if we didn’t expect everything to be ad-supported “free” and rather, we could be convinced to buy a $5000 piece of hardware for your home, that you control, that was privy to your encryption keys and performed all the power-insensitive AI processing for your family. That sounds a lot but compared to things like cars that people happily finance for $70,000 and smartphones which cost $1300 it is only weird because we aren’t used to it.
- CatWChainsaw 2y ago
  Well, now it isn't even that we can't be convinced. Companies have decided that they like this way of operating so they are refusing to offer what you propose. They're more than happy to double-dip and lock people into hard-to-cancel subscriptions while continuing to hoover as much data as they can, though. But even if they did offer what you suggest, the next move would be to keep people poor so they wouldn't want to "waste" money on that option anyway.

269 points by chmaynard 2y ago

username223 2y ago

[flagged]

dingnuts 2y ago
fucking hell are we only now having this discussion? when GitHub Copilot first released, a few years ago, my first reaction was: that's a keylogger! it sends everything in your editor to Microsoft, isn't anyone worried about that at all? No, no one was. I despair
- AyyEye 2y ago
  "Keylogger" and "spyware" have become so normalized that they have disappeared from the public lexicon.
  - jazzyjackson 2y ago
    With apologies, 1960s: “I BETTER NOT SAY THAT OR THE GOVERNMENT WILL WIRETAP MY HOUSE” 2020s: “HEY WIRETAP, DO YOU HAVE A RECIPE FOR PANCAKES?”
airstrike 2y ago
> tl;dr -- Most AI stuff won't run at an acceptable speed on your computer ...today. There's no real reason we can't get acceptable speeds in the near future.
- deathanatos 2y ago
  Yes, but also no: surface area is a proxy for compute power, because it's a proxy to heat dissipation. Phones are also only passively air-cooled — which is like the absolute worst cooling mechanism, too¹. So yes, processors do get better, but watts are watts, and there's a hard upper limit to how many watts a phone can consume, and you have to get better within that limit. Or … IDK, maybe you don't. Recent Android phones are now capable of prompting the user to actively cool them off because the ~bloat~ software is consuming too much compute/power. What basically translates to "help I'm dying" is an amusing message, but also a depressing state of affairs. ¹compare to a laptop, or worse for the phone, a desktop. The comparative cooling surface area is multiple times larger. And both have fans, desktops can be liquid cooled — it's not mere chance that they're faster, and phone form factor literally presents challenges. Even with an infinite battery, or a USB capable, you can't dissipate 60 W into a human hand.
  - airstrike 2y ago
    I agree but I was replying to the comment about computers specifically, meaning desktop and laptop form factors rather than mobile phones, especially when I say "near future"
deleted 2y ago
[deleted]

crackalamoo 2y ago

See also CrypTen, Meta's library for privacy preserving machine learning: https://github.com/facebookresearch/CrypTen https://github.com/facebookresearch/CrypTen. This isn't fully homomorphic encryption, but it is multi-party computation (MPC), which hides the inputs from the company owning the model. But while not revealing user input, it would still reveal the outputs of the model to the company. And yeah, as the article mentions, unfortunately this kind of thing (MPC or fully-homomorphic encryption) probably won't be feasible for the most powerful ML models.

EGreg 2y ago

I heard that homomorphic encryption can actually preserve all the operations in neural networks, since they are differentiable. Is this true? What is the slowdown in practice?

crackalamoo 2y ago
This is true in principle, yes. In practice, the way this usually works is by converting inputs to bits and bytes, and then computing the result as a digital circuit (AND, OR, XOR). Doing this encrypted is very slow: without hardware acceleration or special tricks, running the circuit is 1 million times slower than unencrypted, or about 1ms for a single gate. (https://www.jeremykun.com/2024/05/04/fhe-overview/ https://www.jeremykun.com/2024/05/04/fhe-overview/) When you think about all the individual logic gates involved in just a matrix multiplication, and scale it up to a diffusion model or large transformer, it gets infeasible very quickly.
- j2kun 2y ago
  There are FHE schemes that do better than binary gates (cf. CKKS) but they have other problems in that they require polynomial approximations for all the activation functions. Still they are much better than the binary-FHE schemes for stuff like neural networks, and most hardware accelerators in the pipeline right now are targeting CKKS and similar for this reason. For some numbers, a ResNet-20 inference can be done in CKKS in like 5 minutes on CPU. With custom changes to the architecture you can get less than one minute, and in my view HW acceleration will improve that by another factor of 10-100 at least, so I'd expect 1s inference of these (still small) networks within the next year or two. LLMs, however, are still going to be unreasonably slow for a long time.

jFriedensreich 2y ago

I think this has also a silver lining. The E2E encryption movement especially for messenger apps was largely also used to silently lock users out of their own data and effectively prevent user agency to use their own data to move apps, write automations or archive, this is not just true for whatsapp (the data export feature does not fully work since its launch and was just made to appease some EU law that did not properly check if the button works until the end.) Also signal does not have a way to do this. Maybe with ai coming into the game companies finally decide to provide access to data, I just hope it's in a transparent way with user opt in and user control.

do_not_redeem 2y ago
1. Is data encrypted in transit? 2. Can the user access their data at rest? Those two things are entirely orthogonal. I don't think you can extrapolate a trend from a few apps having bugs in their export code. Google Takeout is also notoriously buggy and they don't use E2E encryption. A more likely explanation is companies of all kinds don't care that much about export functionality, due to the incentives involved.
- jFriedensreich 2y ago
  you CAN extrapolate from nearly all e2e encrypted apps not giving a way to use the data. And there is a big difference between buggy google export features or facebook actively making export unusable to lock in users.
walrus01 2y ago
Signal does not have a way to manually export your private keys and chat history, but the process of "moving" your signal account to a new phone is quite straightforward. You put both devices on the same wifi/LAN layer 2 broadcast segment, start the transfer process in the app, input the verification codes displayed on the screen from both devices, and it sends everything over. This moves the private key in a way that does not result in all of your contacts receiving the scary "this person's key has changed" message.

_wire_ 2y ago

[flagged]

bobbiechen 2y ago

There is always going to be a gap between local-first processing and what can be achieved in a full-sized datacenter/cloud. That leads to the risks mentioned in the article. I wrote about Apple's Private Cloud Compute last year; for the foreseeable future, I still think server-side Confidential Computing is the most practical way to do processing without huge privacy risks: https://www.anjuna.io/blog/apple-is-using-secure-enclaves-to-ensure-ai-privacy-why-arent-you https://www.anjuna.io/blog/apple-is-using-secure-enclaves-to...

tonygiorgio 2y ago

> Although PCC is currently unique to Apple, we can hope that other privacy-focused services will soon crib the idea. IMHO, Apple's PCC is a step in the right direction in terms of general AI privacy nightmares where they are at today. It's not a perfect system, since it's not fully transparent and auditable, and I do not like their new opt-out photo scanning feature running on PCC, but there really is a lot to be inspired by it. My startup is going down this path ourselves, building on top of AWS Nitro and Nvidia Confidential Compute to provide end to end encryption from the AI user to the model running on the enclave side of an H100. It's not very widely known that you can do this with H100s but I really want to see this more in the next few years.

mnahkies 2y ago
I didn't actually realize that AWS supported this, I thought Azure was the only one offering it (https://azure.microsoft.com/en-us/blog/azure-confidential-computing-with-nvidia-gpus-for-trustworthy-ai/ https://azure.microsoft.com/en-us/blog/azure-confidential-co...) Are you speaking of this functionality? https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/ https://developer.nvidia.com/blog/confidential-computing-on-... (and am I just failing to find the relevant AWS docs?)
- tonygiorgio 2y ago
  Yes, you're correct on both, though I think Google Cloud recently started supporting it as well. AWS will likely have GPU enclave support with Trainium 2 soon (AFAIK, that feature is not publicly offered yet but could be wrong). We work with Edgeless Systems who manages the GPU enclave on Azure that we speak to from our AWS Nitro instance. While not ideal, the power of enclaves and the attestation verification process, we at least know that we're not leaking privacy by going with a third party GPU enclave provider.
  - calum-bird 2y ago
    Do you have any resources you can share on managing the enclave’s public key? We found documentation to be sparse.

bee_rider 2y ago

The author helpfully emphasized the interesting question at the end > This future worries me because it doesn’t really matter what technical choices we make around privacy. It does not matter if your model is running locally, or if it uses trusted cloud hardware — once a sufficiently-powerful general-purpose agent has been deployed on your phone, the only question that remains is who is given access to talk to it. Will it be only you? Or will we prioritize the government’s interest in monitoring its citizens over various fuddy-duddy notions of individual privacy. I do think there are interesting policy questions there. I mean it could hypothetically be mandated that the government must be given access to the agent (in the sense that we and these companies exist in jurisdictions that can pass arbitrary laws; let’s skip the boring and locale specific discussion of whether you think your local government would pass such a law). But, on a technical level—it seems like it ought to be possible to run an agent locally, on a system with full disk encryption, and not allow anyone who doesn’t have access to the system to talk with it, right? So on a technical level I don’t see how this is any different from where we were previously. I mean you could also run a bunch of regex’s from the 80’s to find whether or not somebody has, whatever, communist pamphlets on their computers. There’s always been a question of whether the government should be able to demand access to your computer. I guess it is good to keep in mind that if they are demanding access to an AI agent that ran on your computer, they are basically asking for a lossy record of your entire hard drive.

_boffin_ 2y ago
Unreasonable search?
- bee_rider 2y ago
  > (in the sense that we and these companies exist in jurisdictions that can pass arbitrary laws; let’s skip the boring and locale specific discussion of whether you think your local government would pass such a law) Anyway the idea of what’s a reasonable search in the US has been whittled away to almost nothing, right? “The dog smelled weed on your hard drive.” - A cop, probably.
- chgs 2y ago

walrus01 2y ago

It's a good thing that encrypted data at rest on your local device is inaccessible to cloud based "AI" tools. The problem is that your average person will blithely click "yes/accept/proceed/continue/I consent" on pop up dialogs in a GUI and agree to just about any Terms of Service, including decrypting your data before it's sent to some "cloud" based service. I see "AI" tools being used even more in the future to permanently tie people to monthly recurring billing services for things like icloud, microsoft's personal grade of office365, google workspace, etc. You'll pay $15 a month forever, and the amount of your data and dependency on the cloud based provider will mean that you have no viable path to ever stop paying it without significant disruption to your life.

Animats 2y ago

> Who does your AI agent actually work for? Yes. I made that point a few weeks ago. The legal concept of principal and agent applies. Running all content through an AI in the cloud to check for crimethink[1] is becoming a reality. Currently proposed: - "Child Sexual Abuse Material", which is a growing category that now includes AI-generated images in the US and may soon extend to Japanese animation. - Threats against important individuals. This may be extended to include what used to be considered political speech in the US. - Threats against the government. Already illegal in many countries. Bear in mind that Trump likes to accuse people of "treason" for things other than making war against the United States. - "Grooming" of minors, which is vague enough to cover most interactions. - Discussing drugs, sex, guns, gay activity, etc. Variously prohibited in some countries. - Organizing protests or labor unions. Prohibited in China and already searched for. Note that talking around the issue or jargon won't evade censorship. LLMs can deal with that. Run some ebonics or leetspeak through an LLM and ask it to translate it to standard English. Translation will succeed. The LLM has probably seen more of that dialect than most people. "If you want a vision of the future, imagine a boot stepping on a face, forever" - Orwell [1] https://www.orwell.org/dictionary/ https://www.orwell.org/dictionary/

iugtmkbdfil834 2y ago
A cynic in me is amused at the yet unknown corporation being placed under investigation due to a trigger phrase in one of the meetings transcribed incorrectly. Your point is worth reiterating.
- Terr_ 2y ago
  Or poisoned-data that sets up a trap, so that a system will later confabulate false innocence or guilt when certain topics or targets come up.
crooked-v 2y ago
"Grooming" in particular is the angle that Republicans want to use to illegalize any kind of gender-nonconforming behavior, up to and including desired states like "women wearing pants is crossdressing, and doing so around children is a felony".

nashashmi 2y ago

The most depressing realization in all of this is that the vast treasure trove of data that we used to have in the cloud thinking it was not scannable even for criminal activity has now become a vector where we shall have thought police coming down upon us for simple ideas of dissent.

AlexandrB 2y ago
A lot of people tried to sound the alarm. It's not "the cloud", it's "other people's computers". And given that other people own these machines, their interests - whether commercial or ideological - will always come first.
- tokioyoyo 2y ago
  To be fair, most people understand that risk. It’s just it is very convenient in a lot of scenarios and some businesses might have not even started without it. Privacy is not that big of a concern for a big chunk of people. And they’re basically voting with their wallets.
  - like_any_other 2y ago
    "Voting with their wallets", where all the choices are picked by entities hostile to consumer privacy and autonomy, and then they mislead you about those choices. People who bought LG (and now most other, now "smart") TVs did not in any meaningful way "vote" to be spied on and support DRM - simply all the TVs in a store would spy and show ads, and not disclose any of it at time of sale.
    - tokioyoyo 2y ago
      Well, it shows how it’s not the biggest issue for an average customer, as it, in theory, can make the TV cheaper. If people cared about absence of ads that much, all social media would not be ad-ridden since people would stop using it.
      - deleted 2y ago

blueblimp 2y ago

> Yet this approach is obviously much better than what’s being done at companies like OpenAI, where the data is processed by servers that employees (presumably) can log into and access. No need for presumption here: OpenAI is quite transparent about the fact that they retain data for 30 days and have employees and third-party contractors look at it. https://platform.openai.com/docs/models/how-we-use-your-data https://platform.openai.com/docs/models/how-we-use-your-data > To help identify abuse, API data may be retained for up to 30 days, after which it will be deleted (unless otherwise required by law). https://openai.com/enterprise-privacy/ https://openai.com/enterprise-privacy/ > Our access to API business data stored on our systems is limited to (1) authorized employees that require access for engineering support, investigating potential platform abuse, and legal compliance and (2) specialized third-party contractors who are bound by confidentiality and security obligations, solely to review for abuse and misuse.

deleted 2y ago
[deleted]
chefandy 2y ago
I have to say — I’m kind of amazed that anyone would expect privacy out of chat bot companies and products. You’re literally having a “conversation” with the servers of companies that built their entire product line using other people’s professional and personal output whether they approved, or even knew about it or not. Less a “it’s better to ask for forgiveness than permission” sort of thing than a “we’d rather just not ask and be pretty cagey about it if they ask, and then if they prove it, tell them they tacitly agreed to it by not hiding it from us even though they had no way to know we were looking at it“ sort of thing. Frankly I’m astonished that open ai, specifically, promises as much as they do in their privacy policy. Based on their alleged bait-and-switch tactics quietly swapping out models or reducing compute for paying customers after the initial “gee wiz look at that” press cycle, I can’t imagine those privacy policies will have much longevity when the company gets a more stable footing… and whooops looks like they figured out how to extract the training data from the models! And it’s different data since we extracted it from the model so the old privacy policy doesn’t apply! Haha sorry, that’s business and we’re building a techno utopian society here, so you should feel honored to be included! You think Altman wouldn’t sell that in a heartbeat to try and fund some big moonshot product if they get clobbered in the marketplace? Never mind the sketchy girlfriend-in-an-app-class chatbots. Don’t get me wrong — I absolutely think the privacy SHOULD be there, but I’m just shocked that anyone would assume it was. Maybe I’m being overly cynical? These days when I think I might be, in the end, it seems I wasn’t being cynical enough.

lowbatt 2y ago

Maybe a little off topic, but is there a way for a distributed app to connect to one of the LLM companies (OpenAI, etc.) without the unencrypted data hitting an in-between proxy server? An app I'm building uses LLMs to process messages. I don’t want the unencrypted message to hit my server - and ideally I wouldn’t have the ability to decrypt it. But I can’t communicate directly from client -> LLM Service without leaking the API key.

simonw 2y ago
"But I can’t communicate directly from client -> LLM Service without leaking the API key." There is a way you can do that right now: the OpenAI WebRTC API introduced the idea of an "ephemeral key": https://platform.openai.com/docs/guides/realtime-webrtc https://platform.openai.com/docs/guides/realtime-webrtc This provides a way for your server to create a limited-time API key for a user which their browser can then use to talk to OpenAI's API directly without proxying through you. I love this idea, but I want it for way more than just the WebRTC API, and I'd like it for other API providers too. My ideal version would be a way to create an ephemeral API key that's only allowed to talk to a specific model with a specific pre-baked system prompt (and maybe tool configuration and suchlike) and that only works for a limited time and has a limited token budget.
- lowbatt 2y ago
  interesting, will check that out. thanks!
michaelmrose 2y ago
Will such processing be cheap enough to be done by a box that plugs into a customers router to handle such? Would they buy them? Notably not just for this use case but others
whyage 2y ago
Check out https://www.opaque.co/ https://www.opaque.co/

ozgune 2y ago

> Apple even says it will publish its software images (though unfortunately not the source code) so that security researchers can check them over for bugs. I think Apple recently changed their stance on this. Now, they say that "source code for certain security-critical PCC components are available under a limited-use license." Of course, would have loved it if the whole thing was open source. ;) https://github.com/apple/security-pcc/ https://github.com/apple/security-pcc/ > The goal of this system is to make it hard for both attackers and Apple employees to exfiltrate data from these devices. I think Apple is claiming more than that. They are saying 1/ they don't keep any user data (data only gets processed during inference), 2/ no privileged runtime access, so their support engineers can't see user data, and 3/ they make binaries and parts of the source code available to security researchers to validate 1/ and 2/. You can find Apple PCC's five requirements here: https://security.apple.com/documentation/private-cloud-compute/corerequirements https://security.apple.com/documentation/private-cloud-compu... Note: Not affiliated with Apple. We read through the PCC security guide to see what an equivalent solution would look like in open source. If anyone is interested in this topic, please hit me up at ozgun @ ubicloud . com.

saagarjha 2y ago
Some of the core elements of the boot process are not source available, unfortunately.

natch 2y ago

From Apple's document on Advanced Data Protection: >With Advanced Data Protection enabled, Apple doesn't have the encryption keys needed to help you recover your end-to-end encrypted data. Apple doesn't have the keys. Somebody else might. Somebody other than you. Also, I think they meant to say decryption keys, although they're probably just dumbing down terminology for the masses. >If you ever lose access to your account, you’ll need to use one of your account recovery methods "You'll need to use." Not "there is no way except to use." >Note: Your account recovery methods are never shared with or known to Apple. "shared with or known to Apple." Not "shared with or known to anyone else." The encryption is there, I believe that. I just don't know how many copies of the keys there are. If the only key is with me, it would be super easy for Apple to just say that. I believe that they have said that in the past, but the wording has now changed to this hyper-specific "Apple does not have the key" stuff.

deleted 2y ago
[deleted]
musicale 2y ago
As you suggest, the wording should be clarified to say that the key is never copied, is only stored on your device, is not accessible to others, etc.
- natch 2y ago
  Maybe they are unable to make that clarification, if it would be false.
- systoll 2y ago
  It does say > It’s protected with the new key which is controlled solely by the user’s trusted devices I think main thing they’re avoiding is an explicit guarantee that the key cannot be retrieved from your phone by a third party.

rglover 2y ago

> We are about to face many hard questions about these systems, including some difficult questions about whether they will actually be working for us at all. And how. I'd lean towards no. Where we're headed feels like XKEYSCORE on steroids. I'd love to take the positive, optimistic bent on this, but when you look at where we've been combined with the behavior of the people in charge of these systems (to be clear, not the researchers or engineers, but c-suite), hope of a neutral, privacy-first future seems limited.

deleted 2y ago
[deleted]
ActorNightly 2y ago
Given how politics and companies evolved, I actually trust those people in charge of XKEYSCORE systems more than ever. They may wear suits, but those people usually come from some military background, and have a sense of duty towards defending US, from threats both foreign and domestic, and historically have not really abused their powers no matter what the administration is. XKEYSCORE for example, wasn't really about hacking people, it was just about collecting mass metadata and building profiles, well within the legal system, and the blame should be on the companies that didn't provide privacy tools, because any big government could have build the same system. Meanwhile, the anti anti-establishment Republican Party since 2016 who cried about big tech turned out to be the biggest pro-establishment fans, giving Elmo an office in a white house and Zucc bending a knee to avoid prosecution. With these new systems, Id rather have smart people who only work in US defensive forces because of a sense of duty (considering they could get paid much more in the private sector) in charge.
- schmidtleonard 2y ago
  > well within the legal system It's not a search if we don't find anything, and it's not a seizure if we charge the money with the crime. These are court approved arguments, so they must be correct interpretations. Point is: modern bureaucrats have proven that they are absolutely willing to abuse power, even in the best of times when there is no real domestic political strife.

jrm4 2y ago

"The goal of encryption is to ensure that only two parties, the receiver and sender, are aware of the contents of your data. Thus, AI training on your data breaks this, because it's another party. You now don't have encryption." Thanks for coming to my blah blah blah

dialup_sounds 2y ago
The article has nothing to do with model training.

peppertree 2y ago

Is embeddings enough to preserve privacy? If I run the encoder/decoder on device and only communicate with server in embeddings?

nostradumbasp 2y ago
For a short time maybe, but that kind of activity might look like something interesting enough for someone to target you and remove all privacy from your entire existence.
hallh 2y ago
No, the original text can largely be recovered from embeddings[0] if you know which embedding model was used. [0] https://arxiv.org/abs/2310.06816 https://arxiv.org/abs/2310.06816

klik99 2y ago

> You might even convince yourself that these questions are “privacy preserving,” since no human police officer would ever rummage through your papers, and law enforcement would only learn the answer if you were (probably) doing something illegal. Something I've started to see happen but never mentioned is the effect automated detection has on systems: As detection becomes more automated (previously authored algorithms, now with large AI models), there's less cash available for individual case workers, and more trust at the managerial level on automatic detection. This leads to false positives turning into major frustrations since it's hard to get in touch with a person to resolve the issue. When dealing with businesses it's frustrating, but as these get more used in law enforcement, this could be life ruining. For instance - I got flagged as illegal reviews on Amazon years ago and spent months trying to make my case to a human. Every year or so I try to raise the issue again to leave reviews, but it gets nowhere. Imagine this happening for a serious criminal issue, with the years long back log on some courts, this could ruin someones life. More automatic detection can work (and honestly, it's inevitable) but it's got to acknowledge that false positives will happen and allocate enough people to resolve those issues. As it stands right now, these detection systems get built and immediately human case workers get laid off, there's this assumption that detection systems REPLACE humans, but it should be that they augment and focus human case workers so you can do more with less - the human aspect needs to be included in the budgeting. But the incentives aren't there, and the people making the decisions aren't the ones working the actual cases so they aren't confronted with the problem. For them, the question is why save $1m when you could save $2m? With large AI models making it easier and more effective to build automated detection I expect this problem to get significantly worse over the next years.

smallmancontrov 2y ago
The UK Post Office scandal is bone-chilling. Update this to a world where every corner of your life is controlled by a platform monopoly that doesn't even provide the most bare-bones customer service and yeah, this is going to get a lot worse before it gets better.
- Vampiero 2y ago
  And that's the early game. Imagine when AI will be monitoring all internet traffic and arresting people for thoughtcrime. What wasn't feasible to do before is now quite in reach and the consequences are dire. Though of course it won't happen overnight. First they will let AI encroach every available space (backed by enthusiastic techbros). THEN, once it's established, boom. Authoritarian police state dystopia times 1000. And it's not like they need evidence to bin you. They just need inference. People who share your psychological profile will act and speak and behave in a similar way to you, so you can be put in the same category. When enough people in that category are tagged as criminals, you will be too. All because you couldn't be arsed to write some boilerplate

deathanatos 2y ago

TFA makes some rather basic errors. First, > Prior to 2011, most cloud-connected devices simply uploaded their data in plaintext. > Around 2011 our approach to data storage began to evolve. […] began to roll out default end-to-end encryption […] This technology changed the way that keys are managed, to ensure that servers would never see the plaintext content of your messages. "changed the way that keys are managed" is at a confused contradiction with "uploaded their data in plaintext". If you're going from TLS → E2EE, then yeah, "changed the way keys are managed" miiight make sense, though that's not how I'd phrase it. Then later, > On the one hand they can (1) send plaintext off to a server, in the process resurrecting many of the earlier vulnerabilities that end-to-end encryption sought to close. Or else (2) they can limit their processing to whatever can be performed on the device itself. We're still confusing "transmit plaintext" with plaintext being available to the server; the clear option of "use TLS" is omitted. It doesn't really undermine the argument — the server would still have access to the data, and could thus maliciously train AI on it — but it is surprising for a "cryptographer". > For example, imagine that Apple keeps its promise to deliver messages securely, but then your (Apple) phone goes ahead and uploads (the plaintext) message content to a different set of servers where Apple really can decrypt it. Apple is absolutely using end-to-end encryption in the dullest technical sense… yet is the statement above really accurate? Is Apple keeping its broader promise that it “can’t decrypt the data”? No, no reasonable person would believe that (though I am sure that if the scenario ever came to be, Apple, or whoever, would likely argue "yes") since it would utterly scuttle the term "E2EE". If you say "Our product supports X", and then have to caveat away 100% of what makes X X, then it's just grift, plain and simple. (Now, whether grift sees regulatory action … well.) > Now imagine that some other member of the group — not you, but one of your idiot friends — decides to turn on some service that uploads (your) received plaintext messages to WhatsApp. > In general, what we’re asking here is a question about informed consent. I would sort of agree, but corporations will expose the consent here to the "friend", and then argue that because the friend consented to your data being uploaded, it is fine. An argument for privacy regulations. (I don't think you have to go through all this … work. Just upload the user's data. They'll complain, for a bit, but the market has already consolidated into at least an ologopoly, users have shown that, for the most part, they're going to keep using the product rather than leave, or else I'll be ending this comment with a free "2025 will be the Year of the Linux Desktop". What's gonna happen, regulation to ensure a free market remains free¹? Please. Cf. MS Recall, currently in the "complain" phase, but give it time, and we'll reach the "we heard your concerns, and we value your input and take your feedback with the utmost respect ram it down their throats" stage.) (¹free as in "dictated by the laws of supply & demand", not laissez-faire which is where the US will be headed for the next 4.) (and … 2011? I'd've said 2013 is when we found out the 4A meant way less than we thought it did, leading to the rise in massive adoption of TLS. Less so E2EE.)

lifeisstillgood 2y ago

So if I understand it 1. E2E encryption does work 2. But phones can send plaintext back to the cloud to get help doing AI things 3. And we tend not to know because it’s all “assisstance” But the solution like anything is pricing. I mean yet again (uber, Airbnb) billions of dollars of VC money is used as subsidy so my photos can get OCR’d. If phones said “hey for a dollar fifty I can work out what the road sign says behind your dogs head in 32 photos your mum sent you last week” I think we woukd see a different threat landscape This is - again - unsustainable cash spending distorting markets and common sense. If the market was “we can OCR and analyse these insurance claims” the. Things like privacy and encryption would be first class requirements and harder to sell and build. By spending a billion they can sell services to people without regulators to ask awkward questions and then they hope step 3. Profit. I short not even AI can spot patterns in encrypted data, it’s only when plaintext gets sent around in the hope of profit do we see a threat. That’s seems a simple fix if not an easy one

fragmede 2y ago

The article hinges on a bad assertion that > Apple can’t rely on every device possessing enough power to perform inference locally. This means inference will be outsourced to a remote cloud machine. If you go look at Apple's site https://www.apple.com/apple-intelligence/ https://www.apple.com/apple-intelligence/ and scroll down, you get: Apple Intelligence is compatible with these devices. iPhone 16 A18 iPhone 16 Plus A18 iPhone 16 Pro Max A18 Pro iPhone 16 Pro A18 Pro iPhone 15 Pro Max A17 Pro iPhone 15 Pro A17 Pro iPad Pro M1 and later iPad Air M1 and later iPad mini A17 Pro MacBook Air M1 and later MacBook Pro M1 and later iMac M1 and later Mac mini M1 and later Mac Studio M1 Max and later Mac Pro M2 Ultra If you don't have one of those devices, Apple did the obvious thing and disabled features on devices that don't have the hardware to do it. While Apple has this whole private server architecture, they're not sending iMessages off device for summarization, that's happening on device.

flossposse 2y ago

Green, (the author), makes an important point: > a technical guarantee is different from a user promise. [...] End-to-end encrypted messaging systems are intended to deliver data securely. They don’t dictate what happens to it next. Then Green seems to immediately forget the point they just made, and proceed to talk about PCC as if it were something other than just another technical guarantee. PCC only helps to increase confidence that the software running on the server is the software Apple intended to be there. It doesn't give me any guarantees about where else my data might be transferred from there, or whether Apple will only use it for purposes I'm okay with. PCC makes Apple less vulnerable to hacks, but doesn't make them any more transparent or accountable. In fact, to the extent that some hackers hack for pro-social purposes like exposing corporate abuse, increased security also serves as a better shield against accountability. Of course, I'm not suggesting that we should do away with security to achieve transparency. I am, however, suggesting that transparency, moreso than security, is the major unaddressed problem here. I'd even go so far as to say that the woeful state of security is enabled in no small part by lack of transparency. If we want AI to serve society, then we must reverse the extreme information imbalance we currently inhabit wherein every detail of each person's life is exposed to the service provider, but the service provider is a complete black-box to the user. You want good corporate actors? Don't let them operate invisibly. You want ethical tech? Don't let it operate invisibly. (Edit: formatting)

p4bl0 2y ago

This article should be accompanied by Five things privacy experts know about AI [1]. They pair really well and should probably link to each other. [1] https://news.ycombinator.com/item?id=42695628 https://news.ycombinator.com/item?id=42695628

fourthark 2y ago
(which discusses privacy issues on the training side)

ajb 2y ago

The real threat here is going to be when AI expands from being applied to accelerate the work of individuals, to being applied to the control of organisations. And it will be tempting to do that. We all know the limitations of managers, management hierarchies, metrics, OKRs etc. It's easy to think of a CEO deciding that all the communications between their employees should just be fed into an AI that they can query . (Ironically that would be easier to enforce if everyone was remote). It's quite possible that it would enable more effective organisations, as the CEO and upper level management can have a better idea of what is really happening. But it will reduce the already tenuous belief of the powerful that their ordinary staff are real human beings. And it will inevitably leak out from private organisations, as the executive class see no reason why they shouldn't have the same tools when running the country as when running a corporation. Advocates of mass surveillance like to point out that no human now needs to listen to your calls. But the real danger was never the guy in a drab suit transcribing your conversions from reel-to-reel tape. It was always the chief who could call for the dossier on anyone they were finding inconvenient, and have it looked at with an eye to making you never inconvenience them again. The full consequences of mass surveillance have not played out simply because no one had the means to process that much ad-hoc unstructured data. Now they do.

TeMPOraL 2y ago
> It's easy to think of a CEO deciding that all the communications between their employees should just be fed into an AI that they can query . (Ironically that would be easier to enforce if everyone was remote). This is already happening, whether the CEOs want it or not - when there's a legal issue requiring discovery, e-discovery software may be used to pull in all digital communications that can be accessed, and feed it all to AI for, among other things, sentiment analysis. Applications of GenAI for legal work, in general, is a hot topic in legal circles now.

1vuio0pswjnm7 2y ago

"You've probably also noticed Silicon Valley's enthusiasm to find applications for this tech." A "solution" looking for a "problem". Like cryptocurrency. Like the "metaverse". And so on. When the "solution" brings new problems SillyCon Valley then proclaims they have a solution to the problem they created. "Even if these firms don't quite know how AI will be useful to their customers, they've already decided that models are the future." More often than not the "customers" are advertisers. Other computer users are just ad targets.

TeMPOraL 2y ago
GenAI (and LLMs in particular) is a "solution looking for a problem" in the sense petroleum or electricity was, not in the sense cryptocurrency is. I.e. you take one look at it and it's 100% obvious it's a general technology that's going to be useful in some way for all kinds of work people do, and be doing in the near future, and itself will make possible things and endeavors that weren't possible before. The devil is always in the details, but assuming that GenAI might be important for your business is correct no matter what business you're in, so it's worth it to pay attention and test it sooner rather than later.
1vuio0pswjnm7 2y ago
"Artificial intelligence (AI), in its broadest sense, is intelligence exhibited by machines, particularly computer systems." "A cryptocurrency, crypto-currency, or colloquially, crypto, is a digital currency designed to work through a computer network that is not reliant on any central authority, such as a government or bank, to uphold or maintain it." "The metaverse is a loosely defined term referring to virtual worlds in which users represented by avatars interact,^[1] usually in 3D and focused on social and economic connection." "Electricity is the set of physical phenomena associated with the presence and motion of electric charge." "Petroleum is a naturally occurring yellowish-black liquid mixture." (Definitions taken from one of ChatGPTS's primary training sources.) A careful reader is expected to notice the first three have some charactersitics in common, specifically ones that the last two do not.

reader9274 2y ago

For some reason I felt like this article had a lot of promise to start, but turned into fluff and I learned nothing. It covered very basic ideas of both AI and encryption, which I didn't expect from such a highly regarded expert.

dathinab 2y ago

> They’ll order your food and find the best deals on shopping, swipe your dating profile, negotiate with your lenders, and generally anticipate your every want or need. Why should they do so? I mean seriously. There is more money to make in telling you that the AI will buy you the beast deal but instead buy premeditated (i.e. bought) "okay" looking deals instead. Similar dating apps and the related ecosystems has a long history of scamy behavior in all kind of ways as they want to keep you using the app. And people with money have always found ways to highlight themself more. I.e. there is more money to make in "swiping for you" in way which looks on the surface honest but isn't. etc. etc. There is basically always more money to make in systematically deceiving people as long as you do it well enough so that people don't notice, or they don't have a real/realistic choice, i.e. are forced by the circumstances. So the moment you take the human and human conscience/moral out of the loop and also have no transparency there is a 0% of this ending well if regulators don't preclude abuse. And AI is pretty effective at removing transparency and the humanity out of the loop. With how things currently look, especially in the US, they (edit: they==regulators) are more likely to do the opposite (edit: i.e. remove consumer protections).

CatWChainsaw 2y ago
It's like nobody actually learned anything from the Honey scandal!

xp84 2y ago

“who does the ai work for” I think even ignoring the more scary government/cops questions, this gets to a key problem. Because for 25 years we’ve taught 2 generations of digital natives as well as everybody else that everything Internet should be “free as in beer” paid for by trashy advertising, and that consumers’ only cash cost should be hardware and ISP/data plan. Therefore the answer to “who’s going to foot the bill for that AI” “must” be Big Adtech meaning the AI 100% will work for them and often directly against users’ interests (just as the YouTube algorithm mindlessly but intentionally prefers to radicalize a user or drive them to obsession on any number of topics vs. having them arrive at a healthy level of satisfaction and then sign off and go outside). In my opinion a lot of these problems we see in the scary law enforcement scenarios would be easier to solve if we didn’t expect everything to be ad-supported “free” and rather, we could be convinced to buy a $5000 piece of hardware for your home, that you control, that was privy to your encryption keys and performed all the power-insensitive AI processing for your family. That sounds a lot but compared to things like cars that people happily finance for $70,000 and smartphones which cost $1300 it is only weird because we aren’t used to it.

CatWChainsaw 2y ago
Well, now it isn't even that we can't be convinced. Companies have decided that they like this way of operating so they are refusing to offer what you propose. They're more than happy to double-dip and lock people into hard-to-cancel subscriptions while continuing to hoover as much data as they can, though. But even if they did offer what you suggest, the next move would be to keep people poor so they wouldn't want to "waste" money on that option anyway.