3 ms·

If we can't trust independent audits of code and hardware, what can we trust?

jeroenhd 2y ago
I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty. I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either. Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
- ReverseCold 2y ago
  > I have no idea what code is running on a server I can't access. That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.) The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction. E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740 https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
  - jeroenhd 2y ago
    I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that. Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware. It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
- BlobberSnobber 2y ago
  Exactly, Apple has barely any oversight or accountability for their privacy claims. Sad to see so many people taking their word at face value.
ENGNR 2y ago
Open code, inspected and used by a large number of users, hosted on hardware you physically control
- ryr11 2y ago
  I think that's fair, but impractical for most users. I have a number of Home Assistant integrations with locally hosted AI models for smart home features, but I wouldn't expect my grandma to set up a server and a few VMs when she could just give her HomePod a prompt that works with AI and have no worries about the implementation. Do you feel like Apple's "independent" auditing is insufficient?
  - ENGNR 2y ago
    > Do you feel like Apple's "independent" auditing is insufficient? Yeah, pretty much Also, your grandma might not setup a VM, but it sounds like the off-device processing is essentially stateless, or at most might have a very lightweight session. It seems like the kind of thing one person could setup for their family (with the same tamper-proof signatures, plus physical security), or provide a privacy focused appliance for anyone to just plug into a wall, if they wanted to.
- threeseed 2y ago
  Most open source code isn't inspected though. There have been many cases recently of compromised code being in the wild for quite some time and then only known about by accident.
  - spywaregorilla 2y ago
    100% of closed code is not inspected
    - sodality2 2y ago
      By you. Three comments above references "independent audits". Meaning professional cybersecurity firms
l33tman 2y ago
I think it boils down to that it doesn't matter what they promise, if you send a videocap of all you ever do on your computer to some company on the internet, you just have to take your chances. Would you put mics and cameras in all of your rooms in your home that send data to Apple (or someone else) to analyze "for your benefit" even if they say and promise they won't do anything bad with the feeds? At least with gmail and chat clients etc. things are somewhat put in compartments, one of the services might screw up and do something with your emails but your Messenger or WhatsApp chats are not affected by that, or vice versa. But when you bake it into the OS (laptop or phone) you're IMHO taking a much bigger risk, no matter what the intentions are.
- simondotau 2y ago
  There is nothing which Apple Intelligence can do that a hypothetically evil Apple couldn't have done before, given sufficiently treacherous code in their operating systems. Thus if you use an Apple device, you're already trusting Apple to not betray you. These new features don't increase the number of entities one must place their trust in. Whereas with apps like Gmail and WhatsApp on an iPhone, you must trust Google and Meta in addition to Apple, not in place of Apple. It doesn't distribute who you trust, it multiplies it.
  - l33tman 2y ago
    I still think it's a big difference between trusting existing OS'es and apps, which are under scrutiny by hundreds of security researchers and thousands of security nerds all the time, and willingly sending away all your data to a party who promises they will treat it well (I know it doesn't work like this in this case, but just for the sake of argument). In essence, what you're doing is training an assistant to learn all of your details of your life and habits and the question is if that "assistant" is really secure forever. Taken to the extreme, the assistant becomes a sort of "backup" of yourself. But yeah it's an individual decision with the pro's and con's of this.
    - simondotau 2y ago
      If you were already in on iCloud, that one residual distinction is moot.

I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty. I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either. Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.

ReverseCold 2y ago
> I have no idea what code is running on a server I can't access. That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.) The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction. E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740 https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
- jeroenhd 2y ago
  I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that. Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware. It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
BlobberSnobber 2y ago
Exactly, Apple has barely any oversight or accountability for their privacy claims. Sad to see so many people taking their word at face value.

ENGNR 2y ago

Open code, inspected and used by a large number of users, hosted on hardware you physically control

ryr11 2y ago
I think that's fair, but impractical for most users. I have a number of Home Assistant integrations with locally hosted AI models for smart home features, but I wouldn't expect my grandma to set up a server and a few VMs when she could just give her HomePod a prompt that works with AI and have no worries about the implementation. Do you feel like Apple's "independent" auditing is insufficient?
- ENGNR 2y ago
  > Do you feel like Apple's "independent" auditing is insufficient? Yeah, pretty much Also, your grandma might not setup a VM, but it sounds like the off-device processing is essentially stateless, or at most might have a very lightweight session. It seems like the kind of thing one person could setup for their family (with the same tamper-proof signatures, plus physical security), or provide a privacy focused appliance for anyone to just plug into a wall, if they wanted to.
threeseed 2y ago
Most open source code isn't inspected though. There have been many cases recently of compromised code being in the wild for quite some time and then only known about by accident.
- spywaregorilla 2y ago
  100% of closed code is not inspected
  - sodality2 2y ago
    By you. Three comments above references "independent audits". Meaning professional cybersecurity firms

l33tman 2y ago

I think it boils down to that it doesn't matter what they promise, if you send a videocap of all you ever do on your computer to some company on the internet, you just have to take your chances. Would you put mics and cameras in all of your rooms in your home that send data to Apple (or someone else) to analyze "for your benefit" even if they say and promise they won't do anything bad with the feeds? At least with gmail and chat clients etc. things are somewhat put in compartments, one of the services might screw up and do something with your emails but your Messenger or WhatsApp chats are not affected by that, or vice versa. But when you bake it into the OS (laptop or phone) you're IMHO taking a much bigger risk, no matter what the intentions are.

simondotau 2y ago
There is nothing which Apple Intelligence can do that a hypothetically evil Apple couldn't have done before, given sufficiently treacherous code in their operating systems. Thus if you use an Apple device, you're already trusting Apple to not betray you. These new features don't increase the number of entities one must place their trust in. Whereas with apps like Gmail and WhatsApp on an iPhone, you must trust Google and Meta in addition to Apple, not in place of Apple. It doesn't distribute who you trust, it multiplies it.
- l33tman 2y ago
  I still think it's a big difference between trusting existing OS'es and apps, which are under scrutiny by hundreds of security researchers and thousands of security nerds all the time, and willingly sending away all your data to a party who promises they will treat it well (I know it doesn't work like this in this case, but just for the sake of argument). In essence, what you're doing is training an assistant to learn all of your details of your life and habits and the question is if that "assistant" is really secure forever. Taken to the extreme, the assistant becomes a sort of "backup" of yourself. But yeah it's an individual decision with the pro's and con's of this.
  - simondotau 2y ago
    If you were already in on iCloud, that one residual distinction is moot.