6 ms·

Improving Siri's Privacy Protection

92 points by epaga 7y ago

sandbags 7y ago
I'd have preferred that they asked me to grade my interactions but this seems to be the right move.
- olliej 7y ago
  I feel if they provided you with a way (on device) to provide a corrected version and/or mark individual transcription entirely on device, with the option to forward individual options. E.g. Wanting to “help Siri get better” shouldn’t have to be an all-or-nothing opt-in. Though obviously individual posts are also probably less likely to be sent. Or maybe simply sending the local model updates would be sufficient to improve things globally?
  - sirn 7y ago
    You can actually provide a corrected version to Siri by pressing the little "Tap to edit" under the transcript! I'm not sure if the correction ever get sent to Apple (maybe it's on-device training), but I've found after a while Siri accuracy improved a lot, e.g. no longer interpret "Log" as "Lock" when I said it.
    - dawnerd 7y ago
      Is there a way to do that for HomePod?
      - sirn 7y ago
        None that I’m aware of, sorry :(
MrMember 7y ago
I feel like I'm in bizarro world. It opens with: >At Apple, we believe privacy is a fundamental human right. We design our products to protect users’ personal data, and we are constantly working to strengthen those protections. And ends with Apple saying they will no longer store audio recordings listened to by third party contractors by default. If Apple cares about privacy, why was that not opt-in to begin with?
- bobwaycott 7y ago
  A couple simple, potential explanatory thoughts/ideas strike my mind: - All this digital privacy stuff is new to humanity - We've only been trying to figure it out for a couple decades - We're still learning—sadly, mostly through trial and error—all the ways in which people can come up with privacy-invading techniques and tools - Motivated actors are clever, and think up ways to invade, violate, and compromise privacy, often in ways software/hardware makers haven't already thought of—but seem obvious in retrospect - Apple's concern for privacy has emerged alongside its development of devices and software that can be used to invade privacy—they can only take action against a danger/threat after they've thought or become aware of it - Again, all this stuff is new. We used to run e-commerce and punch in credit cards without thinking about SSL certs—cos shit was new, shiny, and magical. We're all becoming increasingly aware of just how much privacy needs to be a core development premise of just about any technology. And we're currently stuck in a weird place of individuals and companies trying to increase privacy levels while other individuals, companies, and governments are trying to break it down, get backdoors, invade everyone's privacy and track them everywhere they go, etc. tl;dr — Respecting, protecting, and increasing digital privacy is a constant process.
  - izacus 7y ago
    Are you going to put the same post under Facebook's, Google's and some other news... or does this "It's all new!" cop-out count only for Apple? Remember, they plastered "What happens on your iPhone, stays on your iPhone" ads all over CES this years while they kept transcribing Siri audio. And they did that until they got caught. Just like they throttled iPhone performance until they got caught and THEN they finally told users about that. Why would they get a benefit of doubt?
joe_hoyle 7y ago
> Siri uses a random identifier — a long string of letters and numbers associated with a single device — to keep track of data while it’s being processed, rather than tying it to your identity through your Apple ID or phone number — a process that we believe is unique among the digital assistants in use today. For further protection, after six months, the device’s data is disassociated from the random identifier. Interesting, I thought I had heard it widely reported that Apple was keeping hold of audio records tagged with your Apple ID for 6 months, before anonymizing. That looks like it wasn't the case, and Apple was only tagging those recordings with a device ID, presumably to associate recordings with other recordings.
- mikestew 7y ago
  Yeah, "widely-reported" was The Verge. As John Gruber points out[0], The Verge wasn't wrong, but I can't say their reporting would give the average reader a good grasp on what was really going on. That would include myself: https://news.ycombinator.com/item?id=20724558 https://news.ycombinator.com/item?id=20724558 [0] https://daringfireball.net/2019/08/apple_siri_privacy https://daringfireball.net/2019/08/apple_siri_privacy
  - Terretta 7y ago
    From 2017, about the recording and tokenization steps: Siri records your queries too, but she doesn’t catalog them or provide access to the running list of requests. You can’t listen to your history of Siri interactions in Apple’s app universe. While Apple logs and stores Siri queries, they’re tied to a random string of numbers for each user instead of an Apple ID or email address. Apple deletes the association between those queries and those numerical codes after six months. Your Amazon and Google histories, on the other hand, stay there until you decide to delete them. http://themillenniumreport.com/2017/03/not-only-are-alexa-siri-echo-home-listening-to-everything-you-say/ http://themillenniumreport.com/2017/03/not-only-are-alexa-si... From Wired, “Apple finally reveals how long Siri keeps your data”, in 2013, about later disassociation from the tokens: Once the voice recording is six months old, Apple "disassociates" your user number from the clip, deleting the number from the voice file. But it keeps these disassociated files for up to 18 more months for testing and product improvement purposes. "Apple may keep anonymized Siri data for up to two years," Muller says "If a user turns Siri off, both identifiers are deleted immediately along with any associated data." https://www.wired.com/2013/04/siri-two-years/ https://www.wired.com/2013/04/siri-two-years/
1e-9 7y ago
"As a result of our review, we realize we haven’t been fully living up to our high ideals, and for that we apologize. " This is a good response by Apple. I hope the incident will motivate a higher level of proactive privacy protection and provide an example for others to follow. Apple is in a position to do much good. I think their privacy focus is a great business decision as it promotes an advantage their competitors will be hard pressed to follow as consumer privacy awareness and demand increases. It is still not enough though. If only Apple would become as aggressive about privacy as are entities such as Purism, The Tor Project, and the Electronic Frontier Foundation; individual privacy could improve in a big way very quickly.
professorTuring 7y ago
>>Third, when customers opt in, only Apple employees will be allowed to listen to audio samples of the Siri interactions. Our team will work to delete any recording which is determined to be an inadvertent trigger of Siri. Unless the government send a warrant, then we will share anything we have with them and we will do what they want to do. i.e.: "we want all audios from this zipcode".
- lern_too_spel 7y ago
  This is essentially what has happened to Chinese iPhone users except not just one zipcode but the entire country.
GeekyBear 7y ago
The concept of data collection being off by default, and explicitly asking users for permission to turn it on is exactly what I would expect from all of FAANG going forward. It's the default settings that matter most. Especially when the privacy permissions are frequently hidden away and defended by dark UI patterns intended to keep user from finding data access permissions and turning them off. Now that Apple is adopting this position, will Google, Facebook, Microsoft and Amazon follow suit?
starsinspace 7y ago
It's interesting how they talk about "doing as much on device as possible"... but Siri's voice recognition still works by sending the Siri request audio to an Apple server and doing the voice recognition there (only the trigger phrase "hey siri" is recognized on-device). Why isn't it all done on-device? I'm pretty sure even older iPhones have more than enough CPU-power for that.
- GeekyBear 7y ago
  Isn't this something that only recently has become possible, and still has an accuracy cost?
  - starsinspace 7y ago
    iOS already has a separate feature "offline dictation" which works on-device. I don't understand why that isn't used for Siri. Also about "recently"... back in the late 90s I had a desktop PC running Windows 98, I think it was a Pentium 166 MHz with 32 MB RAM. On it, I had a voice recognition program called "Dragon Naturally Speaking". It required a little training with my voice but after that, it worked remarkably well. And that was over 20 years ago on a PC with a - by today's standards - very primitive CPU. Decent voice recognition isn't new technology.
    - GeekyBear 7y ago
      The problem with Dragon was that it was so inaccurate that a moderately skilled typist could produce final corrected text much more quickly than they could dictate text and then make corrections. Looking online, this is a feature Apple is adding in this year's iOS and Google is adding to Pixel devices only so far, so I would expect to have to give them some time to get on device speech recognition working at all as a first step.
    - jbverschoor 7y ago
      Exactly. Also, smart Siri is so stupid, it recently started correcting the contact I want to call to someone else, even though it was properly recognized and transcribed. Funny thing is I call this person daily, and the other one was 6-7 years ago Frustrating as hell
dawnerd 7y ago
The inadvertent triggers is what worries me. My HomePod is always randomly saying things like “I’m sorry I didn’t get that” or “go ahead”. And creepily one night just said “I’m still here”. Ideally they’d fix that but in the mean time I’m glad I can at least rest a little easier knowing Apple isn’t listening unless I opt in (which I might consider if it helps them actually fix the false triggers). Maybe they need a dashboard that shows all the requests and let you mark which ones were false or wrong instead of having someone directly listen?
andrerm 7y ago
> users will be able to opt in Would you help make Siri better? Yes|No

92 points by epaga 7y ago

sandbags 7y ago

I'd have preferred that they asked me to grade my interactions but this seems to be the right move.

olliej 7y ago
I feel if they provided you with a way (on device) to provide a corrected version and/or mark individual transcription entirely on device, with the option to forward individual options. E.g. Wanting to “help Siri get better” shouldn’t have to be an all-or-nothing opt-in. Though obviously individual posts are also probably less likely to be sent. Or maybe simply sending the local model updates would be sufficient to improve things globally?
- sirn 7y ago
  You can actually provide a corrected version to Siri by pressing the little "Tap to edit" under the transcript! I'm not sure if the correction ever get sent to Apple (maybe it's on-device training), but I've found after a while Siri accuracy improved a lot, e.g. no longer interpret "Log" as "Lock" when I said it.
  - dawnerd 7y ago
    Is there a way to do that for HomePod?
    - sirn 7y ago
      None that I’m aware of, sorry :(

MrMember 7y ago

I feel like I'm in bizarro world. It opens with: >At Apple, we believe privacy is a fundamental human right. We design our products to protect users’ personal data, and we are constantly working to strengthen those protections. And ends with Apple saying they will no longer store audio recordings listened to by third party contractors by default. If Apple cares about privacy, why was that not opt-in to begin with?

bobwaycott 7y ago
A couple simple, potential explanatory thoughts/ideas strike my mind: - All this digital privacy stuff is new to humanity - We've only been trying to figure it out for a couple decades - We're still learning—sadly, mostly through trial and error—all the ways in which people can come up with privacy-invading techniques and tools - Motivated actors are clever, and think up ways to invade, violate, and compromise privacy, often in ways software/hardware makers haven't already thought of—but seem obvious in retrospect - Apple's concern for privacy has emerged alongside its development of devices and software that can be used to invade privacy—they can only take action against a danger/threat after they've thought or become aware of it - Again, all this stuff is new. We used to run e-commerce and punch in credit cards without thinking about SSL certs—cos shit was new, shiny, and magical. We're all becoming increasingly aware of just how much privacy needs to be a core development premise of just about any technology. And we're currently stuck in a weird place of individuals and companies trying to increase privacy levels while other individuals, companies, and governments are trying to break it down, get backdoors, invade everyone's privacy and track them everywhere they go, etc. tl;dr — Respecting, protecting, and increasing digital privacy is a constant process.
- izacus 7y ago
  Are you going to put the same post under Facebook's, Google's and some other news... or does this "It's all new!" cop-out count only for Apple? Remember, they plastered "What happens on your iPhone, stays on your iPhone" ads all over CES this years while they kept transcribing Siri audio. And they did that until they got caught. Just like they throttled iPhone performance until they got caught and THEN they finally told users about that. Why would they get a benefit of doubt?

joe_hoyle 7y ago

> Siri uses a random identifier — a long string of letters and numbers associated with a single device — to keep track of data while it’s being processed, rather than tying it to your identity through your Apple ID or phone number — a process that we believe is unique among the digital assistants in use today. For further protection, after six months, the device’s data is disassociated from the random identifier. Interesting, I thought I had heard it widely reported that Apple was keeping hold of audio records tagged with your Apple ID for 6 months, before anonymizing. That looks like it wasn't the case, and Apple was only tagging those recordings with a device ID, presumably to associate recordings with other recordings.

mikestew 7y ago
Yeah, "widely-reported" was The Verge. As John Gruber points out[0], The Verge wasn't wrong, but I can't say their reporting would give the average reader a good grasp on what was really going on. That would include myself: https://news.ycombinator.com/item?id=20724558 https://news.ycombinator.com/item?id=20724558 [0] https://daringfireball.net/2019/08/apple_siri_privacy https://daringfireball.net/2019/08/apple_siri_privacy
- Terretta 7y ago
  From 2017, about the recording and tokenization steps: Siri records your queries too, but she doesn’t catalog them or provide access to the running list of requests. You can’t listen to your history of Siri interactions in Apple’s app universe. While Apple logs and stores Siri queries, they’re tied to a random string of numbers for each user instead of an Apple ID or email address. Apple deletes the association between those queries and those numerical codes after six months. Your Amazon and Google histories, on the other hand, stay there until you decide to delete them. http://themillenniumreport.com/2017/03/not-only-are-alexa-siri-echo-home-listening-to-everything-you-say/ http://themillenniumreport.com/2017/03/not-only-are-alexa-si... From Wired, “Apple finally reveals how long Siri keeps your data”, in 2013, about later disassociation from the tokens: Once the voice recording is six months old, Apple "disassociates" your user number from the clip, deleting the number from the voice file. But it keeps these disassociated files for up to 18 more months for testing and product improvement purposes. "Apple may keep anonymized Siri data for up to two years," Muller says "If a user turns Siri off, both identifiers are deleted immediately along with any associated data." https://www.wired.com/2013/04/siri-two-years/ https://www.wired.com/2013/04/siri-two-years/

1e-9 7y ago

"As a result of our review, we realize we haven’t been fully living up to our high ideals, and for that we apologize. " This is a good response by Apple. I hope the incident will motivate a higher level of proactive privacy protection and provide an example for others to follow. Apple is in a position to do much good. I think their privacy focus is a great business decision as it promotes an advantage their competitors will be hard pressed to follow as consumer privacy awareness and demand increases. It is still not enough though. If only Apple would become as aggressive about privacy as are entities such as Purism, The Tor Project, and the Electronic Frontier Foundation; individual privacy could improve in a big way very quickly.

professorTuring 7y ago

>>Third, when customers opt in, only Apple employees will be allowed to listen to audio samples of the Siri interactions. Our team will work to delete any recording which is determined to be an inadvertent trigger of Siri. Unless the government send a warrant, then we will share anything we have with them and we will do what they want to do. i.e.: "we want all audios from this zipcode".

lern_too_spel 7y ago
This is essentially what has happened to Chinese iPhone users except not just one zipcode but the entire country.

GeekyBear 7y ago

The concept of data collection being off by default, and explicitly asking users for permission to turn it on is exactly what I would expect from all of FAANG going forward. It's the default settings that matter most. Especially when the privacy permissions are frequently hidden away and defended by dark UI patterns intended to keep user from finding data access permissions and turning them off. Now that Apple is adopting this position, will Google, Facebook, Microsoft and Amazon follow suit?

starsinspace 7y ago

It's interesting how they talk about "doing as much on device as possible"... but Siri's voice recognition still works by sending the Siri request audio to an Apple server and doing the voice recognition there (only the trigger phrase "hey siri" is recognized on-device). Why isn't it all done on-device? I'm pretty sure even older iPhones have more than enough CPU-power for that.

GeekyBear 7y ago
Isn't this something that only recently has become possible, and still has an accuracy cost?
- starsinspace 7y ago
  iOS already has a separate feature "offline dictation" which works on-device. I don't understand why that isn't used for Siri. Also about "recently"... back in the late 90s I had a desktop PC running Windows 98, I think it was a Pentium 166 MHz with 32 MB RAM. On it, I had a voice recognition program called "Dragon Naturally Speaking". It required a little training with my voice but after that, it worked remarkably well. And that was over 20 years ago on a PC with a - by today's standards - very primitive CPU. Decent voice recognition isn't new technology.
  - GeekyBear 7y ago
    The problem with Dragon was that it was so inaccurate that a moderately skilled typist could produce final corrected text much more quickly than they could dictate text and then make corrections. Looking online, this is a feature Apple is adding in this year's iOS and Google is adding to Pixel devices only so far, so I would expect to have to give them some time to get on device speech recognition working at all as a first step.
  - jbverschoor 7y ago
    Exactly. Also, smart Siri is so stupid, it recently started correcting the contact I want to call to someone else, even though it was properly recognized and transcribed. Funny thing is I call this person daily, and the other one was 6-7 years ago Frustrating as hell

dawnerd 7y ago

The inadvertent triggers is what worries me. My HomePod is always randomly saying things like “I’m sorry I didn’t get that” or “go ahead”. And creepily one night just said “I’m still here”. Ideally they’d fix that but in the mean time I’m glad I can at least rest a little easier knowing Apple isn’t listening unless I opt in (which I might consider if it helps them actually fix the false triggers). Maybe they need a dashboard that shows all the requests and let you mark which ones were false or wrong instead of having someone directly listen?

andrerm 7y ago

> users will be able to opt in Would you help make Siri better? Yes|No