16 ms·

I think the responses to this can be broken down into a 2x2 matrix: level of concern vs. understanding of technology. 1) Don't understand ML; not concerned - "

I think the responses to this can be broken down into a 2x2 matrix: level of concern vs. understanding of technology. 1) Don't understand ML; not concerned - "I have nothing to hide." 2) Don't understand ML; concerned - "I bought this device and now people are spying on me!" 3) Understand ML; not concerned - "Of course, Google needs to label its training data." 4) Understand ML; concerned - "How can we train models/collect data in an ethical way?" To me, category 3 is the most dangerous. Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. And as others have pointed out, this extends beyond home speakers to any voice-enabled device in general. In conversations about this with engineers the response I've gotten is essentially: "Just trust that we [Google/Amazon/etc.] handle the data correctly." This is worrying.

Spooky23 7y ago
The meta-issue in the United States is that once your data is accessible to a third party, you have no sovereignty over it, and abuse by private actors is "agreed to" by click-wrap and access by government actors is a simple subpoena. The law needs to catch up. Sharing should require specific informed consent and legislation needs to establish a scope where data stored as a "tenant" on a third party server is given 4th amendment protection.
ovi256 7y ago
This classification is very useful to discuss this issue. The difference between 3 and 4, noble as it is, can be caused by feasability concerns that push people into 3, not just ignorance of the privacy impact. Human labelling of training data sets is a big thing in supervised learning. Methods that dispense with this would be valuable for purely economic reasons beyond privacy - the cost of human labelling of data samples. Yet we don't have them! Techniques like federated learning or differential privacy can train models on opaque (encrypted or unavailable) data. This is nice, but they assume too much: that the data is already validated and analyzed. In real life modelling problems, one starts with an exploratory data analysis, the first step being looking at data samples. Opaque encrypted datasets also stop ML engineers from doing error analysis (look at your errors to better target model/dataset improvements) which is an even bigger issue, IMO, as error analysis is crucial when iterating on a model. Even for an already productivized model, one has to do maintenance work like checking for concept drift, which I can't see how to do on an opaque dataset.
- joey_bob 7y ago
  Handling the data in an ethical way doesn't need to be handling the data in an completely anonymous fashion. That would be one solution, but you can also create a tust-based system for how the data being labeled is handled, similar to HPAA. In addition, there are simple operational methods that could help ensure the data is processed as close to anonymously as possible. For example with voice data, you could filter the voices, work with the data in segments, and ensure that metadata for the samples is only accesible by trusted individuals certified under the above framework.
  - blackflame7000 7y ago
    In trust-based systems like HIPPA or Clearances, there is a fundamental aspect of requiring 2 conditions to access data: privilege, and the necessity to know. Taking data and mining for valuable insights isn't a "need to know" it's a "need to discover something unknown". This is where the security breaks down. In a conventional HIPPA system, only your doctor needs to access your info. You don't have to worry about some other doctors accessing your information in bulk to try and conduct a study on cancer rates. They don't NEED to know your info, they just WANT to know. When you WANT to know how to accurately fingerprint people by their voice, then obfuscating it is counterproductive.
    - vharuck 7y ago
      >You don't have to worry about some other doctors accessing your information in bulk to try and conduct a study on cancer rates. This not only happens, it's my job (though I'm not a doctor). Of course, it's tightly controlled on my end. I work for the government, but health systems have their own analysts. As part of my job, I have access to sensitive and identifying information. This isn't to be contrairian. There are existing systems using very personal data in bulk for analysis. The wheel doesn't need reinvented.
- whatshisface 7y ago
  It's not wrong for humans to label training data. It's wrong to let humans listen to voice recordings that users believed would be between them and a computer. The solutions are obvious: sell the things with a big sticker that says, "don't say anything private in earshot," revert to old fashioned research methods where you pay people to participate in your studies and get their permission, or ask people for permission to send in mis-heard commands like how Ubuntu asks me if I want to send them my core dumps.
ethbro 7y ago
I like that matrix! One thing that I think gets lost in engineers (and humans) is scale. Googazon doing {thing} might be "meh" for 10 people. But the implications look very different when it's doing {thing} for 10%+ of a country's population. At 10 people, I may find out Ted likes to eat Italian. At 10%, I may find out an Italian chain has a sudden health issue and short their stock. Which is in essence their original playbook: do things that only work at a scale that only we can play at.
- jancsika 7y ago
  > At 10%, I may find out an Italian chain has a sudden health issue and short their stock. Let's use scale at both ends please. At 10% of the population a nation state may ask Googazon to silently make changes to identify troublemakers.
  - ben_w 7y ago
    Here’s an entirely plausible scenario: The person in the story said they thought they heard domestic violence in some of the recordings. I know some people who are into “consensual nonconsent“, a form of BDSM which I do not understand, but as these acquaintances tell me they like being on the receiving end I have reason to trust them. Any system or person which incorrectly identifies one of these groups as the other, in either direction, has life-altering negative consequences. Note that in the UK, some forms of consensual BDSM have been prosecuted as serious assault, and the person on the receiving end has been prosecuted for conspiracy to commit assault because they consented to it. Any system which prevents DV information reaching the police is bad. Any system which reports BDSM to the police as DV is bad. I don’t even know what the relative frequencies of the two acts are, so I cannot even make a utilitarian ethical judgement.
    - outworlder 7y ago
      > Note that in the UK, some forms of consensual BDSM have been prosecuted as serious assault, and the person on the receiving end has been prosecuted for conspiracy to commit assault because they consented to it. Can someone versed in UK law clarify how such a stupid system emerged out of their laws? I'd guess that's an unintended consequence of how laws were worded, but it is mind boggling.
notinversed 7y ago
You're getting too lost in the weeds here. It doesn't matter if a company takes your data, does a poor job of anonymizing it, and then decides to label it as "training" for their "AI" or if they just stick it all in a flat .txt files and process it in Fortran. It's the exact same thing. You should be mad about the data being saved and used. Splitting hairs over implementation details to try and find a loophole is just a waste of time.
- flattone 7y ago
  1000000%
- skybrian 7y ago
  You could also compare this to hospitals sending voice files to India to be transcribed. This is not automated at all. It's not clear that hospitals are any better at getting informed consent for this than Google. https://en.m.wikipedia.org/wiki/Medical_transcription https://en.m.wikipedia.org/wiki/Medical_transcription
  - bogwog 7y ago
    1) A person going into a hospital, having their voice recorded, and then having the recording sent to another hospital where it might help treat and/or save their lives vs 2) A company exploiting the lack of regulation and public knowledge/education on the dangers of mining personal data, to mine personal data and make a profit with no regard for the safety of the individual If explicit consent had to be obtained, with the requirement that the person consenting be fully informed on the details of what they're giving up, in which of the scenarios above do you think people would be more likely to refuse consent?
    - skybrian 7y ago
      You raise safety concerns. What risks do you have in mind?
      - bogwog 7y ago
isostatic 7y ago
5) Understand ML; concerned - "Why do other people in the ML industry think it's OK to use and store peoples data in without informed consent (which are only those in group 3, and group 1+2 don't have informed consent)"
- delecti 7y ago
  That's group 4. How is worrying about informed consent not just a subset of concern about the ethics of collecting training data?
  - isostatic 7y ago
    No, group 4 is "we must collect training data, how do we get arround the ethical questions" Group 5 is "We must be ethical, can we still collect training data" Different priorities, different outcomes.
    - joshuamorton 7y ago
      I don't see the distinction between "if we wish to do a thing, we must do it ethically when doing so" (4) and "we must act ethically if we do a thing" (5). If training data can't be collected ethically, do you think group 4 still would, or something? That just seems like trying to out yourself on an ethical high horse without a real distinction in action.
      - yellowapple 7y ago
        The distinction (from how I'm reading the GP's comment) is that Group 4 presupposes that data collection is necessary, and seeks to minimize unethical means of collecting that data, while Group 5 presupposes that ethics are paramount, and seeks to establish whether or not data collection can actually be ethical at all. That is: Group 4 would be more willing to compromise ethics if absolutely necessary to get the data said group needs, while Group 5 would be more willing to compromise data collection if there's no ethical way to collect that data.
okmokmz 7y ago
>To me, category 3 is the most dangerous. Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. And as others have pointed out, this extends beyond home speakers to any voice-enabled device in general. Yes I'm frequently amazed how many coworkers I have that are still completely plugged into Google, Facebook, Amazon services/spyware, fill their homes with internet enabled "smart devices", have alexa/google assistance etc, and yet they act like I'm paranoid when I try to discuss security concerns or just flat out don't care. As much as I hate to say it, I think there needs to be a massive breach or abuse of power from one of these organizations/services that has severe real world consequences for those that utilize/support them. Until then nothing will change.
- Godel_unicode 7y ago
  It sounds like you're hoping for this as evidence that you were right to be concerned; have you considered that you might be wrong? What if your coworkers are right, and the risk is actually extremely low? How would you determine that?
  - okmokmz 7y ago
    >It sounds like you're hoping for this as evidence that you were right to be concerned Not at all, if the constant security breaches and lack of response from consumers, regulators, companies, etc. isn't enough evidence for you that there is a serious problem then you fall into the group I'm describing
    - joshuamorton 7y ago
      What constant security breaches? There are two major examples of security breaches I can think of that happened recently at amagoofaceoft: mis-stored passwords (fb/insta, and google for a short period earlier this year), and variants of the Cambridge analytics attacks, which steal public information but at scale. While those certainly aren't good, I wouldn't classify either as a security breach. The first was a loss of defense in depth, and the second, like I said, just got public info, but lots of it. Are you saying that breaches at other companies mean we shouldn't trust the big ones to be secure? Like because Equifax has terrible security practice, google by definition must also have bad security? Or...what? (I work at Google).
vincent-toups 7y ago
I understand ML and I'm concerned, but my position isn't "how can we train models/collect data in an ethical way?" Its more like "Do we really need to train models like this?" I continually come back to the same thought: The Amish (to take a random example) consume probably thousands of times less than I do. They do without the conveniences of technology which I "enjoy". They have neither computers nor an endless stream of distracting media and social interactions. Yet they are probably, on average, as happy as or happier than other Americans. These technologies don't really exist to serve people - they exist to create profit, and they do that by meeting superficial needs. There is great potential for machine learning to help the human race. But the vast majority of energy dedicated to the question by the information megaopolies isn't invested in that direction except under the most facile sorts of hypercapitalist justifications. We should, in short, regulate data collection and analysis into oblivion, with exceptions made only for democratically determined use cases which serve the public good.
- Rapzid 7y ago
  I'd like to offer a counterpoint to romanticizing the Amish way of life. Love it or hate it, our current world would not have developed the way it has if everyone was following the Amish way of life. They enjoy a massive number of benefits that were born out of alternative-to-them life styles. For example we may have "an endless stream of distracting media" but we also have a hugely increased life span and reduced mortality rate due to advances in modern science. To me, this is a bit like the "self-made" billionaire ignoring all the societal infrastructure afforded to them..
  - vincent-toups 7y ago
    You're not wrong, but also, I doubt it makes much of a difference in absolute happiness levels.
rndgermandude 7y ago
What we need is an AI to label the training data for ML to cut out the need for human workers to listen to people scream "Alexa" during sex. Or new laws to stop people from naming their kids Alexa or variations of it such as Alexander. I am sure we can whip up some such AI easily with some quantum computing. Preferably in a blockchain so we can verify correct operation and scale better. ducks
- RosanaAnaDana 7y ago
  Cloud. Data lake. Immutable.
iamnothere 7y ago
The Mycroft project has a better approach to this: "Mycroft uses opt-in privacy. This means we will only record what you say to Mycroft with your explicit permission. Don’t want us to record your voice? No problem! If you’d like us to help Mycroft become more accurate, you can opt in to have your voice anonymously recorded." (project is open source, at https://mycroft.ai/ https://mycroft.ai/) Let people participate in R&D if they want to, but don't force it.
- taneq 7y ago
  Also there's a huge difference between short anonymized voice clips and taking a transcript of your entire house's audio, as a complete dataset, with your name and address on it.
Rapzid 7y ago
> societal implications of those technologies Asserting your point of view as "educated" and "correct" while labeling people who don't share it as dangerous. Doesn't sound like a great way to start a discussion.
snowwrestler 7y ago
> Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. I think this goes well beyond tech workers. I think it's time for society to legally recognize the balance between the value of ML systems and the privacy concerns of customers of ML. Doctors and lawyers obviously should understand the value of privacy, but we, as a society, have also created legal rights and duties for them. Conversations with lawyers and doctors are legally privileged; at the same time, there are specific consequences for medical companies or lawyers who do not protect that information. Companies like Google, Apple, Amazon, etc. certainly have the resources, intelligence, and sophistication to comply with a similar regulatory regime. IMO it should be possible to construct a law that allows companies to collect, store, and tag customer data for purposes of training ML systems, but sets serious duties, with consequences, on them to do it right. Right now, what is to keep employees at these companies from abusing these systems to stalk, to surveil, to harass, or even just to feed their own curiosity? These data systems are core trade secrets for these companies, which means they are opaque to any kind of oversight from outside the company. The free market can't create the necessary balance because customers need information to make decisions--information that they don't have. The result will be an increasingly chaotic "hero/shithead rollercoaster" as customers make snap judgments based on scanty or wrong information about what these companies are actually doing. This is a classic case for regulation, which prevents a "race to the bottom" of sketchy practices for short term gain, while also protecting the ability of people and companies to use this technology to create value. Doing this right will help data-leveraging companies in the long run, just like attorney-client privilege and HIPAA have helped lawyers and doctors build trust (and therefore value) in their customer relationships.
novok 7y ago
Get real, obvious, informed consent by asking if you would like your voice prompts to be improved on / heard by real live humans as an opt in. I bet 1/500 of the population would opt in to it. And the first one to do it should be apple itself.
mejari 7y ago
>To me, category 3 is the most dangerous. Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. Do you think its possible to be educated on the societal implications of these technologies and still not be concerned? Seems like you've written your own viewpoint into the only "logical" one here.
- gringoDan 7y ago
  That's a fair point. I do think it is possible to be educated on the societal implications of these technologies and still not be concerned. I would disagree with that opinion, but it is certainly valid. Maybe these philosophical debates are going on behind closed doors. If so, this should be communicated to the public/end consumers. Much like in legal proceedings, the process itself is just as important, if not more so, than the outcome. That being said, based on conversations that I've had with people working on these very products, the interest level and incentive structure for engaging with the tech side of things far exceeds that of engaging with the broader societal implications. Creating the tech earns your salary, questioning its morality may get you fired. So many choose simply to not engage in the philosophical discussion, which to me is a big problem in the industry.
cblades 7y ago
I agree, but I think this issue is incredibly mishandled by reporting. The title in the linked article being a great example. There is absolutely no proof of number 2 in your list, but that is by far the widest-held belief. It's infuriating, because we can't have a useful societal dialog about the issue if the largest chunk of concerned people are, essentially, conspiracy theorist.
flatline 7y ago
I fall into (3) but not for the reason you mentioned. "Privacy" is largely an industrial age phenomenon and practically speaking we can never go back. I think we need new constructs to talk productively about issues like this and the research in this area hasn't even really started to take shape yet. We are only beginning to understand the implications of social graphs, information security, data as a product - let alone critically analyzing their intersection and having informed discussions thereabout.
- joaobeno 7y ago
  Privacy is not a Industrial Age stuff... I could be private in Rome, just as I'm now, just a random dude working... A small cog, no one gets to know me, no on cares to gossip about me... Be it now, be it on Renascence in Venice, be it on Rome just before the senators kill Caesar, be it on Luoyang as the tensions rose between the Warring States... At any point in time, I could have a private conversation, write and hide a document, like some stuff that was socially unacceptable and keep it a secret, meet someone in secret, and, in general, lead a private life... Stop comparing urban life as if it is something modern... Of course it is much more prevalent now, as the ratio vs rural residents shifted, and sure, it is much harder to hide something on a small village of some dozens of people, or at a farm with a huge family, but in any major city of any era, you could be as hidden and private as today, if not more...
- lostlogin 7y ago
  Privacy is not new and examples go back as far you you look. Curtains, doors, walls, envelopes, seals for envelopes. Some types and bits of clothing. Privacy is built into a awful lot of what people have done when you look for it.
RosanaAnaDana 7y ago
Category 3 checking in here. TBH its not the Googles I'm worried about. They're under a fair amount of scrutiny, but ultimately, I think its mostly chicken little screaming regarding the 'how' of how they'll use this data. The ones I'd be more concerned about are the ones not on your radar. Google knows security, anonymity, they have share-holders, governments know to regulate them, etc. What you really have to worry about is the companies getting data on you with out your permission that you have ZERO knowledge they are doing so, and as well, almost no ability to know they are taking data on you. The reality is, most people are almost completely ignorant of how much real world data on them is out in the wild with no google or facebook or any tech giant involved. I can't get into too much detail on it, but the data is there. You just have to be savvy enough to find it, process it, predict, and refine. In terms of morality? I don't know.
munificent 7y ago
> In conversations about this with engineers the response I've gotten is essentially: "Just trust that we [Google/Amazon/etc.] handle the data correctly." No one is afraid of power when it's in their own hands. A common failure mode is that people assume a given power that's in their hands today will always be.
Balgair 7y ago
>Tech workers have a responsibility not just (to) understand the technologies that they work with Ok, I agree completely with you, 100%. However, based on my limited worldview, tech workers barely understand the tech they work with at all [0]. Asking for the ethical implications to be mulled over is unlikely to happen considering the near-weekly HN threads on "interviewing sucks, heres how to fix it, lol". We can't even figure out how to hire someone let alone how to impedance-match with them on deep issues like ethical implications of ML/AI. [0] https://stackoverflow.com/ https://stackoverflow.com/
- wsinks 7y ago
  Your source is stack overflow?! XD that really tickled me. It's a great point.
Alex3917 7y ago
The reason companies use lots of live data is because people get upset when facial recognition doesn't work on people with different skin colors, voice recognition doesn't work for people with accents, etc. Doing #3 is literally the result of #4.
- JangoSteve 7y ago
  Keep in mind that there's no rule that says there must be a correct answer. If the only way to do something is approach A or approach B, and people don't like either approach, then other options include figuring out a new approach C or not doing the thing.
- PeterStuer 7y ago
  This is in no way 'the reason'. THE reason is that using live data without having to bother with consent is 'fast, easy and cheap'. Having to get consent is seen as a nuisance, and num's the word as we don't want to wake sleeping dogs. And we don't want to pay for generating training data, so yeah, we basically just steal the data from our uninformed users without their consent. I hope the GDPR will pursue this as a landmark case.
mbrumlow 7y ago
I don't think this is just a ML issue. Any device with a mic and connected to the internet could at any time be updates with new code that send audio back outside of what is needed for ML. I think that issue is much bigger than just training ML based off real world data. Laws might give you some recourse if you can prove this is happening or not. But really we need to start thinking about ways we can ensure the entire stack is under the users control. I have said this in the past. Free software and open sourc movements have worked decades to open up software and hardware for greater transparency and user control. Yet within a few years we have wiped that out with all the closed hardware and software we happily hold in out hands and place in our homes. We need more open source hardware and software projects offering alternatives to these systems that have become a daily part of our lifes.
rexpop 7y ago
It's such a relief to hear your conclusions, after so thorough an analysis.
axaxs 7y ago
Good breakdown. But I don't understand why any company thinks it is ethical to ever listen/record/send home data from -inside- a home. There are a plethora of public, or semi public places they could use to obtain real conversation(think of a subway, restaurants, DMV, etc). Set up a booth and let people talk to it. Using real speech from within someone's private quarters is disturbing IMO.
- bryan_w 7y ago
  I dunno if I agree with you assessment, if you buy a device that purposefully made to record your voice and send it to a company, I wouldn't find it unreasonable that the company could listen to what I sent to them. It's not like they are being sneaky about when it is recording or where the recordings are going.
Nelson69 7y ago
Anyone remember the 3d printed gun stuff from a few years back? I think this isn't very different from it. You can take these raw pieces and explain how they are simple and good and draw these simple ethical conclusions from them, but then you add it up and the bigger picture doesn't feel quite the same way. 3D printers are good, sharing 3D printing plans is good, it's good to help your neighbor, no regulations and we're experiencing tremendous growth in the 3D space, people are inventing new stuff, starting new businesses, etc.. all good stuff. but letting any jackass off the street print a working gun when we have how many mass shootings a year? People don't feel the same way. All the pieces are totally okay until you've got a more questionable global intention, and how can you regulate intention? Google using the data to train models is just a tool, it's a baby step, they aren't doing that to sell the models or in and of itself, they're doing it so that they can generate data that they might consider theirs and not yours from your voice data and then feed that in to other systems which generate tremendous profits for them in ways you don't even know. They have intended uses already. Is it a remotely fair question to talk about ethical training in this context without some idea as to the intended use and distribution of the meta data?
ousta 7y ago
you should be concerned that someone else is listening what you re saying in a private space 1 or 3 does not exist or they are not concerned because they dont have a google home
tgsovlerkhgsel 7y ago
Assuming categories 1 and 3 are sufficiently large (and I assume that is the case), this is easily resolved by allowing users to choose whether to donate their data for training or not. If the training already only happens on a 1/500 sample, skewing the sample towards "people who don't care about their privacy" will probably not significantly impact the quality of the data. I'm surprised this wasn't already the case, but hopefully the article will help the people responsible make better decisions in the trade-off between minimizing onboarding friction and respecting user's privacy in the future.
prepend 7y ago
There’s a different dimension that may or may not understand ML, but are cognizant that any data created will be viewed at least by the company that creates it. I fall into that category as I have time, nor do I trust any evaluation methods, to determine if a company is using my data ethically. If I create data and store it something that’s not mine, then I only do that in situations where I’m comfortable with the owner doing anything they want with it. I understand ML and know that Google has to at least use it for training. I’ve also worked on IT long enough that even in super tight controlled environments data are misused by administrators.
cronix 7y ago
I'm in the 5th category. 5) Understand ML; concerned - won't allow any of these things in my house period because they will always use them for things behind the scenes that they won't state. I don't care how well trained they are, or "ethical." Ethical... according to who, and at what time period in the future? Ethics change. The data they have on you won't. Look at all of the politicians and other people getting in trouble for things they said 15 years ago, which were generally more acceptable at the time but we've "progressed" since then. Who will be making decisions about you in the future based on last years data? Just don't give it to them.
- foobiekr 7y ago
  This. There’s nothing that binds today’s ethical company to voluntary ethical behavior when things get tough (or they get acquired).
  - JaggedNZ 7y ago
    There's nothing that stops the most ethical companies from being completely compromised by state actors either.
- 50656E6973 7y ago
  >won't allow any of these things in my house period because they will always use them for things behind the scenes that they won't state. As OP stated, "this extends beyond home speakers to any voice-enabled device in general". Unless you're living like the Unibomber, it's not a matter of "just don't give it to them". The moment you step outside your house and socialize with almost anyone anywhere, it's liable to being taken from you.
  - lm28469 7y ago
    Neatpick: "unabomber" as in UNABOM (University and Airline Bomber).
    - jonsen 7y ago
munchbunny 7y ago
I'm perhaps in a subcategory of (3) that falls under "Understand ML; concerned". Knowing what I know about how people I have worked with have come close to or have actually mishandled data despite the best of intentions, I do not trust any of these teams without an explicit accountability mechanism that is observable by an outside entity. I'm not looking to punish slip-ups, because mistakes happen, but I am looking for external enforcement to keep people honest. It's not that I think the engineers using this data are mustache twirling villains, it's that I think mishandling is inevitable due to inattention (yes, even you make mistakes!), and we have to design our data pipelines against that.
- drevil-v2 7y ago
  Exactly. Having worked in teams which handle personal data of consumers I know how easy it would be to misuse the privilege. The legal and marketing teams that come up with the jargon and slogans about privacy are so far removed from the day to day operations that they have no clue about the reality. I don't they would care even if they did.
m3at 7y ago
I'm in both 3 and 4. 4 because not being explicit about the practice is misleading at best, outsourcing the difficult task of keeping the analysis private show how unimportant it's considered, and because big techs have a tendency to decrease privacy over time. Using clients who paid for the product as a dataset generator is also wrong. But 3 at the same time because well, it's important to evaluate the performance of the product in the field not just in the lab. There were so many cases of catastrophic failures for ML models (ex. classifying black people as gorilla) that having a tight feedback loop is important. It has to be done right, but evaluating a product that was primarily developed for (or at least by) English speakers and transfered to other domains seem like the right thing to do. All in all, I don't and wouldn't use one of those assistants because 4 outweigh 3, but it's not binary.
wybiral 7y ago
Even if I trust them to do what they say they're doing with the data I may not trust every party who comes to possess that data. And I may not trust their possession/use of it in all future contexts - as their privacy policy slowly drifts into the unknown year after year. If they're collecting it in a way that can be requested by governments (for instance) or could be leaked by hackers that's another layer of valid "concern" not related to my understanding of the ML aspect of this.
neilpointer 7y ago
I'm between 3 and 4: I just want proof that they remove PII from the audio files. If it's a bunch of audio files with unique IDs and metadata like time of day, count me as a member of group 3.
userbinator 7y ago
I'd consider myself something like "4b": Understand ML; concerned - "DO NOT WANT!" The more I see of this tech, the more disturbed I am about its existence. In some ways, dystopia and utopia really are the same thing.
raghava 7y ago
Essentially, a larger grid, involving agent( tech, management ) # assuming management has power over tech worker understanding-of-ML( yes, no ) concerned-about-ethics-and-privacy( yes, no ) The below combinations are worst in terms of ethics. { agent[tech], understanding-of-ML[yes], concerned-about-ethics-and-privacy[no] } { agent[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] } { tech[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] }
jsharf 7y ago
I think there's a third dimension: how much you've given up. So my thoughts are like #4, but with the undertone of "who am I kidding, of course they don't care, what can I do?". This is perhaps the most dangerous response.

Spooky23 7y ago

The meta-issue in the United States is that once your data is accessible to a third party, you have no sovereignty over it, and abuse by private actors is "agreed to" by click-wrap and access by government actors is a simple subpoena. The law needs to catch up. Sharing should require specific informed consent and legislation needs to establish a scope where data stored as a "tenant" on a third party server is given 4th amendment protection.

ovi256 7y ago

This classification is very useful to discuss this issue. The difference between 3 and 4, noble as it is, can be caused by feasability concerns that push people into 3, not just ignorance of the privacy impact. Human labelling of training data sets is a big thing in supervised learning. Methods that dispense with this would be valuable for purely economic reasons beyond privacy - the cost of human labelling of data samples. Yet we don't have them! Techniques like federated learning or differential privacy can train models on opaque (encrypted or unavailable) data. This is nice, but they assume too much: that the data is already validated and analyzed. In real life modelling problems, one starts with an exploratory data analysis, the first step being looking at data samples. Opaque encrypted datasets also stop ML engineers from doing error analysis (look at your errors to better target model/dataset improvements) which is an even bigger issue, IMO, as error analysis is crucial when iterating on a model. Even for an already productivized model, one has to do maintenance work like checking for concept drift, which I can't see how to do on an opaque dataset.

joey_bob 7y ago
Handling the data in an ethical way doesn't need to be handling the data in an completely anonymous fashion. That would be one solution, but you can also create a tust-based system for how the data being labeled is handled, similar to HPAA. In addition, there are simple operational methods that could help ensure the data is processed as close to anonymously as possible. For example with voice data, you could filter the voices, work with the data in segments, and ensure that metadata for the samples is only accesible by trusted individuals certified under the above framework.
- blackflame7000 7y ago
  In trust-based systems like HIPPA or Clearances, there is a fundamental aspect of requiring 2 conditions to access data: privilege, and the necessity to know. Taking data and mining for valuable insights isn't a "need to know" it's a "need to discover something unknown". This is where the security breaks down. In a conventional HIPPA system, only your doctor needs to access your info. You don't have to worry about some other doctors accessing your information in bulk to try and conduct a study on cancer rates. They don't NEED to know your info, they just WANT to know. When you WANT to know how to accurately fingerprint people by their voice, then obfuscating it is counterproductive.
  - vharuck 7y ago
    >You don't have to worry about some other doctors accessing your information in bulk to try and conduct a study on cancer rates. This not only happens, it's my job (though I'm not a doctor). Of course, it's tightly controlled on my end. I work for the government, but health systems have their own analysts. As part of my job, I have access to sensitive and identifying information. This isn't to be contrairian. There are existing systems using very personal data in bulk for analysis. The wheel doesn't need reinvented.
whatshisface 7y ago
It's not wrong for humans to label training data. It's wrong to let humans listen to voice recordings that users believed would be between them and a computer. The solutions are obvious: sell the things with a big sticker that says, "don't say anything private in earshot," revert to old fashioned research methods where you pay people to participate in your studies and get their permission, or ask people for permission to send in mis-heard commands like how Ubuntu asks me if I want to send them my core dumps.

ethbro 7y ago

I like that matrix! One thing that I think gets lost in engineers (and humans) is scale. Googazon doing {thing} might be "meh" for 10 people. But the implications look very different when it's doing {thing} for 10%+ of a country's population. At 10 people, I may find out Ted likes to eat Italian. At 10%, I may find out an Italian chain has a sudden health issue and short their stock. Which is in essence their original playbook: do things that only work at a scale that only we can play at.

jancsika 7y ago
> At 10%, I may find out an Italian chain has a sudden health issue and short their stock. Let's use scale at both ends please. At 10% of the population a nation state may ask Googazon to silently make changes to identify troublemakers.
- ben_w 7y ago
  Here’s an entirely plausible scenario: The person in the story said they thought they heard domestic violence in some of the recordings. I know some people who are into “consensual nonconsent“, a form of BDSM which I do not understand, but as these acquaintances tell me they like being on the receiving end I have reason to trust them. Any system or person which incorrectly identifies one of these groups as the other, in either direction, has life-altering negative consequences. Note that in the UK, some forms of consensual BDSM have been prosecuted as serious assault, and the person on the receiving end has been prosecuted for conspiracy to commit assault because they consented to it. Any system which prevents DV information reaching the police is bad. Any system which reports BDSM to the police as DV is bad. I don’t even know what the relative frequencies of the two acts are, so I cannot even make a utilitarian ethical judgement.
  - outworlder 7y ago
    > Note that in the UK, some forms of consensual BDSM have been prosecuted as serious assault, and the person on the receiving end has been prosecuted for conspiracy to commit assault because they consented to it. Can someone versed in UK law clarify how such a stupid system emerged out of their laws? I'd guess that's an unintended consequence of how laws were worded, but it is mind boggling.

notinversed 7y ago

You're getting too lost in the weeds here. It doesn't matter if a company takes your data, does a poor job of anonymizing it, and then decides to label it as "training" for their "AI" or if they just stick it all in a flat .txt files and process it in Fortran. It's the exact same thing. You should be mad about the data being saved and used. Splitting hairs over implementation details to try and find a loophole is just a waste of time.

flattone 7y ago
1000000%
skybrian 7y ago
You could also compare this to hospitals sending voice files to India to be transcribed. This is not automated at all. It's not clear that hospitals are any better at getting informed consent for this than Google. https://en.m.wikipedia.org/wiki/Medical_transcription https://en.m.wikipedia.org/wiki/Medical_transcription
- bogwog 7y ago
  1) A person going into a hospital, having their voice recorded, and then having the recording sent to another hospital where it might help treat and/or save their lives vs 2) A company exploiting the lack of regulation and public knowledge/education on the dangers of mining personal data, to mine personal data and make a profit with no regard for the safety of the individual If explicit consent had to be obtained, with the requirement that the person consenting be fully informed on the details of what they're giving up, in which of the scenarios above do you think people would be more likely to refuse consent?
  - skybrian 7y ago
    You raise safety concerns. What risks do you have in mind?
    - bogwog 7y ago

isostatic 7y ago

5) Understand ML; concerned - "Why do other people in the ML industry think it's OK to use and store peoples data in without informed consent (which are only those in group 3, and group 1+2 don't have informed consent)"

delecti 7y ago
That's group 4. How is worrying about informed consent not just a subset of concern about the ethics of collecting training data?
- isostatic 7y ago
  No, group 4 is "we must collect training data, how do we get arround the ethical questions" Group 5 is "We must be ethical, can we still collect training data" Different priorities, different outcomes.
  - joshuamorton 7y ago
    I don't see the distinction between "if we wish to do a thing, we must do it ethically when doing so" (4) and "we must act ethically if we do a thing" (5). If training data can't be collected ethically, do you think group 4 still would, or something? That just seems like trying to out yourself on an ethical high horse without a real distinction in action.
    - yellowapple 7y ago
      The distinction (from how I'm reading the GP's comment) is that Group 4 presupposes that data collection is necessary, and seeks to minimize unethical means of collecting that data, while Group 5 presupposes that ethics are paramount, and seeks to establish whether or not data collection can actually be ethical at all. That is: Group 4 would be more willing to compromise ethics if absolutely necessary to get the data said group needs, while Group 5 would be more willing to compromise data collection if there's no ethical way to collect that data.

okmokmz 7y ago

>To me, category 3 is the most dangerous. Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. And as others have pointed out, this extends beyond home speakers to any voice-enabled device in general. Yes I'm frequently amazed how many coworkers I have that are still completely plugged into Google, Facebook, Amazon services/spyware, fill their homes with internet enabled "smart devices", have alexa/google assistance etc, and yet they act like I'm paranoid when I try to discuss security concerns or just flat out don't care. As much as I hate to say it, I think there needs to be a massive breach or abuse of power from one of these organizations/services that has severe real world consequences for those that utilize/support them. Until then nothing will change.

Godel_unicode 7y ago
It sounds like you're hoping for this as evidence that you were right to be concerned; have you considered that you might be wrong? What if your coworkers are right, and the risk is actually extremely low? How would you determine that?
- okmokmz 7y ago
  >It sounds like you're hoping for this as evidence that you were right to be concerned Not at all, if the constant security breaches and lack of response from consumers, regulators, companies, etc. isn't enough evidence for you that there is a serious problem then you fall into the group I'm describing
  - joshuamorton 7y ago
    What constant security breaches? There are two major examples of security breaches I can think of that happened recently at amagoofaceoft: mis-stored passwords (fb/insta, and google for a short period earlier this year), and variants of the Cambridge analytics attacks, which steal public information but at scale. While those certainly aren't good, I wouldn't classify either as a security breach. The first was a loss of defense in depth, and the second, like I said, just got public info, but lots of it. Are you saying that breaches at other companies mean we shouldn't trust the big ones to be secure? Like because Equifax has terrible security practice, google by definition must also have bad security? Or...what? (I work at Google).

vincent-toups 7y ago

I understand ML and I'm concerned, but my position isn't "how can we train models/collect data in an ethical way?" Its more like "Do we really need to train models like this?" I continually come back to the same thought: The Amish (to take a random example) consume probably thousands of times less than I do. They do without the conveniences of technology which I "enjoy". They have neither computers nor an endless stream of distracting media and social interactions. Yet they are probably, on average, as happy as or happier than other Americans. These technologies don't really exist to serve people - they exist to create profit, and they do that by meeting superficial needs. There is great potential for machine learning to help the human race. But the vast majority of energy dedicated to the question by the information megaopolies isn't invested in that direction except under the most facile sorts of hypercapitalist justifications. We should, in short, regulate data collection and analysis into oblivion, with exceptions made only for democratically determined use cases which serve the public good.

Rapzid 7y ago
I'd like to offer a counterpoint to romanticizing the Amish way of life. Love it or hate it, our current world would not have developed the way it has if everyone was following the Amish way of life. They enjoy a massive number of benefits that were born out of alternative-to-them life styles. For example we may have "an endless stream of distracting media" but we also have a hugely increased life span and reduced mortality rate due to advances in modern science. To me, this is a bit like the "self-made" billionaire ignoring all the societal infrastructure afforded to them..
- vincent-toups 7y ago
  You're not wrong, but also, I doubt it makes much of a difference in absolute happiness levels.

rndgermandude 7y ago

What we need is an AI to label the training data for ML to cut out the need for human workers to listen to people scream "Alexa" during sex. Or new laws to stop people from naming their kids Alexa or variations of it such as Alexander. I am sure we can whip up some such AI easily with some quantum computing. Preferably in a blockchain so we can verify correct operation and scale better. ducks

RosanaAnaDana 7y ago
Cloud. Data lake. Immutable.

iamnothere 7y ago

The Mycroft project has a better approach to this: "Mycroft uses opt-in privacy. This means we will only record what you say to Mycroft with your explicit permission. Don’t want us to record your voice? No problem! If you’d like us to help Mycroft become more accurate, you can opt in to have your voice anonymously recorded." (project is open source, at https://mycroft.ai/ https://mycroft.ai/) Let people participate in R&D if they want to, but don't force it.

taneq 7y ago
Also there's a huge difference between short anonymized voice clips and taking a transcript of your entire house's audio, as a complete dataset, with your name and address on it.

Rapzid 7y ago

> societal implications of those technologies Asserting your point of view as "educated" and "correct" while labeling people who don't share it as dangerous. Doesn't sound like a great way to start a discussion.

snowwrestler 7y ago

> Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. I think this goes well beyond tech workers. I think it's time for society to legally recognize the balance between the value of ML systems and the privacy concerns of customers of ML. Doctors and lawyers obviously should understand the value of privacy, but we, as a society, have also created legal rights and duties for them. Conversations with lawyers and doctors are legally privileged; at the same time, there are specific consequences for medical companies or lawyers who do not protect that information. Companies like Google, Apple, Amazon, etc. certainly have the resources, intelligence, and sophistication to comply with a similar regulatory regime. IMO it should be possible to construct a law that allows companies to collect, store, and tag customer data for purposes of training ML systems, but sets serious duties, with consequences, on them to do it right. Right now, what is to keep employees at these companies from abusing these systems to stalk, to surveil, to harass, or even just to feed their own curiosity? These data systems are core trade secrets for these companies, which means they are opaque to any kind of oversight from outside the company. The free market can't create the necessary balance because customers need information to make decisions--information that they don't have. The result will be an increasingly chaotic "hero/shithead rollercoaster" as customers make snap judgments based on scanty or wrong information about what these companies are actually doing. This is a classic case for regulation, which prevents a "race to the bottom" of sketchy practices for short term gain, while also protecting the ability of people and companies to use this technology to create value. Doing this right will help data-leveraging companies in the long run, just like attorney-client privilege and HIPAA have helped lawyers and doctors build trust (and therefore value) in their customer relationships.

novok 7y ago

Get real, obvious, informed consent by asking if you would like your voice prompts to be improved on / heard by real live humans as an opt in. I bet 1/500 of the population would opt in to it. And the first one to do it should be apple itself.

mejari 7y ago

>To me, category 3 is the most dangerous. Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. Do you think its possible to be educated on the societal implications of these technologies and still not be concerned? Seems like you've written your own viewpoint into the only "logical" one here.

gringoDan 7y ago
That's a fair point. I do think it is possible to be educated on the societal implications of these technologies and still not be concerned. I would disagree with that opinion, but it is certainly valid. Maybe these philosophical debates are going on behind closed doors. If so, this should be communicated to the public/end consumers. Much like in legal proceedings, the process itself is just as important, if not more so, than the outcome. That being said, based on conversations that I've had with people working on these very products, the interest level and incentive structure for engaging with the tech side of things far exceeds that of engaging with the broader societal implications. Creating the tech earns your salary, questioning its morality may get you fired. So many choose simply to not engage in the philosophical discussion, which to me is a big problem in the industry.

cblades 7y ago

I agree, but I think this issue is incredibly mishandled by reporting. The title in the linked article being a great example. There is absolutely no proof of number 2 in your list, but that is by far the widest-held belief. It's infuriating, because we can't have a useful societal dialog about the issue if the largest chunk of concerned people are, essentially, conspiracy theorist.

flatline 7y ago

I fall into (3) but not for the reason you mentioned. "Privacy" is largely an industrial age phenomenon and practically speaking we can never go back. I think we need new constructs to talk productively about issues like this and the research in this area hasn't even really started to take shape yet. We are only beginning to understand the implications of social graphs, information security, data as a product - let alone critically analyzing their intersection and having informed discussions thereabout.

joaobeno 7y ago
Privacy is not a Industrial Age stuff... I could be private in Rome, just as I'm now, just a random dude working... A small cog, no one gets to know me, no on cares to gossip about me... Be it now, be it on Renascence in Venice, be it on Rome just before the senators kill Caesar, be it on Luoyang as the tensions rose between the Warring States... At any point in time, I could have a private conversation, write and hide a document, like some stuff that was socially unacceptable and keep it a secret, meet someone in secret, and, in general, lead a private life... Stop comparing urban life as if it is something modern... Of course it is much more prevalent now, as the ratio vs rural residents shifted, and sure, it is much harder to hide something on a small village of some dozens of people, or at a farm with a huge family, but in any major city of any era, you could be as hidden and private as today, if not more...
lostlogin 7y ago
Privacy is not new and examples go back as far you you look. Curtains, doors, walls, envelopes, seals for envelopes. Some types and bits of clothing. Privacy is built into a awful lot of what people have done when you look for it.

RosanaAnaDana 7y ago

Category 3 checking in here. TBH its not the Googles I'm worried about. They're under a fair amount of scrutiny, but ultimately, I think its mostly chicken little screaming regarding the 'how' of how they'll use this data. The ones I'd be more concerned about are the ones not on your radar. Google knows security, anonymity, they have share-holders, governments know to regulate them, etc. What you really have to worry about is the companies getting data on you with out your permission that you have ZERO knowledge they are doing so, and as well, almost no ability to know they are taking data on you. The reality is, most people are almost completely ignorant of how much real world data on them is out in the wild with no google or facebook or any tech giant involved. I can't get into too much detail on it, but the data is there. You just have to be savvy enough to find it, process it, predict, and refine. In terms of morality? I don't know.

munificent 7y ago

> In conversations about this with engineers the response I've gotten is essentially: "Just trust that we [Google/Amazon/etc.] handle the data correctly." No one is afraid of power when it's in their own hands. A common failure mode is that people assume a given power that's in their hands today will always be.

Balgair 7y ago

>Tech workers have a responsibility not just (to) understand the technologies that they work with Ok, I agree completely with you, 100%. However, based on my limited worldview, tech workers barely understand the tech they work with at all [0]. Asking for the ethical implications to be mulled over is unlikely to happen considering the near-weekly HN threads on "interviewing sucks, heres how to fix it, lol". We can't even figure out how to hire someone let alone how to impedance-match with them on deep issues like ethical implications of ML/AI. [0] https://stackoverflow.com/ https://stackoverflow.com/

wsinks 7y ago
Your source is stack overflow?! XD that really tickled me. It's a great point.

Alex3917 7y ago

The reason companies use lots of live data is because people get upset when facial recognition doesn't work on people with different skin colors, voice recognition doesn't work for people with accents, etc. Doing #3 is literally the result of #4.

JangoSteve 7y ago
Keep in mind that there's no rule that says there must be a correct answer. If the only way to do something is approach A or approach B, and people don't like either approach, then other options include figuring out a new approach C or not doing the thing.
PeterStuer 7y ago
This is in no way 'the reason'. THE reason is that using live data without having to bother with consent is 'fast, easy and cheap'. Having to get consent is seen as a nuisance, and num's the word as we don't want to wake sleeping dogs. And we don't want to pay for generating training data, so yeah, we basically just steal the data from our uninformed users without their consent. I hope the GDPR will pursue this as a landmark case.

mbrumlow 7y ago

I don't think this is just a ML issue. Any device with a mic and connected to the internet could at any time be updates with new code that send audio back outside of what is needed for ML. I think that issue is much bigger than just training ML based off real world data. Laws might give you some recourse if you can prove this is happening or not. But really we need to start thinking about ways we can ensure the entire stack is under the users control. I have said this in the past. Free software and open sourc movements have worked decades to open up software and hardware for greater transparency and user control. Yet within a few years we have wiped that out with all the closed hardware and software we happily hold in out hands and place in our homes. We need more open source hardware and software projects offering alternatives to these systems that have become a daily part of our lifes.

rexpop 7y ago

It's such a relief to hear your conclusions, after so thorough an analysis.

axaxs 7y ago

Good breakdown. But I don't understand why any company thinks it is ethical to ever listen/record/send home data from -inside- a home. There are a plethora of public, or semi public places they could use to obtain real conversation(think of a subway, restaurants, DMV, etc). Set up a booth and let people talk to it. Using real speech from within someone's private quarters is disturbing IMO.

bryan_w 7y ago
I dunno if I agree with you assessment, if you buy a device that purposefully made to record your voice and send it to a company, I wouldn't find it unreasonable that the company could listen to what I sent to them. It's not like they are being sneaky about when it is recording or where the recordings are going.

Nelson69 7y ago

Anyone remember the 3d printed gun stuff from a few years back? I think this isn't very different from it. You can take these raw pieces and explain how they are simple and good and draw these simple ethical conclusions from them, but then you add it up and the bigger picture doesn't feel quite the same way. 3D printers are good, sharing 3D printing plans is good, it's good to help your neighbor, no regulations and we're experiencing tremendous growth in the 3D space, people are inventing new stuff, starting new businesses, etc.. all good stuff. but letting any jackass off the street print a working gun when we have how many mass shootings a year? People don't feel the same way. All the pieces are totally okay until you've got a more questionable global intention, and how can you regulate intention? Google using the data to train models is just a tool, it's a baby step, they aren't doing that to sell the models or in and of itself, they're doing it so that they can generate data that they might consider theirs and not yours from your voice data and then feed that in to other systems which generate tremendous profits for them in ways you don't even know. They have intended uses already. Is it a remotely fair question to talk about ethical training in this context without some idea as to the intended use and distribution of the meta data?

ousta 7y ago

you should be concerned that someone else is listening what you re saying in a private space 1 or 3 does not exist or they are not concerned because they dont have a google home

tgsovlerkhgsel 7y ago

Assuming categories 1 and 3 are sufficiently large (and I assume that is the case), this is easily resolved by allowing users to choose whether to donate their data for training or not. If the training already only happens on a 1/500 sample, skewing the sample towards "people who don't care about their privacy" will probably not significantly impact the quality of the data. I'm surprised this wasn't already the case, but hopefully the article will help the people responsible make better decisions in the trade-off between minimizing onboarding friction and respecting user's privacy in the future.

prepend 7y ago

There’s a different dimension that may or may not understand ML, but are cognizant that any data created will be viewed at least by the company that creates it. I fall into that category as I have time, nor do I trust any evaluation methods, to determine if a company is using my data ethically. If I create data and store it something that’s not mine, then I only do that in situations where I’m comfortable with the owner doing anything they want with it. I understand ML and know that Google has to at least use it for training. I’ve also worked on IT long enough that even in super tight controlled environments data are misused by administrators.

cronix 7y ago

I'm in the 5th category. 5) Understand ML; concerned - won't allow any of these things in my house period because they will always use them for things behind the scenes that they won't state. I don't care how well trained they are, or "ethical." Ethical... according to who, and at what time period in the future? Ethics change. The data they have on you won't. Look at all of the politicians and other people getting in trouble for things they said 15 years ago, which were generally more acceptable at the time but we've "progressed" since then. Who will be making decisions about you in the future based on last years data? Just don't give it to them.

foobiekr 7y ago
This. There’s nothing that binds today’s ethical company to voluntary ethical behavior when things get tough (or they get acquired).
- JaggedNZ 7y ago
  There's nothing that stops the most ethical companies from being completely compromised by state actors either.
50656E6973 7y ago
>won't allow any of these things in my house period because they will always use them for things behind the scenes that they won't state. As OP stated, "this extends beyond home speakers to any voice-enabled device in general". Unless you're living like the Unibomber, it's not a matter of "just don't give it to them". The moment you step outside your house and socialize with almost anyone anywhere, it's liable to being taken from you.
- lm28469 7y ago
  Neatpick: "unabomber" as in UNABOM (University and Airline Bomber).
  - jonsen 7y ago

munchbunny 7y ago

I'm perhaps in a subcategory of (3) that falls under "Understand ML; concerned". Knowing what I know about how people I have worked with have come close to or have actually mishandled data despite the best of intentions, I do not trust any of these teams without an explicit accountability mechanism that is observable by an outside entity. I'm not looking to punish slip-ups, because mistakes happen, but I am looking for external enforcement to keep people honest. It's not that I think the engineers using this data are mustache twirling villains, it's that I think mishandling is inevitable due to inattention (yes, even you make mistakes!), and we have to design our data pipelines against that.

drevil-v2 7y ago
Exactly. Having worked in teams which handle personal data of consumers I know how easy it would be to misuse the privilege. The legal and marketing teams that come up with the jargon and slogans about privacy are so far removed from the day to day operations that they have no clue about the reality. I don't they would care even if they did.

m3at 7y ago

I'm in both 3 and 4. 4 because not being explicit about the practice is misleading at best, outsourcing the difficult task of keeping the analysis private show how unimportant it's considered, and because big techs have a tendency to decrease privacy over time. Using clients who paid for the product as a dataset generator is also wrong. But 3 at the same time because well, it's important to evaluate the performance of the product in the field not just in the lab. There were so many cases of catastrophic failures for ML models (ex. classifying black people as gorilla) that having a tight feedback loop is important. It has to be done right, but evaluating a product that was primarily developed for (or at least by) English speakers and transfered to other domains seem like the right thing to do. All in all, I don't and wouldn't use one of those assistants because 4 outweigh 3, but it's not binary.

wybiral 7y ago

Even if I trust them to do what they say they're doing with the data I may not trust every party who comes to possess that data. And I may not trust their possession/use of it in all future contexts - as their privacy policy slowly drifts into the unknown year after year. If they're collecting it in a way that can be requested by governments (for instance) or could be leaked by hackers that's another layer of valid "concern" not related to my understanding of the ML aspect of this.

neilpointer 7y ago

I'm between 3 and 4: I just want proof that they remove PII from the audio files. If it's a bunch of audio files with unique IDs and metadata like time of day, count me as a member of group 3.

userbinator 7y ago

I'd consider myself something like "4b": Understand ML; concerned - "DO NOT WANT!" The more I see of this tech, the more disturbed I am about its existence. In some ways, dystopia and utopia really are the same thing.

raghava 7y ago

Essentially, a larger grid, involving agent( tech, management ) # assuming management has power over tech worker understanding-of-ML( yes, no ) concerned-about-ethics-and-privacy( yes, no ) The below combinations are worst in terms of ethics. { agent[tech], understanding-of-ML[yes], concerned-about-ethics-and-privacy[no] } { agent[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] } { tech[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] }

jsharf 7y ago

I think there's a third dimension: how much you've given up. So my thoughts are like #4, but with the undertone of "who am I kidding, of course they don't care, what can I do?". This is perhaps the most dangerous response.