8 ms·

> when strictly speaking the raw data has indeed been deleted after being used to create a derivative work that can for all important purposes be used to recrea

by darawk 9y ago

> when strictly speaking the raw data has indeed been deleted after being used to create a derivative work that can for all important purposes be used to recreate the original? To be precise, you almost certainly cannot use this data to recreate anything remotely resembling the original dataset. This type of dimensionality reduction would throw away enormous volumes of data. There is no meaningful sense in which you can reconstruct the data from it. What they have done is distill some insights about people from this data. It's arguable whether they should be allowed to keep those insights, but there's no privacy risk there really. It's honestly kind of disingenuous to describe dimensionality reduction in the way that they do here. It is like reducing the resolution of a photo, but it'd best be described as reducing that resolution to say, the 20 most representative pixels. There's no real sense in which the photo still exists.

anf 9y ago
> It's arguable whether they should be allowed to keep those insights, but there's no privacy risk there really. So if Google has distilled someone's emails over the years into "closeted homosexual with a deeply repressed leather fetish", that's not an invasion of their privacy as long as they throw away the source materials?
- spydum 9y ago
  As long as they retain no data which could specifically identify the original person, yes. There is nothing wrong with building segmentation models as long as they aren't specific enough to identify a specific person. My concern would be, how granular is too granular? What if we added "and live in zip code 12355 and is registered Green Party"? This now gets eerily specific, and might be sufficient to identify an individual.
  - anf 9y ago
    Why would they ever discard that? Why would there be a granularity where ML suddenly stops working? Why would you even stop at one model per person, instead of one model per mood, or modes of thought at different stress points?
    - creaghpatr 9y ago
      In fact they would desire that granularity most of all so as to reconcile the past and future state psychographic profiles for an individual- then they could attempt to isolate the causation of a state change- basically they need to identify the moment an individuals profile reflects the change from democrat to republican or vice versa. Or Religious to atheist etc.
- darawk 9y ago
  If they kept information like that, then yes that would be an invasion of privacy. But that sort of information is almost certainly not encoded in an ML model trained on 50 million people's data.
  - anf 9y ago
    How do you know what CA trained on, or what's possible? Do you have qualifications in ML?
    - darawk 9y ago
      I know what they trained on because it's been reported on. They got around 50 million people's FB profiles, and a smaller subset's (300k, I think) personality test results. I use ML models every day in my work, and understand how they function. It is true that individuals information is probabilistically encoded into the parameters of the model. However, if the model is any good, the people they trained on's information is encoded only a bit more than that of the entire population. There is sort of a privacy issue in the following sense: The models they've built have learned relationships between preferences and personalities that they wouldn't otherwise have been able to learn. But these relationships are abstract. They are not tethered to any particular, identifiable individual. A reasonable argument can be made that those learned relationships are, in a sense, stolen property. And I think arguments along those lines are interesting things that we'll have to explore as this sort of thing becomes more common. But the idea that this model invades individuals privacy just isn't really true.
      - pflanze 9y ago
        But if the resulting model doesn't contain information about individuals, how does this help targeting individuals for the campaign? Edit: is it that the model is then applied to only strictly public data about the person? If so I guess the interesting question then becomes whether the model is definitely not anything near overfitting (i.e. containing enough information to match a person's public data directly since it was trained on it (amongst other data))? (I'm not an ML developer.) Edit 2: also, going with your comparison with the "20 most representative pixels", it seems interesting then that 'this much' (although not exactly sure how much) information can be inferred from a public profile when just also knowing enough about the whole Facebook population. OK, so perhaps a human would be able to infer about as much, but doesn't scale, and that's why the model becomes valuable?
- jcranberry 9y ago
  That's a ridiculous response. If they managed to infer this characteristic from emails, what they would keep is a tool which, given that set of emails again, infer the same characteristics (and theoretically a similar set of emails). They would by no means be allowed to keep the kind of information you described. What is more relevant is a model which, given characteristics such as "closeted homosexual with a deeply repressed leather fetish", they would be able to infer other characteristics, such as support of particular political candidates, responsiveness towards targeted political or commercial ad campaigns, etc. That's what's relevant here.
- caseysoftware 9y ago
  Since the source data was deleted - according to current standards and policies - their hands are probably technically clean. But there may be another angle of attack. In the US, you're not allowed to benefit directly from a crime you committed. For example, if you rob a bank, you can't buy your mother a car with the money and say "sorry, it's gone!" when the police come knocking. With that line of reasoning and if there was a legal, privacy, or at least a TOS breach in collecting the data, the derivative machine learning models may be tainted also. Then again, it's likely impossible to prove exactly what data went into the model, so hard to establish which models might be tainted.
berkay 9y ago
Analogy does not work here and is misleading. You cannot do much if anything with 20 most representative pixels (if there is such a thing) but you can infer highly valuable characteristics about the person. Yes, you cannot recreate the original data but what you end up is potentially much worse (sensitive/private) than the original data.
- darawk 9y ago
  That's not really true, and is kind of a fundamental misunderstanding of how these things work.
  - wybiral 9y ago
    Unless the data is completely random it's not crazy to say that the data can be reconstructed from a reduced version. If you have a million points that largely fall on a 3-dimensional line and you project that into 2 dimensions, you can easily recover that lost dimension with losses relative to the deviation. And that loss may not even matter depending on the kinds of data and margins of error you're working in.
    - darawk 9y ago
      Except that that toy example bears no resemblance to the actual situation.
      - wybiral 9y ago
        How many dimensions were they working with and how much variance and correlation was there in the features? What's the margin of error for the end product?
        darawk 9y ago
        I don't know precisely, but it's pretty obvious that there'd be no way to reconstruct personally identifying info from it.
stochastic_monk 9y ago
That's only accurate in the sense that because an LSTM's hidden layer is much smaller in dimension than the data on which it is trained, there is less information in it. However, it concisely represents a manifold in a much larger dimensional space and effectively captures most of the information in it. It may be (and is) lossy, but don't underestimate the expressive power of a deep neural network.
- wybiral 9y ago
  Especially when the dataset probably isn't that high in entropy. Something like PCA can drop the dimensionality by significant amounts as long as the data has enough clear signals in it.
- salty_biscuits 9y ago
  Only if the "true" data actually lives in a lower dimensional manifold and the data acurrately can encode it with low noise. I doubt anyone can tell who you will vote for depending on which cat videos you liked, no matter how magic your regressor.
  - stochastic_monk 9y ago
    I do think that the most significant components of a personality will likely be targetable with a relatively low nuclear norm. And, for example, where someone's proclivity on the exploration/exploitation spectrum, if you will, (IE, how strongly do they respond to fear-based messaging) falls is probably quite predictable from a spectrum of likes. Cat pictures may be less informative, but not all of these people clicked exclusively on feline fuzzy photos.
    - salty_biscuits 9y ago
      I'm not an expert on personality so won't disagree (except to say that I am a little sceptical of a static personality profile actually existing and I think people who always vote a certain way would be the easiest to regress and also the most useless to target). As I said in another post, it really depends on what part of your privacy you are trying to protect. It is also a mistake to think of anything on the internet as a private forum.
deleted 9y ago
[deleted]
chii 9y ago
> distill some insights about people from this data i'd argue that insight is the bit that's important, and the bit that's the privacy risk.
stevenicr 9y ago
" cannot use this data to recreate anything remotely resembling the original dataset." This by itself may be mostly true perhaps - and many of the comments get into ways of playing with this dataset to make it better, I don't have experience with those methods, but, what I have not seen anyone mention, if you have this dumbed down dataset, the original is gone.. you can still combine with other data sets that are either public or previously created and likely fine tune; dumbed down set + public voter records + public arrest records + previous whatever records - sort, match, what's left over. and pretty much recreate what you needed from the original, maybe not 100%, but I would guess you could get really close.
- firebones 9y ago
  Or "better" (for some value of better), join the data to something identifiable (e.g., the public records you listed) before developing models for everything you want to retain, then discard the original data once your deep enough net has effectively auto-encoded whatever you wanted to retain in an identifiable manner.
etiam 9y ago
> To be precise, you almost certainly cannot use this data to recreate anything remotely resembling the original dataset. This type of dimensionality reduction would throw away enormous volumes of data. There is no meaningful sense in which you can reconstruct the data from it. First off, I think that's wrong. The idea is after all to keep the information that will result in the smallest error compared to the original on the dimensions one cares about. Within what the model emphasizes a reconstruction can be not only "remotely resembling the original dataset" but as closely resembling the original dataset as is possible with the capacity of the representation. Next, I'm really not talking only about the particular method described in the post. It's definitely possible to choose to make a light enough reduction to preserve the aspects of the information one is interested in, and to optimize for recall rather than generalization. A more realistic context is going to be that some information about the affected individuals is still exposed or kept (maybe in a compact derived form), which would in many cases give excellent possibilities to restore information accurately enough that claims to have the removed the data are effectively deceptive. Even for cases where the models are in good faith created only to "distill some insights" I'm skeptical that they really are useless for recovering individual information. I'm by no means an expert in differential privacy but I do listen when it comes up, and a lot of what we see from that field seems to come down to being able to trade off the relation between keeping the data useful and how many pieces of additional information (or assumptions and brute force) are needed to break the integrity protections. With surprises that tend to be on the side of 'Oops. Turns out this clever trick can recover the originals easier than we thought.' > It's honestly kind of disingenuous to describe dimensionality reduction in the way that they do here. It is like reducing the resolution of a photo, but it'd best be described as reducing that resolution to say, the 20 most representative pixels. There's no real sense in which the photo still exists. In my honest opinion the original analogy does an excellent job of intuitively explaining that most of the informative aspects of the data are kept (we can still see just fine what's in the image) while irrelevant details are discarded, and that is probably what was intended. If anything comes off as disingenuous in that context it's your representation that it's like a strong reduction in the pixel domain (where it does indeed destroy a lot of the information). What can be done is much more like running the picture through a high-performance Imagenet classifier and keeping the 20 (or 2048, or whatever's needed) most informative values at a level that corresponds strongly to semantic content of the picture, and holding on the model. We could probably generate images that people would have a hard time distinguishing from the original with that.
zaphos 9y ago
This type being ... something like PCA? It's up to the user how much to actually reduce the dimensionality. The pixel analogy is bad, but to use it anyway -- you get to choose how many pixels you keep. You could keep literally all of them.
deleted 9y ago
[deleted]
kurthr 9y ago
So if I compress a BMP by 100:1 using "lossy" techniques then it's not equivalent to the original? I'd say that depends on how recognizable the result of reconstruction is, and not on the amount of reduction. MPAA would be very unhappy with your argument. To be more extreme there are many compression/extraction methods that can perfectly reconstruct the original data with very high compression ratios. GIF/PNG can reproduce many images exactly. Certainly, they are derivative works?
- darawk 9y ago
  This compression analogy is really a bad one. A much better analogy would be: If you read 10 novels by a particular author, and are now able to recognize his/her style, because you have a compressed representation in your mind of the way they write.
- fjsolwmv 9y ago
  PNG isn't a derivate work. It's an alternate encoding of the same work.

by darawk 9y ago

anf 9y ago

> It's arguable whether they should be allowed to keep those insights, but there's no privacy risk there really. So if Google has distilled someone's emails over the years into "closeted homosexual with a deeply repressed leather fetish", that's not an invasion of their privacy as long as they throw away the source materials?

spydum 9y ago
As long as they retain no data which could specifically identify the original person, yes. There is nothing wrong with building segmentation models as long as they aren't specific enough to identify a specific person. My concern would be, how granular is too granular? What if we added "and live in zip code 12355 and is registered Green Party"? This now gets eerily specific, and might be sufficient to identify an individual.
- anf 9y ago
  Why would they ever discard that? Why would there be a granularity where ML suddenly stops working? Why would you even stop at one model per person, instead of one model per mood, or modes of thought at different stress points?
  - creaghpatr 9y ago
    In fact they would desire that granularity most of all so as to reconcile the past and future state psychographic profiles for an individual- then they could attempt to isolate the causation of a state change- basically they need to identify the moment an individuals profile reflects the change from democrat to republican or vice versa. Or Religious to atheist etc.
darawk 9y ago
If they kept information like that, then yes that would be an invasion of privacy. But that sort of information is almost certainly not encoded in an ML model trained on 50 million people's data.
- anf 9y ago
  How do you know what CA trained on, or what's possible? Do you have qualifications in ML?
  - darawk 9y ago
    I know what they trained on because it's been reported on. They got around 50 million people's FB profiles, and a smaller subset's (300k, I think) personality test results. I use ML models every day in my work, and understand how they function. It is true that individuals information is probabilistically encoded into the parameters of the model. However, if the model is any good, the people they trained on's information is encoded only a bit more than that of the entire population. There is sort of a privacy issue in the following sense: The models they've built have learned relationships between preferences and personalities that they wouldn't otherwise have been able to learn. But these relationships are abstract. They are not tethered to any particular, identifiable individual. A reasonable argument can be made that those learned relationships are, in a sense, stolen property. And I think arguments along those lines are interesting things that we'll have to explore as this sort of thing becomes more common. But the idea that this model invades individuals privacy just isn't really true.
    - pflanze 9y ago
      But if the resulting model doesn't contain information about individuals, how does this help targeting individuals for the campaign? Edit: is it that the model is then applied to only strictly public data about the person? If so I guess the interesting question then becomes whether the model is definitely not anything near overfitting (i.e. containing enough information to match a person's public data directly since it was trained on it (amongst other data))? (I'm not an ML developer.) Edit 2: also, going with your comparison with the "20 most representative pixels", it seems interesting then that 'this much' (although not exactly sure how much) information can be inferred from a public profile when just also knowing enough about the whole Facebook population. OK, so perhaps a human would be able to infer about as much, but doesn't scale, and that's why the model becomes valuable?
jcranberry 9y ago
That's a ridiculous response. If they managed to infer this characteristic from emails, what they would keep is a tool which, given that set of emails again, infer the same characteristics (and theoretically a similar set of emails). They would by no means be allowed to keep the kind of information you described. What is more relevant is a model which, given characteristics such as "closeted homosexual with a deeply repressed leather fetish", they would be able to infer other characteristics, such as support of particular political candidates, responsiveness towards targeted political or commercial ad campaigns, etc. That's what's relevant here.
caseysoftware 9y ago
Since the source data was deleted - according to current standards and policies - their hands are probably technically clean. But there may be another angle of attack. In the US, you're not allowed to benefit directly from a crime you committed. For example, if you rob a bank, you can't buy your mother a car with the money and say "sorry, it's gone!" when the police come knocking. With that line of reasoning and if there was a legal, privacy, or at least a TOS breach in collecting the data, the derivative machine learning models may be tainted also. Then again, it's likely impossible to prove exactly what data went into the model, so hard to establish which models might be tainted.

berkay 9y ago

Analogy does not work here and is misleading. You cannot do much if anything with 20 most representative pixels (if there is such a thing) but you can infer highly valuable characteristics about the person. Yes, you cannot recreate the original data but what you end up is potentially much worse (sensitive/private) than the original data.

darawk 9y ago
That's not really true, and is kind of a fundamental misunderstanding of how these things work.
- wybiral 9y ago
  Unless the data is completely random it's not crazy to say that the data can be reconstructed from a reduced version. If you have a million points that largely fall on a 3-dimensional line and you project that into 2 dimensions, you can easily recover that lost dimension with losses relative to the deviation. And that loss may not even matter depending on the kinds of data and margins of error you're working in.
  - darawk 9y ago
    Except that that toy example bears no resemblance to the actual situation.
    - wybiral 9y ago
      How many dimensions were they working with and how much variance and correlation was there in the features? What's the margin of error for the end product?
      - darawk 9y ago
        I don't know precisely, but it's pretty obvious that there'd be no way to reconstruct personally identifying info from it.

stochastic_monk 9y ago

That's only accurate in the sense that because an LSTM's hidden layer is much smaller in dimension than the data on which it is trained, there is less information in it. However, it concisely represents a manifold in a much larger dimensional space and effectively captures most of the information in it. It may be (and is) lossy, but don't underestimate the expressive power of a deep neural network.

wybiral 9y ago
Especially when the dataset probably isn't that high in entropy. Something like PCA can drop the dimensionality by significant amounts as long as the data has enough clear signals in it.
salty_biscuits 9y ago
Only if the "true" data actually lives in a lower dimensional manifold and the data acurrately can encode it with low noise. I doubt anyone can tell who you will vote for depending on which cat videos you liked, no matter how magic your regressor.
- stochastic_monk 9y ago
  I do think that the most significant components of a personality will likely be targetable with a relatively low nuclear norm. And, for example, where someone's proclivity on the exploration/exploitation spectrum, if you will, (IE, how strongly do they respond to fear-based messaging) falls is probably quite predictable from a spectrum of likes. Cat pictures may be less informative, but not all of these people clicked exclusively on feline fuzzy photos.
  - salty_biscuits 9y ago
    I'm not an expert on personality so won't disagree (except to say that I am a little sceptical of a static personality profile actually existing and I think people who always vote a certain way would be the easiest to regress and also the most useless to target). As I said in another post, it really depends on what part of your privacy you are trying to protect. It is also a mistake to think of anything on the internet as a private forum.

deleted 9y ago

[deleted]

chii 9y ago

> distill some insights about people from this data i'd argue that insight is the bit that's important, and the bit that's the privacy risk.

stevenicr 9y ago

" cannot use this data to recreate anything remotely resembling the original dataset." This by itself may be mostly true perhaps - and many of the comments get into ways of playing with this dataset to make it better, I don't have experience with those methods, but, what I have not seen anyone mention, if you have this dumbed down dataset, the original is gone.. you can still combine with other data sets that are either public or previously created and likely fine tune; dumbed down set + public voter records + public arrest records + previous whatever records - sort, match, what's left over. and pretty much recreate what you needed from the original, maybe not 100%, but I would guess you could get really close.

firebones 9y ago
Or "better" (for some value of better), join the data to something identifiable (e.g., the public records you listed) before developing models for everything you want to retain, then discard the original data once your deep enough net has effectively auto-encoded whatever you wanted to retain in an identifiable manner.

etiam 9y ago

> To be precise, you almost certainly cannot use this data to recreate anything remotely resembling the original dataset. This type of dimensionality reduction would throw away enormous volumes of data. There is no meaningful sense in which you can reconstruct the data from it. First off, I think that's wrong. The idea is after all to keep the information that will result in the smallest error compared to the original on the dimensions one cares about. Within what the model emphasizes a reconstruction can be not only "remotely resembling the original dataset" but as closely resembling the original dataset as is possible with the capacity of the representation. Next, I'm really not talking only about the particular method described in the post. It's definitely possible to choose to make a light enough reduction to preserve the aspects of the information one is interested in, and to optimize for recall rather than generalization. A more realistic context is going to be that some information about the affected individuals is still exposed or kept (maybe in a compact derived form), which would in many cases give excellent possibilities to restore information accurately enough that claims to have the removed the data are effectively deceptive. Even for cases where the models are in good faith created only to "distill some insights" I'm skeptical that they really are useless for recovering individual information. I'm by no means an expert in differential privacy but I do listen when it comes up, and a lot of what we see from that field seems to come down to being able to trade off the relation between keeping the data useful and how many pieces of additional information (or assumptions and brute force) are needed to break the integrity protections. With surprises that tend to be on the side of 'Oops. Turns out this clever trick can recover the originals easier than we thought.' > It's honestly kind of disingenuous to describe dimensionality reduction in the way that they do here. It is like reducing the resolution of a photo, but it'd best be described as reducing that resolution to say, the 20 most representative pixels. There's no real sense in which the photo still exists. In my honest opinion the original analogy does an excellent job of intuitively explaining that most of the informative aspects of the data are kept (we can still see just fine what's in the image) while irrelevant details are discarded, and that is probably what was intended. If anything comes off as disingenuous in that context it's your representation that it's like a strong reduction in the pixel domain (where it does indeed destroy a lot of the information). What can be done is much more like running the picture through a high-performance Imagenet classifier and keeping the 20 (or 2048, or whatever's needed) most informative values at a level that corresponds strongly to semantic content of the picture, and holding on the model. We could probably generate images that people would have a hard time distinguishing from the original with that.

zaphos 9y ago

This type being ... something like PCA? It's up to the user how much to actually reduce the dimensionality. The pixel analogy is bad, but to use it anyway -- you get to choose how many pixels you keep. You could keep literally all of them.

deleted 9y ago

[deleted]

kurthr 9y ago

So if I compress a BMP by 100:1 using "lossy" techniques then it's not equivalent to the original? I'd say that depends on how recognizable the result of reconstruction is, and not on the amount of reduction. MPAA would be very unhappy with your argument. To be more extreme there are many compression/extraction methods that can perfectly reconstruct the original data with very high compression ratios. GIF/PNG can reproduce many images exactly. Certainly, they are derivative works?

darawk 9y ago
This compression analogy is really a bad one. A much better analogy would be: If you read 10 novels by a particular author, and are now able to recognize his/her style, because you have a compressed representation in your mind of the way they write.
fjsolwmv 9y ago
PNG isn't a derivate work. It's an alternate encoding of the same work.