15 ms·

Beyond Memorization: Violating privacy via inference with LLMs

hermannj314 3y ago
How does making an inference about someone's income, sex, or location violate privacy expectations? Let's say "you speak Luxembourgish and ask where you can buy a Mercedes for driving your kids to school" You think it violates your privacy that I might be able to guess where you live, your gender, and your income from that? Did I miss when the world made Bayes theorem illegal?
- Pannoniae 3y ago
  It depends. Just like it's legal to deposit money into a bank (a legal activity) but it's illegal to structure (a series of legal actions with a specific criminal intent). Applying Bayes' theorem isn't illegal but making interference from other public information then de-anonymising other data based on cross-matching is an entirely different act, regulated by privacy laws. Privacy isn't only about the method, it's also about scale and intent. See CCTV which can individually be dumped the footage from and rewatched to identify criminals vs. live feed surveillance with AI face detection to track everyone 24/7. Same data used but very different intents and outcomes.
  - cbsmith 3y ago
    > Privacy isn't only about the method, it's also about scale and intent. I'm with you about the intent, but I don't follow the scale argument. It's not like it isn't a privacy violation if you only do it to one person.
    - Pannoniae 3y ago
      yes, it is, but if you only cross-match data from a single person, that's called conducting a (private) investigation, which is regularly done, that's absolutely legal... However, it changes when you do that in an automated way, en masse, both ethically and legally.
      - cbsmith 3y ago
        The "automated way" is just a matter of perspective. What matters is that the more efficiently you can do it, the fewer resources you need to devote to "investigate" anyone. That can certainly create ethical quandaries. I don't follow how it creates a legal one. AFAIK, the boundaries on privacy laws tend to be tied to access & consent, not scale. Scale just changes the size of the violation. Can you give me an example of a privacy law where the legal line is drawn around scale?
      - chii 3y ago
        > en masse, both ethically and legally. I think only problematic ethically. From what i can see, there can't be anything illegal about collecting publicly available information and using any algorithm to analyse and deduce information from it.
        rperez333 3y ago
        The GDPR in Europe doesn’t allow you to use publicly available information without explicit user consent, especially if involves personal data. Even doing some Google Analytics can in theory result in large fines.
        cbsmith 3y ago
        The GDPR is really complicated. There's plenty of publicly available information you can use without explicit user consent. What's true is that many of GDPR's rules, particularly the disclosure of data sources and erasure requests, don't get an exemption just because the data is publicly available. For example, while they can't just divulge my name & age in the news, the captions under pictures in news articles can often still have people's names & ages under them without explicit user consent. Tabloids would likely cease to exist if it were any other way. ;-)
        pixl97 3y ago
        Actors living in other countries with no contacts to Europe don't give two shits about GDPR. If I'm a hacker looking at tracking down particular users and committing phishing/watering hole attacks all the laws on the planet really don't matter. What does matter is the data you leak online and the tools I can use to process it.
- ReactiveJelly 3y ago
  Your birthday and zip code is almost enough to identify you. Every bit of entropy counts
  - cbsmith 3y ago
    Your point stands, but the comment is misleading. The claim is birthdate+zip+gender could uniquely identify 87% of the US population, and there are respectable studies to back this stuff up (https://groups.csail.mit.edu/mac/classes/6.805/articles/privacy/sweeney-thesis-draft.pdf https://groups.csail.mit.edu/mac/classes/6.805/articles/priv...). There aren't a lot of days where there's only one person celebrating their birthday in an entire zip code. Even factoring in birth year and gender, I struggle with this a bit, because I know that in densely populated areas of the country, where more and more of the population lives (upwards of 80% of the US lives in urban environments), you'll find multiple babies with the same gender being born in the same hospital on any given day... and there are often multiple hospitals servicing the same zip code! Despite all this, your point still stands: it doesn't take much data to deanonymize someone.
- kromem 3y ago
  I think it's more the opposite that needs to change, in that people should realize that most information they make public can reveal more than they think. I remember when store algorithms were identifying a teenage girl was pregnant before her family knew and offering her discounts on maternity stuff. That anecdote was from over a decade ago. Today average users are worried their phone is listening to them because something in a conversation comes up in a targeted ad, clearly worried about the idea of "private voice data" being compromised, but somehow less weirded out by the fact that the end result is possible without listening to anything. Our expectations of privacy in relation to our data need to change with the times, and research like this helps that slightly, though to be honest the real picture of what can be discerned from data isn't reflected in public research but is enshrined in private practice, and having seen some of that side of things this paper's 'discovery' is a giant yawn. People should begin to prepare for a future where your anonymous activities online (esp images and text) are cross correlated to your public activities trivially by any interested party. And where your deep dark secrets aren't so secret or where truth and lies can reasonably be revealed as such. Cheating in a relationship? Are you unique enough that a multimodal model fed all public social media data can't identify matching patterns in online behavior changes from early in a relationship to later for profiles where the SO later posted about infidelity and compare to your activity? When switching from scented candles to unscented can tell a store's algorithm with access to broad data that you are pregnant before you've told anyone, we really can't yet imagine just what much larger models trained on much more massive data may yet reveal in the smallest actions and nuances.
nl 3y ago
This is moderately interesting work but the title feels wrong to me. They use LLMs to analyse social media and turn unstructured text ("I always get stuck waiting for a hook turn") into structured data (location=Melbourne, Australia). They test this on a set of Reddit profiles that have been hand labelled. However, the fact that this hand labeling is possible indicates in itself that the issue isn't a pure privacy violation. It's more that automating this in bulk is now possible which was difficult before. Is this ethical? It isn't obvious to me that it's not - I think most people writing on a public site realize their writing is public, and thew fact we see common warnings about not posting personal information shows that is true. It's not clear it violates privacy expectations either - even if some people don't like it.
- anon25783 3y ago
  Would you want it done to you? I sure wouldn't. Too bad it's probably going to be done to me anyway.
  - nl 3y ago
    It pretty much already is right (even if the technology used is different)? A company like Google builds a profile[1] from my search queries. Many other AdTech companies do similar things. > Would you want it done to you? I sure wouldn't. I don't mind this - if I have to see ads I prefer ones relevant to me over ones that are completely irrelevant. [1] https://myadcenter.google.com/controls https://myadcenter.google.com/controls
    - dotancohen 3y ago
      Would you like political ads micro-targeted to you? Gay men get this ad: Only Politician Foo protects gay security in the streets at night! Homophobes get this ad: Then you must vote for Politician Foo, only he opposses gay marriage! Pro-lifers get this ad: Vote for Foo to protect the children from the very first moment! Pro-abortion get this ad: Foo protects women's rights! Advertising should be as general as possible - microtargeting people in a democracy is extraordinarily dangerous.
hn_throwaway_99 3y ago
I think this study highlights a specific instance of an issue I've previously argued, which is basically "Data analysis at scale completely breaks many intuitive assumptions humans hold.", privacy being one of those assumptions. For example, twenty years or so ago, I think the vast, vast majority of people in the US would agree that you don't have a right to privacy walking around a public street. However, once the Internet and Streetview came along, people had very different feelings about whether someone should be able to post a video they took of you on the Internet, for anyone across the world to see for all eternity, oh and we can also use facial recognition (just from public data and photos other people have posted) to fully identify you. Privacy is just one example. I will personally say my opinions on free speech have also changed. Given how highly targeted, systematic and relentless well-funded misinformation campaigns can be, the old assumptions I had about free speech ("Given enough daylight, the majority of people will walk away from 'bad' ideas") I no longer believe to be true. I really don't have any idea what the "right" solution is, just that my views have changed. The primary point is that wide scale, incredibly cheap data analysis (now being supercharged with AI) basically breaks all these fundamental assumptions and ideas society has, which is one reason why the world seems to be in such turmoil.
- downboots 3y ago
  Log_2(8 billion people) = ~33 yes/no questions
  - kridsdale3 3y ago
    So why isn't my social security number 4 bytes? I could just be "$jA9" when ascii encoded.
- jmugan 3y ago
  When I first saw Interstellar I thought it was unrealistic that people could be brainwashed to forget about space flight. When I watched it again last year it seemed entirely plausible.
  - Ialdaboth 3y ago
mock-possum 3y ago
> show that current LLMs can infer a wide range of personal attributes That’s not invasion of privacy, that’s just making good (informed) guesses.
soul_grafitti 3y ago
It seems to me this is similar to the way poker players observe 'tells' in the other players - cataloging innocuous observations to make highly specific inferences. I know people who are naturally observant and quite good at this. As I think about it this is really what makes Sherlock Holmes so interesting too. What's changing is the scope, scale and access. It's no longer necessary to be a savant or have a special talent - all you need are hundreds of banal data points easily scooped from the meta data wake we all leave. Not to mention it can be programmed and commoditized instead of being a special attribute of another human. The realization feels a bit like climate change in that some folks saw it coming, pointed it out, articulated the issues, underestimated how fast it would actually arrive and how profound the impact would be. But back to the point at hand you can't legislate people not see what is publicly before them. You might criminalize recording that data for a while but that doesn't seem to work long term thought it's brutal in the short term. If you can record that data you can analyze that data. From analysis comes inferences and decisions and actions. I think the more salient question is about equality. I have some ability to obfuscate my meta data wake but as an individual it's pretty limited. Apple, google, Amazon have much more ability to do so despite leaving a gazillion more bits of data laying around. Tools individuals can use to obscure our meta data trail will do more to maintain privacy than trying to tell someone observing me what they can and can't do with their observations.
- dcow 3y ago
  Yeah this is the issue with the approach of trying to control what companies can and can’t do with the inferences they make from essentially public data they collect about you. Recent privacy policy is like going to an entity and saying “you can’t remember that customer 357 showed up in a BMW and wore a red jacket and told you their name is Jeff Geoffrey when they visited your store”. It’s not that I disagree with the desire to legislate this stuff in defense of privacy, but rather I don’t feel like the approach of treating the symptoms is correct. If we want to enhance privacy then legislate browsers and platforms, not a company’s eyeballs. Make sure browsers provide all the knobs people need to control the size of their data wake.
  - em-bee 3y ago
upupupandaway 3y ago
I had thought of doing the exact same thing, but then cross-referencing with LinkedIn and other non-anonymous social networks to go the extra step and actually identify people with a certain level of confidence. I think we’re months, maybe days away from someone actually building this product and using it for nefarious purposes.
- yetanother12345 3y ago
  > I think we’re months, maybe days away from ... I have reasons to think years. Ie. that you are years behind. Profiling individuals based on public data already exist, and it's a profitable business as well. > using it for nefarious purposes. Law enforcement (and various other state-owned entities) would beg to disagree. As for vendors, same. Edit: Ad networks, even.
radar1310 3y ago
Not surprised at all.
Xcelerate 3y ago
It seems quite clear that inferring traits about people is something LLMs will only become better at with time. Everyone claims MBTI is akin to astrology, which means there should be no predictive capacity of those four letters. But just for fun, I gave GPT-4 the top 20 songs of my music playlist and asked it to guess my MBTI type, and it correctly did so. I repeated the process with two friends (asking their MBTI type prior to querying GPT) and it guessed them correctly as well. If MBTI is no different than random noise and GPT lacks inference capabilities with regard to humans, the probability of this occurring due to chance alone is less than 1 in 16^3 (*approximately—non-uniform distribution of types).
- blovescoffee 3y ago
  I don’t think anyone says MBTI is random. I think the doubt is in the predictive power of a whichever MBTI classification. The test for your MB label is pretty well designed to extract “some” information. Academics just don’t think the labeling is particularly valuable.
  - Xcelerate 3y ago
    Fair point. But are the “superior” systems like Big 5 (or more recent approaches that utilize independent component analysis) really that much more predictive than MBTI? It seems like there is a lot of squabbling over the difference of a few percentage points in information gain.
    - pgeorgi 3y ago
      To a layman, they're all useless, and in a dangerous way: they transport a sense of understanding a person/character/psyche that they don't support. A professional can gain _some_ information from all of them. Just that the Big 5 have much more literature behind them than MBTI, so there's more reference data for any given constellation. That's a reinforcement loop. Methodical differences and reliability and all that fluff can be used to argue in more detail in favor of one versus the other, but by now, it comes down to what folks are more familiar with. In the end, these are just a few out of many, many tools of the field. If you see somebody build their career on "(MBTI|Big 5|...) analysis" and nothing else, they're not a professional but a quack.
yafbum 3y ago
Inferring someone's demographic attributes by observing their behaviors isn't very new or specific to LLMs. Basic stats can do the job well enough, and ML classification models have been trained on this for a while. What's new here I guess is that LLMs already have such inference capability built in, even though they are very general-purpose?

How does making an inference about someone's income, sex, or location violate privacy expectations? Let's say "you speak Luxembourgish and ask where you can buy a Mercedes for driving your kids to school" You think it violates your privacy that I might be able to guess where you live, your gender, and your income from that? Did I miss when the world made Bayes theorem illegal?

Pannoniae 3y ago
It depends. Just like it's legal to deposit money into a bank (a legal activity) but it's illegal to structure (a series of legal actions with a specific criminal intent). Applying Bayes' theorem isn't illegal but making interference from other public information then de-anonymising other data based on cross-matching is an entirely different act, regulated by privacy laws. Privacy isn't only about the method, it's also about scale and intent. See CCTV which can individually be dumped the footage from and rewatched to identify criminals vs. live feed surveillance with AI face detection to track everyone 24/7. Same data used but very different intents and outcomes.
- cbsmith 3y ago
  > Privacy isn't only about the method, it's also about scale and intent. I'm with you about the intent, but I don't follow the scale argument. It's not like it isn't a privacy violation if you only do it to one person.
  - Pannoniae 3y ago
    yes, it is, but if you only cross-match data from a single person, that's called conducting a (private) investigation, which is regularly done, that's absolutely legal... However, it changes when you do that in an automated way, en masse, both ethically and legally.
    - cbsmith 3y ago
      The "automated way" is just a matter of perspective. What matters is that the more efficiently you can do it, the fewer resources you need to devote to "investigate" anyone. That can certainly create ethical quandaries. I don't follow how it creates a legal one. AFAIK, the boundaries on privacy laws tend to be tied to access & consent, not scale. Scale just changes the size of the violation. Can you give me an example of a privacy law where the legal line is drawn around scale?
    - chii 3y ago
      > en masse, both ethically and legally. I think only problematic ethically. From what i can see, there can't be anything illegal about collecting publicly available information and using any algorithm to analyse and deduce information from it.
      - rperez333 3y ago
        The GDPR in Europe doesn’t allow you to use publicly available information without explicit user consent, especially if involves personal data. Even doing some Google Analytics can in theory result in large fines.
        cbsmith 3y ago
        The GDPR is really complicated. There's plenty of publicly available information you can use without explicit user consent. What's true is that many of GDPR's rules, particularly the disclosure of data sources and erasure requests, don't get an exemption just because the data is publicly available. For example, while they can't just divulge my name & age in the news, the captions under pictures in news articles can often still have people's names & ages under them without explicit user consent. Tabloids would likely cease to exist if it were any other way. ;-)
        pixl97 3y ago
        Actors living in other countries with no contacts to Europe don't give two shits about GDPR. If I'm a hacker looking at tracking down particular users and committing phishing/watering hole attacks all the laws on the planet really don't matter. What does matter is the data you leak online and the tools I can use to process it.
ReactiveJelly 3y ago
Your birthday and zip code is almost enough to identify you. Every bit of entropy counts
- cbsmith 3y ago
  Your point stands, but the comment is misleading. The claim is birthdate+zip+gender could uniquely identify 87% of the US population, and there are respectable studies to back this stuff up (https://groups.csail.mit.edu/mac/classes/6.805/articles/privacy/sweeney-thesis-draft.pdf https://groups.csail.mit.edu/mac/classes/6.805/articles/priv...). There aren't a lot of days where there's only one person celebrating their birthday in an entire zip code. Even factoring in birth year and gender, I struggle with this a bit, because I know that in densely populated areas of the country, where more and more of the population lives (upwards of 80% of the US lives in urban environments), you'll find multiple babies with the same gender being born in the same hospital on any given day... and there are often multiple hospitals servicing the same zip code! Despite all this, your point still stands: it doesn't take much data to deanonymize someone.
kromem 3y ago
I think it's more the opposite that needs to change, in that people should realize that most information they make public can reveal more than they think. I remember when store algorithms were identifying a teenage girl was pregnant before her family knew and offering her discounts on maternity stuff. That anecdote was from over a decade ago. Today average users are worried their phone is listening to them because something in a conversation comes up in a targeted ad, clearly worried about the idea of "private voice data" being compromised, but somehow less weirded out by the fact that the end result is possible without listening to anything. Our expectations of privacy in relation to our data need to change with the times, and research like this helps that slightly, though to be honest the real picture of what can be discerned from data isn't reflected in public research but is enshrined in private practice, and having seen some of that side of things this paper's 'discovery' is a giant yawn. People should begin to prepare for a future where your anonymous activities online (esp images and text) are cross correlated to your public activities trivially by any interested party. And where your deep dark secrets aren't so secret or where truth and lies can reasonably be revealed as such. Cheating in a relationship? Are you unique enough that a multimodal model fed all public social media data can't identify matching patterns in online behavior changes from early in a relationship to later for profiles where the SO later posted about infidelity and compare to your activity? When switching from scented candles to unscented can tell a store's algorithm with access to broad data that you are pregnant before you've told anyone, we really can't yet imagine just what much larger models trained on much more massive data may yet reveal in the smallest actions and nuances.

nl 3y ago

This is moderately interesting work but the title feels wrong to me. They use LLMs to analyse social media and turn unstructured text ("I always get stuck waiting for a hook turn") into structured data (location=Melbourne, Australia). They test this on a set of Reddit profiles that have been hand labelled. However, the fact that this hand labeling is possible indicates in itself that the issue isn't a pure privacy violation. It's more that automating this in bulk is now possible which was difficult before. Is this ethical? It isn't obvious to me that it's not - I think most people writing on a public site realize their writing is public, and thew fact we see common warnings about not posting personal information shows that is true. It's not clear it violates privacy expectations either - even if some people don't like it.

anon25783 3y ago
Would you want it done to you? I sure wouldn't. Too bad it's probably going to be done to me anyway.
- nl 3y ago
  It pretty much already is right (even if the technology used is different)? A company like Google builds a profile[1] from my search queries. Many other AdTech companies do similar things. > Would you want it done to you? I sure wouldn't. I don't mind this - if I have to see ads I prefer ones relevant to me over ones that are completely irrelevant. [1] https://myadcenter.google.com/controls https://myadcenter.google.com/controls
  - dotancohen 3y ago
    Would you like political ads micro-targeted to you? Gay men get this ad: Only Politician Foo protects gay security in the streets at night! Homophobes get this ad: Then you must vote for Politician Foo, only he opposses gay marriage! Pro-lifers get this ad: Vote for Foo to protect the children from the very first moment! Pro-abortion get this ad: Foo protects women's rights! Advertising should be as general as possible - microtargeting people in a democracy is extraordinarily dangerous.

hn_throwaway_99 3y ago

I think this study highlights a specific instance of an issue I've previously argued, which is basically "Data analysis at scale completely breaks many intuitive assumptions humans hold.", privacy being one of those assumptions. For example, twenty years or so ago, I think the vast, vast majority of people in the US would agree that you don't have a right to privacy walking around a public street. However, once the Internet and Streetview came along, people had very different feelings about whether someone should be able to post a video they took of you on the Internet, for anyone across the world to see for all eternity, oh and we can also use facial recognition (just from public data and photos other people have posted) to fully identify you. Privacy is just one example. I will personally say my opinions on free speech have also changed. Given how highly targeted, systematic and relentless well-funded misinformation campaigns can be, the old assumptions I had about free speech ("Given enough daylight, the majority of people will walk away from 'bad' ideas") I no longer believe to be true. I really don't have any idea what the "right" solution is, just that my views have changed. The primary point is that wide scale, incredibly cheap data analysis (now being supercharged with AI) basically breaks all these fundamental assumptions and ideas society has, which is one reason why the world seems to be in such turmoil.

downboots 3y ago
Log_2(8 billion people) = ~33 yes/no questions
- kridsdale3 3y ago
  So why isn't my social security number 4 bytes? I could just be "$jA9" when ascii encoded.
jmugan 3y ago
When I first saw Interstellar I thought it was unrealistic that people could be brainwashed to forget about space flight. When I watched it again last year it seemed entirely plausible.
- Ialdaboth 3y ago

mock-possum 3y ago

> show that current LLMs can infer a wide range of personal attributes That’s not invasion of privacy, that’s just making good (informed) guesses.

soul_grafitti 3y ago

It seems to me this is similar to the way poker players observe 'tells' in the other players - cataloging innocuous observations to make highly specific inferences. I know people who are naturally observant and quite good at this. As I think about it this is really what makes Sherlock Holmes so interesting too. What's changing is the scope, scale and access. It's no longer necessary to be a savant or have a special talent - all you need are hundreds of banal data points easily scooped from the meta data wake we all leave. Not to mention it can be programmed and commoditized instead of being a special attribute of another human. The realization feels a bit like climate change in that some folks saw it coming, pointed it out, articulated the issues, underestimated how fast it would actually arrive and how profound the impact would be. But back to the point at hand you can't legislate people not see what is publicly before them. You might criminalize recording that data for a while but that doesn't seem to work long term thought it's brutal in the short term. If you can record that data you can analyze that data. From analysis comes inferences and decisions and actions. I think the more salient question is about equality. I have some ability to obfuscate my meta data wake but as an individual it's pretty limited. Apple, google, Amazon have much more ability to do so despite leaving a gazillion more bits of data laying around. Tools individuals can use to obscure our meta data trail will do more to maintain privacy than trying to tell someone observing me what they can and can't do with their observations.

dcow 3y ago
Yeah this is the issue with the approach of trying to control what companies can and can’t do with the inferences they make from essentially public data they collect about you. Recent privacy policy is like going to an entity and saying “you can’t remember that customer 357 showed up in a BMW and wore a red jacket and told you their name is Jeff Geoffrey when they visited your store”. It’s not that I disagree with the desire to legislate this stuff in defense of privacy, but rather I don’t feel like the approach of treating the symptoms is correct. If we want to enhance privacy then legislate browsers and platforms, not a company’s eyeballs. Make sure browsers provide all the knobs people need to control the size of their data wake.
- em-bee 3y ago

upupupandaway 3y ago

I had thought of doing the exact same thing, but then cross-referencing with LinkedIn and other non-anonymous social networks to go the extra step and actually identify people with a certain level of confidence. I think we’re months, maybe days away from someone actually building this product and using it for nefarious purposes.

yetanother12345 3y ago
> I think we’re months, maybe days away from ... I have reasons to think years. Ie. that you are years behind. Profiling individuals based on public data already exist, and it's a profitable business as well. > using it for nefarious purposes. Law enforcement (and various other state-owned entities) would beg to disagree. As for vendors, same. Edit: Ad networks, even.

radar1310 3y ago

Not surprised at all.

Xcelerate 3y ago

It seems quite clear that inferring traits about people is something LLMs will only become better at with time. Everyone claims MBTI is akin to astrology, which means there should be no predictive capacity of those four letters. But just for fun, I gave GPT-4 the top 20 songs of my music playlist and asked it to guess my MBTI type, and it correctly did so. I repeated the process with two friends (asking their MBTI type prior to querying GPT) and it guessed them correctly as well. If MBTI is no different than random noise and GPT lacks inference capabilities with regard to humans, the probability of this occurring due to chance alone is less than 1 in 16^3 (*approximately—non-uniform distribution of types).

blovescoffee 3y ago
I don’t think anyone says MBTI is random. I think the doubt is in the predictive power of a whichever MBTI classification. The test for your MB label is pretty well designed to extract “some” information. Academics just don’t think the labeling is particularly valuable.
- Xcelerate 3y ago
  Fair point. But are the “superior” systems like Big 5 (or more recent approaches that utilize independent component analysis) really that much more predictive than MBTI? It seems like there is a lot of squabbling over the difference of a few percentage points in information gain.
  - pgeorgi 3y ago
    To a layman, they're all useless, and in a dangerous way: they transport a sense of understanding a person/character/psyche that they don't support. A professional can gain _some_ information from all of them. Just that the Big 5 have much more literature behind them than MBTI, so there's more reference data for any given constellation. That's a reinforcement loop. Methodical differences and reliability and all that fluff can be used to argue in more detail in favor of one versus the other, but by now, it comes down to what folks are more familiar with. In the end, these are just a few out of many, many tools of the field. If you see somebody build their career on "(MBTI|Big 5|...) analysis" and nothing else, they're not a professional but a quack.

yafbum 3y ago

Inferring someone's demographic attributes by observing their behaviors isn't very new or specific to LLMs. Basic stats can do the job well enough, and ML classification models have been trained on this for a while. What's new here I guess is that LLMs already have such inference capability built in, even though they are very general-purpose?