11 ms·

I've been saying this for years, but every post you make online, every unencrypted email, IM, text message, etc, will eventually end up getting sold off as trai

by px43 6y ago

I've been saying this for years, but every post you make online, every unencrypted email, IM, text message, etc, will eventually end up getting sold off as training data for future machine learning projects. Every company stores this stuff for ages, and the value of candid conversation data just keeps increasing. Eventually these companies are either going to get hacked, get bought, or go bankrupt, and all the cleartext data they hold is going to get passed around to various data markets and end up incorporated into GPT-12 or whatever. The moral of the story here isn't that companies need to stop storing data, or that we need to run ML researchers out of town. It's that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world. Otherwise these systems, which are being trained to extract as much value out of you and the ones you love as possible, will use this data you're giving them for free against you, and it'll be to late to do anything about it then.

jrockway 6y ago
Isn't that how humans work? Sometimes I wonder if I have free will and sentience, or if I am just stringing words together based on some probability function. I've no doubt taken into account copyrighted works and personal information when training my built-in neural network. The examples the article gives like "misremembering" the murder as the murder victim sounds like something a person would do. Knowing verbatim contact information of some random person is also possible. All in all, GPT-2 sounds a lot like us.
- jakear 6y ago
  It’s my understanding that under our current scientific understanding of the mind, we cannot have “free will”. Where would it come from? All science knows is that neurons receive inputs from sensors, mixes them together (with some quantum randomness sprinkled in), and eventually actuates muscles to produce outputs, exactly as GPT. Thus “free will” must come from some sort of faith. Either it was given to us by “God”, or we’re all part of some simulation and it exists on a level we don’t know about, or something else entirely too advanced for us to understand.
  - ben_w 6y ago
    It depends what you mean by “free will” — the phrase “free will” is as ill-defined as “common sense” and “consciousness”. I take the view that ∀ x ∈ <free will definitions>, Has(x, humans) ≡ Has(x, AI)
    - jakear 6y ago
      I agree with your claim. I also believe many to see “free will”/consciousness/etc as that certain je ne sais quoi that separates the human experience from that of computers: ∀ x ∈ <free will definitions>, Has(x, humans) ≡ ~Has(x, AI) For these to both be true, <free will definitions> becomes empty. Thus I argue we have none. Edit, or rather it’s meaningless to argue if we have any or not, as it cannot be defined.
      - ben_w 6y ago
        Yes, that also fits my observations of such arguments.
      - simplify 6y ago
        > or rather it’s meaningless to argue if we have any or not, as it cannot be defined. This sums up my view too. Can't argue about it if you can't define it.
  - lolc 6y ago
    Free will is a cultural concept, not a biological one. Consciousness may be driven by a biological machine, but development of our ego is influenced a lot by culture. Free will is faith in the sense that we learn the concept from our culture as our sense of self develops. The assumption of having it shapes our thinking such that we can say we actually have it. A person who took their inner voice as voice from the gods would have a restricted free will in this view.
- hyperpape 6y ago
  Of course it isn't how the human mind works. You can ask a person "did you come up with that sentence, or is it something you read?" And they can answer that question. The answer isn't perfectly reliable, but it isn't completely unreliable either. You can't do that with GPT-2/GPT-3. Big picture: the human mind is a machine of sorts. It's not the same kind of machine as any form of machine learning we've developed so far.
  - ben_w 6y ago
    > The answer isn't perfectly reliable, but it isn't completely unreliable either. How would we be able to tell how reliable it is in general? """The left brain of people whose hemispheres have been disconnected has been observed to invent explanations for body movement initiated by the opposing (right) hemisphere, perhaps based on the assumption that their actions are consciously willed.""" - https://en.wikipedia.org/wiki/Neuroscience_of_free_will#Related_brain_disorders https://en.wikipedia.org/wiki/Neuroscience_of_free_will#Rela...
    - alisonkisk 6y ago
      The point is that humans have a basic ability to make specific coherent observations about themselves and the world. GPT can't do that except occasionally by accident in the middle of a muddle.
  - _flux 6y ago
    In this case you could do it with GPT-x as well, if you have access to the learning material (so the OpenAI could do it when providing the model as a service): grep the learning data after-the-fact. No idea if it would be feasible to integrate this information directly into the model. With my limited understanding of neural networks, this seems difficult.
ojnabieoot 6y ago
> The moral of the story here isn't that companies need to stop storing data, or that we need to run ML researchers out of town. There is a lot of space between “user data and ML research on that data needs to be closely regulated” and “run the ML researchers out of town.” This uncharitable exaggeration makes critics sound like an uninformed mob and closes yourself off to legitimate ideas. This directly leads to logically flawed and morally gross suggestions like: > It's that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world. It is simply not reasonable to expect most Facebook/Skype/etc users to know enough about encryption to make good decisions here - for the same reason that you can’t expect people to have detailed understanding of food science to protect themselves from unscrupulous grocers, or to have a detailed understanding of medicine to protect themselves from fraudulent doctors. It really doesn’t matter if this knowledge has been around for “decades” since it’s still specialist knowledge. Protecting digital privacy from unscrupulous tech companies and unethical ML researchers is a job for the government. Suggesting otherwise is victim-blaming. “The problem is that society is dumb and needs to become smart through the power of self-righteous scolding” is not helpful.
- ClumsyPilot 6y ago
  Its amazing that such a sober take on the issue is getting downvoted without a single counterargument
  - grawprog 6y ago
    Probably because the comment can be mostly summed up to: 'The majority of humans are idiots who need to be protected from themselves by the government.'
    - chrismorgan 6y ago
      It’s not that humans need to be protected from themselves (individually), but that they need to be protected from other unscrupulous people, because computers are empowering the unscrupulous (and even the accidentally unscrupulous) more than they are empowering the common person. I broadly agree with ojnabieoot.
visarga 6y ago
Maybe this data will be used to 'upload' you in a virtual world after the singularity. It might be your second life.
- flatline 6y ago
  When thinking about this, you have to ask, who will profit, and how? It doesn’t seem like a clone of your brain running in a simulation is going to be of much advantage to you from the current industry incentives. Maybe if you’re rich?
  - ben_w 6y ago
    If you make some fairly plausible assumptions about Moores law (and equivalents), the simulations are cheaper than the meat bodies before mid-century. • https://worldbuilding.stackexchange.com/questions/51746/when-will-uploaded-minds-be-a-reality/67531#67531 https://worldbuilding.stackexchange.com/questions/51746/when... • https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brains/ https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brai...
    - ClumsyPilot 6y ago
      That is cool, but tangential to the question posed above - given that every part of surveiliance capitalism is currently working against my interest, either to sell me something I dont need, to inflienxe the way I vote, etc. Will the simulation of my brain belong to a megacorp, and be used to do even more damage?
      - visarga 6y ago
        Black Mirror stuff. https://en.wikipedia.org/wiki/White_Christmas_(Black_Mirror) https://en.wikipedia.org/wiki/White_Christmas_(Black_Mirror)
      - sli 6y ago
        There is simply no way corporate hegemony will allow you to own the rights to the digital copy of you that they built using the tools of surveillance capitalism. No corporation or lawmaker will grant you rights over corporate property, even if that property was obtained through questionable means and/or your consent was never actually received. That's why these data snooping agreements corporations force into their products reserve the right to do whatever they want with your data. The obvious response to this is, "Then don't patron those companies," which is just as naive and oversimplified of a solution as the one above suggesting everyone everywhere should just use encryption (but for very different reasons). There is just simply no logical reason to believe consumers would ever get data rights over corporations under capitalism with major reforms putting corporations back under our boots where they belong.
fastball 6y ago
End-to-end encryption is a possible solution to some of this, no?
- ClumsyPilot 6y ago
  I can't use a single public service with E2E encrypted email, whether thats private or commerial. General state of encryption technology is that it's arcane and unusable outside of like a couple of chat apps
  - fastball 6y ago
    Email you can encrypt with PGP tho? The problem isn't really the technology, it's that the people you are interacting with on the internet don't care about privacy as much as you do, and obviously when it comes to keeping things private, everyone has to participate.
chrismorgan 6y ago
> people really need to start using the encryption technologies […] Otherwise these systems, which are being trained to extract as much value out of you […] as possible, will use this data […] against you, […] You are proposing a technical solution to a social problem. That pretty much never works.
agumonkey 6y ago
Data rots. What was true 10 years ago about me is not necessary the case now.
- courtf 6y ago
  It's one of the biggest lies in tech right now, that all this user data is some untapped gold mine, as if text on the internet is the most important measure of human behavior. It's all going to end up rotting on backups while the marketing department spins yarns about the amazing AI that's going to spring forth any day now from a pile of tweets. It's like trying to build a dog by hanging around the park collecting turds.
  - agumonkey 6y ago
    is the data market a bubble ?
    - courtf 6y ago
      Depends what you're doing with it, but I have to think it mostly is a bubble. Maybe if you have a really simple question that already fits what can be measured, or you're trying to surveil someone in particular, but for stuff like AI just doesn't make sense to me.
    - nightcracker 6y ago
      Answer this for yourself: what percentage of ads that you see is relevant? There's your answer.
      - agumonkey 6y ago
        I'm so mentally blind to ads I could never consider my opinion worth a cent on this :D
gwern 6y ago
The value of data decreases. There are diminishing returns to dataset sizes (specifically, for GPT, it's a power law). The more public data there is, and the more data in general, the less any additional piece of private data is worth. Why would any tech company with access to lots of private data, like Google, risk the enormous backlash and legal penalties when they can so easily scrape terabytes and terabytes of text from public datasets to train uncannily intelligent models like GPT-3? Even hobbyists can make the necessary datasets from Common Crawl et al (see EleutherAI's terabyte+ of clean text in The Pile dataset). This is in addition to the ever greater sample-efficiency of bigger models, which learn eerily fast from just a handful of datapoints, rendering the supposed 'moats' of big giant proprietary datasets ever more moot...
headbansown 6y ago
"If one would give me six lines written by the hand of the most honest man, I would find something in them to have him hanged." -- Richelieu

by px43 6y ago

jrockway 6y ago

Isn't that how humans work? Sometimes I wonder if I have free will and sentience, or if I am just stringing words together based on some probability function. I've no doubt taken into account copyrighted works and personal information when training my built-in neural network. The examples the article gives like "misremembering" the murder as the murder victim sounds like something a person would do. Knowing verbatim contact information of some random person is also possible. All in all, GPT-2 sounds a lot like us.

jakear 6y ago
It’s my understanding that under our current scientific understanding of the mind, we cannot have “free will”. Where would it come from? All science knows is that neurons receive inputs from sensors, mixes them together (with some quantum randomness sprinkled in), and eventually actuates muscles to produce outputs, exactly as GPT. Thus “free will” must come from some sort of faith. Either it was given to us by “God”, or we’re all part of some simulation and it exists on a level we don’t know about, or something else entirely too advanced for us to understand.
- ben_w 6y ago
  It depends what you mean by “free will” — the phrase “free will” is as ill-defined as “common sense” and “consciousness”. I take the view that ∀ x ∈ <free will definitions>, Has(x, humans) ≡ Has(x, AI)
  - jakear 6y ago
    I agree with your claim. I also believe many to see “free will”/consciousness/etc as that certain je ne sais quoi that separates the human experience from that of computers: ∀ x ∈ <free will definitions>, Has(x, humans) ≡ ~Has(x, AI) For these to both be true, <free will definitions> becomes empty. Thus I argue we have none. Edit, or rather it’s meaningless to argue if we have any or not, as it cannot be defined.
    - ben_w 6y ago
      Yes, that also fits my observations of such arguments.
    - simplify 6y ago
      > or rather it’s meaningless to argue if we have any or not, as it cannot be defined. This sums up my view too. Can't argue about it if you can't define it.
- lolc 6y ago
  Free will is a cultural concept, not a biological one. Consciousness may be driven by a biological machine, but development of our ego is influenced a lot by culture. Free will is faith in the sense that we learn the concept from our culture as our sense of self develops. The assumption of having it shapes our thinking such that we can say we actually have it. A person who took their inner voice as voice from the gods would have a restricted free will in this view.
hyperpape 6y ago
Of course it isn't how the human mind works. You can ask a person "did you come up with that sentence, or is it something you read?" And they can answer that question. The answer isn't perfectly reliable, but it isn't completely unreliable either. You can't do that with GPT-2/GPT-3. Big picture: the human mind is a machine of sorts. It's not the same kind of machine as any form of machine learning we've developed so far.
- ben_w 6y ago
  > The answer isn't perfectly reliable, but it isn't completely unreliable either. How would we be able to tell how reliable it is in general? """The left brain of people whose hemispheres have been disconnected has been observed to invent explanations for body movement initiated by the opposing (right) hemisphere, perhaps based on the assumption that their actions are consciously willed.""" - https://en.wikipedia.org/wiki/Neuroscience_of_free_will#Related_brain_disorders https://en.wikipedia.org/wiki/Neuroscience_of_free_will#Rela...
  - alisonkisk 6y ago
    The point is that humans have a basic ability to make specific coherent observations about themselves and the world. GPT can't do that except occasionally by accident in the middle of a muddle.
- _flux 6y ago
  In this case you could do it with GPT-x as well, if you have access to the learning material (so the OpenAI could do it when providing the model as a service): grep the learning data after-the-fact. No idea if it would be feasible to integrate this information directly into the model. With my limited understanding of neural networks, this seems difficult.

ojnabieoot 6y ago

> The moral of the story here isn't that companies need to stop storing data, or that we need to run ML researchers out of town. There is a lot of space between “user data and ML research on that data needs to be closely regulated” and “run the ML researchers out of town.” This uncharitable exaggeration makes critics sound like an uninformed mob and closes yourself off to legitimate ideas. This directly leads to logically flawed and morally gross suggestions like: > It's that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world. It is simply not reasonable to expect most Facebook/Skype/etc users to know enough about encryption to make good decisions here - for the same reason that you can’t expect people to have detailed understanding of food science to protect themselves from unscrupulous grocers, or to have a detailed understanding of medicine to protect themselves from fraudulent doctors. It really doesn’t matter if this knowledge has been around for “decades” since it’s still specialist knowledge. Protecting digital privacy from unscrupulous tech companies and unethical ML researchers is a job for the government. Suggesting otherwise is victim-blaming. “The problem is that society is dumb and needs to become smart through the power of self-righteous scolding” is not helpful.

ClumsyPilot 6y ago
Its amazing that such a sober take on the issue is getting downvoted without a single counterargument
- grawprog 6y ago
  Probably because the comment can be mostly summed up to: 'The majority of humans are idiots who need to be protected from themselves by the government.'
  - chrismorgan 6y ago
    It’s not that humans need to be protected from themselves (individually), but that they need to be protected from other unscrupulous people, because computers are empowering the unscrupulous (and even the accidentally unscrupulous) more than they are empowering the common person. I broadly agree with ojnabieoot.

visarga 6y ago

Maybe this data will be used to 'upload' you in a virtual world after the singularity. It might be your second life.

flatline 6y ago
When thinking about this, you have to ask, who will profit, and how? It doesn’t seem like a clone of your brain running in a simulation is going to be of much advantage to you from the current industry incentives. Maybe if you’re rich?
- ben_w 6y ago
  If you make some fairly plausible assumptions about Moores law (and equivalents), the simulations are cheaper than the meat bodies before mid-century. • https://worldbuilding.stackexchange.com/questions/51746/when-will-uploaded-minds-be-a-reality/67531#67531 https://worldbuilding.stackexchange.com/questions/51746/when... • https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brains/ https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brai...
  - ClumsyPilot 6y ago
    That is cool, but tangential to the question posed above - given that every part of surveiliance capitalism is currently working against my interest, either to sell me something I dont need, to inflienxe the way I vote, etc. Will the simulation of my brain belong to a megacorp, and be used to do even more damage?
    - visarga 6y ago
      Black Mirror stuff. https://en.wikipedia.org/wiki/White_Christmas_(Black_Mirror) https://en.wikipedia.org/wiki/White_Christmas_(Black_Mirror)
    - sli 6y ago
      There is simply no way corporate hegemony will allow you to own the rights to the digital copy of you that they built using the tools of surveillance capitalism. No corporation or lawmaker will grant you rights over corporate property, even if that property was obtained through questionable means and/or your consent was never actually received. That's why these data snooping agreements corporations force into their products reserve the right to do whatever they want with your data. The obvious response to this is, "Then don't patron those companies," which is just as naive and oversimplified of a solution as the one above suggesting everyone everywhere should just use encryption (but for very different reasons). There is just simply no logical reason to believe consumers would ever get data rights over corporations under capitalism with major reforms putting corporations back under our boots where they belong.

fastball 6y ago

End-to-end encryption is a possible solution to some of this, no?

ClumsyPilot 6y ago
I can't use a single public service with E2E encrypted email, whether thats private or commerial. General state of encryption technology is that it's arcane and unusable outside of like a couple of chat apps
- fastball 6y ago
  Email you can encrypt with PGP tho? The problem isn't really the technology, it's that the people you are interacting with on the internet don't care about privacy as much as you do, and obviously when it comes to keeping things private, everyone has to participate.

chrismorgan 6y ago

> people really need to start using the encryption technologies […] Otherwise these systems, which are being trained to extract as much value out of you […] as possible, will use this data […] against you, […] You are proposing a technical solution to a social problem. That pretty much never works.

agumonkey 6y ago

Data rots. What was true 10 years ago about me is not necessary the case now.

courtf 6y ago
It's one of the biggest lies in tech right now, that all this user data is some untapped gold mine, as if text on the internet is the most important measure of human behavior. It's all going to end up rotting on backups while the marketing department spins yarns about the amazing AI that's going to spring forth any day now from a pile of tweets. It's like trying to build a dog by hanging around the park collecting turds.
- agumonkey 6y ago
  is the data market a bubble ?
  - courtf 6y ago
    Depends what you're doing with it, but I have to think it mostly is a bubble. Maybe if you have a really simple question that already fits what can be measured, or you're trying to surveil someone in particular, but for stuff like AI just doesn't make sense to me.
  - nightcracker 6y ago
    Answer this for yourself: what percentage of ads that you see is relevant? There's your answer.
    - agumonkey 6y ago
      I'm so mentally blind to ads I could never consider my opinion worth a cent on this :D

gwern 6y ago

The value of data decreases. There are diminishing returns to dataset sizes (specifically, for GPT, it's a power law). The more public data there is, and the more data in general, the less any additional piece of private data is worth. Why would any tech company with access to lots of private data, like Google, risk the enormous backlash and legal penalties when they can so easily scrape terabytes and terabytes of text from public datasets to train uncannily intelligent models like GPT-3? Even hobbyists can make the necessary datasets from Common Crawl et al (see EleutherAI's terabyte+ of clean text in The Pile dataset). This is in addition to the ever greater sample-efficiency of bigger models, which learn eerily fast from just a handful of datapoints, rendering the supposed 'moats' of big giant proprietary datasets ever more moot...

headbansown 6y ago

"If one would give me six lines written by the hand of the most honest man, I would find something in them to have him hanged." -- Richelieu