6 ms·

The problem of preventing leaks of forbidden knowledge seems impossible without excluding the forbidden knowledge from the training data. I fail to see how that

by norir 3y ago

The problem of preventing leaks of forbidden knowledge seems impossible without excluding the forbidden knowledge from the training data. I fail to see how that would be possible without crippling the desirable use cases. Companies need to be honest about this and stop pretending that it is a solvable problem with technical safeguards. Your child safety locks may work for a while, but eventually the child grows up and figures out how to bypass them. Also, sure, you can punish any users who ask about forbidden topics but is that really the kind of society we want to live in?

s1artibartfast 3y ago
>The problem of preventing leaks of forbidden knowledge seems impossible without excluding the forbidden knowledge from the training data. I fail to see how that would be possible without crippling the desirable use cases. This isnt accurate. There already several tools and methods explicitly for excluding forbidden knowledge from training data, or reweighting the training data so that the LLM have the perspective that the owners want. e.g. if you want the model from being racist, you run the training data through a classifier to remove data with negative sentiment around race. https://openai.com/research/dall-e-2-pre-training-mitigations https://openai.com/research/dall-e-2-pre-training-mitigation...
- apwell23 3y ago
  that doesn't address the last part "without crippling the desirable use cases." ?
  - s1artibartfast 3y ago
    It just cripples the undesirable use cases. It makes a targeted blind spot for the LLM, as decided by their owner.
- INGSOCIALITE 3y ago
  why is everyone saying the term "forbidden knowledge" like that's just fine and dandy?
  - s1artibartfast 3y ago
    because if you dont pretend you want your information censored, you get labeled as an antisocial racist.
    - sangnoir 3y ago
      How do you define censorship? Is it different from editing and curation? Is the the choice to exclude some sources from the training data "censorship"? How about the choice of loss minimization strategy?
      - s1artibartfast 3y ago
        I think excluding data and sources is both censorship, and something more. Excluding data is censoring from them model. Then when the model provides incorrect or biased, the firm is actively deceiving the user. Im not sure what you mean by loss minimization strategy. what I said is focused on the case where data is manipulated with the intent to curate a different perspective than an the source data.
adamrezich 3y ago
this is the only saving grace in this looming nightmare world of corporate-run omniscient-seeming language-model "oracles" that refuse to espouse any Truth deemed sufficiently politically inconvenient.
jacinda 3y ago
Yes, the fact that this is an extremely hard problem seems to be part of our mythology. Obtaining forbidden knowledge are part of many early creation stories (Garden of Eden - Tree of Knowledge, Prometheus and fire, etc.).
commandlinefan 3y ago
I've thought for decades that everybody would eventually come to their senses and realize that censorship in general is impossible to achieve, but they're still trying as hard as they ever were, and I've come to doubt that they'll ever give up on it.
- rnd0 3y ago
  You have to shift your thinking to realize that their problem isn't with words or even knowledge itself; but with protecting their power or ideology. In short, fanatics and power mongers are always going to fanatic and power monger.
  - rexpop 3y ago
    Everyone has thresholds behind which they store fanaticism. I am sure you would like to see cencored the location and combination to your home safe, your bank balance and passwords, etc. Why? Because distribution of this knowledge might potentially harm you.
    - commandlinefan 3y ago
      This (common) misconstrual of censorship is disingenuous. Scott Alexander proposed a better definition of censorship than "something somebody wants to keep secret": "something that somebody wants to share, and somebody else wants to consume, but a third party wants to suppress".
      - lazyasciiart 3y ago
        So if I think you’re a jerk and someone should steal your shit for lolz, so I want to share your address and safe combo, and someone else wants to have them to cash in, but you don’t want that to happen….
      - deleted 3y ago
        [deleted]
- 3y ago
bambax 3y ago
Yes. When Google finally made the whole of the web searchable, what would have happened, what would we have said, if they had decided some pages were inappropriate and therefore it should be their duty to prevent users to ever find them? Why should AI companies become editors of the human experience, and who decides what topics are off-limits? Sexuality? Isn't sexuality an important topic? Why can it not be part of our discussions with a chatbot? "Illegal" activities? Aside from the fact that all laws are local, there are many illegal activities that are described in great realistic details in novels, movies, documentaries. If chatbots can't discuss those, how long will it take for the corresponding books to be banned from public libraries? Etc. This seems quite vain in any case. LLMs are in many ways comparable to a compressed index of the web; if it's on the Internet, it will end up in a model, and once there, it will be possible to extract it.
- 1oooqooq 3y ago
  > Why should AI companies become editors of the human experience, and who decides what topics are off-limits? because that is the point of "AI". Search is getting less and less useful to both please advertisers and/or to hide piracy/forbidden content. To the point google is useless today. AI is nothing but an interface for search. Training is just the new indexing. And being the interface to search, it must control all that google is expected to control today.
- trogdor 3y ago
  >When Google finally made the whole of the web searchable, what would have happened, what would we have said, if they had decided some pages were inappropriate and therefore it should be their duty to prevent users to ever find them? Google does this every day.
- dragonwriter 3y ago
  > When Google finally made the whole of the web searchable, what would have happened, what would we have said, if they had decided some pages were inappropriate and therefore it should be their duty to prevent users to ever find them? Its called "deindexing" and Google (and every other search engine, probably, but Google owns enough of the usage that no really cares about the other ones) has done it forever, for a variety of reasons. And apparently, you wouldn't have said anything, because it happened, and you never even noticed it.

by norir 3y ago

s1artibartfast 3y ago

>The problem of preventing leaks of forbidden knowledge seems impossible without excluding the forbidden knowledge from the training data. I fail to see how that would be possible without crippling the desirable use cases. This isnt accurate. There already several tools and methods explicitly for excluding forbidden knowledge from training data, or reweighting the training data so that the LLM have the perspective that the owners want. e.g. if you want the model from being racist, you run the training data through a classifier to remove data with negative sentiment around race. https://openai.com/research/dall-e-2-pre-training-mitigations https://openai.com/research/dall-e-2-pre-training-mitigation...

apwell23 3y ago
that doesn't address the last part "without crippling the desirable use cases." ?
- s1artibartfast 3y ago
  It just cripples the undesirable use cases. It makes a targeted blind spot for the LLM, as decided by their owner.
INGSOCIALITE 3y ago
why is everyone saying the term "forbidden knowledge" like that's just fine and dandy?
- s1artibartfast 3y ago
  because if you dont pretend you want your information censored, you get labeled as an antisocial racist.
  - sangnoir 3y ago
    How do you define censorship? Is it different from editing and curation? Is the the choice to exclude some sources from the training data "censorship"? How about the choice of loss minimization strategy?
    - s1artibartfast 3y ago
      I think excluding data and sources is both censorship, and something more. Excluding data is censoring from them model. Then when the model provides incorrect or biased, the firm is actively deceiving the user. Im not sure what you mean by loss minimization strategy. what I said is focused on the case where data is manipulated with the intent to curate a different perspective than an the source data.

adamrezich 3y ago

this is the only saving grace in this looming nightmare world of corporate-run omniscient-seeming language-model "oracles" that refuse to espouse any Truth deemed sufficiently politically inconvenient.

jacinda 3y ago

Yes, the fact that this is an extremely hard problem seems to be part of our mythology. Obtaining forbidden knowledge are part of many early creation stories (Garden of Eden - Tree of Knowledge, Prometheus and fire, etc.).

commandlinefan 3y ago

I've thought for decades that everybody would eventually come to their senses and realize that censorship in general is impossible to achieve, but they're still trying as hard as they ever were, and I've come to doubt that they'll ever give up on it.

rnd0 3y ago
You have to shift your thinking to realize that their problem isn't with words or even knowledge itself; but with protecting their power or ideology. In short, fanatics and power mongers are always going to fanatic and power monger.
- rexpop 3y ago
  Everyone has thresholds behind which they store fanaticism. I am sure you would like to see cencored the location and combination to your home safe, your bank balance and passwords, etc. Why? Because distribution of this knowledge might potentially harm you.
  - commandlinefan 3y ago
    This (common) misconstrual of censorship is disingenuous. Scott Alexander proposed a better definition of censorship than "something somebody wants to keep secret": "something that somebody wants to share, and somebody else wants to consume, but a third party wants to suppress".
    - lazyasciiart 3y ago
      So if I think you’re a jerk and someone should steal your shit for lolz, so I want to share your address and safe combo, and someone else wants to have them to cash in, but you don’t want that to happen….
    - deleted 3y ago
      [deleted]
3y ago

bambax 3y ago

Yes. When Google finally made the whole of the web searchable, what would have happened, what would we have said, if they had decided some pages were inappropriate and therefore it should be their duty to prevent users to ever find them? Why should AI companies become editors of the human experience, and who decides what topics are off-limits? Sexuality? Isn't sexuality an important topic? Why can it not be part of our discussions with a chatbot? "Illegal" activities? Aside from the fact that all laws are local, there are many illegal activities that are described in great realistic details in novels, movies, documentaries. If chatbots can't discuss those, how long will it take for the corresponding books to be banned from public libraries? Etc. This seems quite vain in any case. LLMs are in many ways comparable to a compressed index of the web; if it's on the Internet, it will end up in a model, and once there, it will be possible to extract it.

1oooqooq 3y ago
> Why should AI companies become editors of the human experience, and who decides what topics are off-limits? because that is the point of "AI". Search is getting less and less useful to both please advertisers and/or to hide piracy/forbidden content. To the point google is useless today. AI is nothing but an interface for search. Training is just the new indexing. And being the interface to search, it must control all that google is expected to control today.
trogdor 3y ago
>When Google finally made the whole of the web searchable, what would have happened, what would we have said, if they had decided some pages were inappropriate and therefore it should be their duty to prevent users to ever find them? Google does this every day.
dragonwriter 3y ago
> When Google finally made the whole of the web searchable, what would have happened, what would we have said, if they had decided some pages were inappropriate and therefore it should be their duty to prevent users to ever find them? Its called "deindexing" and Google (and every other search engine, probably, but Google owns enough of the usage that no really cares about the other ones) has done it forever, for a variety of reasons. And apparently, you wouldn't have said anything, because it happened, and you never even noticed it.