6 ms·

In short: their finding is that quantizing a model undoes various “unlearning” methods. An unlearning method is a specific update to model weights that make it

In short: their finding is that quantizing a model undoes various “unlearning” methods. An unlearning method is a specific update to model weights that make it forget specific facts. These are often meant to satisfy copyright claims, although I don’t know if these are ever used in practice. I feel that this needs a good threat model analysis. Like, you possess an fp32 model, which someone has fine-tuned to forget some facts, which you can then quantize to recover those facts. When would this lead to a dangerous situation?

discreteevent 2y ago
Unlearning is described as: "process aims to erase specific knowledge from LLMs while preserving as much model utility as possible." i.e. We know that our model would be useless without your source. So we will take the useful part of your source and obfuscate the rest so that we can charge our users for utility provided by you without having to pay you anything.
- short_sells_poo 2y ago
  > We know that our model would be useless without your source. So we will take the useful part of your source and obfuscate the rest so that we can charge our users for utility provided by you without having to pay you anything. Isn't this basically the entirety of the latest AI craze? They basically took a public good - the information available on the Internet - and hid behind some thin veneer of "we are not stealing, we just trained an AI on the information" and then they sell it. Note, I'm intentionally not writing "free information available on the Internet", because information is not free. Someone has to pay (in time or money) to generate it and host it. They might have provided it gratis to the public, but nobody asked them if an AI can come along, harvest it all and regurgitate it without a hint of reference to the original source. Much of that information is not even free in the monetary sense, it is supported by ads. The AI will not only not click through the \ds, it won't even generate repeat traffic as once the information is harvested, there's no need to access the source anymore. If you really think about it, it's a brilliant business model. It's a perfect theft, where the affected group is too diffuse and uncoordinated, it's extremely difficult to prove anything anyway, and the "thieves" are flush with investment capital so they sleep well at night. LLMs have undoubtedly great utility as a research tool and I'm not at all against them. I think they (or a model similar in objectives) are the next step in accessing the knowledge humanity has amassed. However, there's a distinct danger that they will simply suck they sources dry and leave the internet itself even more of a wasteland than it has already become. I have no illusions that AI companies will simply regress to the lowest cost solution of simply not giving anything back to whoever created the information in the first place. The fact that they are cutting off the branch that they are sitting on is irrelevant for them, because the current crop of owners will be long gone with their billions by the time the branch snaps.
  - cachvico 2y ago
    I think it's fair and reasonable to assume that the AI companies will at some point start licensing their source content. Through gov/legal oversight or not remains to be seen, but OpenAI are already beginning to do so: https://searchengineland.com/openais-growing-list-of-partnerships-442974 https://searchengineland.com/openais-growing-list-of-partner...
    - rvnx 2y ago
      Google is using for 20 years unlicensed source content for their search snippets, they seem to be doing fine with it (with the exception of few news publishers).
      - grugagag 2y ago
        The idea with internet search was to get people to find the source of the information they were searching for. As a matter of fact a lot information indexing was requested at the source. Google did respect the bargain for a while until they started to obfuscate getting to the source with AMP and their info snippets directly in the search, bypassing redirecting to the source. Then they started not displaying all that info at all, not even on the nth page of search results. The broth has been getting sour for a while now. Some people never wanted crawlers indexing and there were numerous discussions about how those robot.txt were ignored. So what I see here is the historical trend broken bargains which is more or less digital theft
    - short_sells_poo 2y ago
      Thanks for the link, I appreciate it. I suppose the issue is that this just further enshittifies the internet into a small handful of walled gardens. Big players get their payday, because they could feasibly sue OpenAI and generate them enough headache. But the vast amount of content on the internet was not built by a small handful of media companies, but rather by masses of small creators. It is their work that OpenAI is profiting from and I have yet to see a credible suggestion on how they will compensate them.
- rowanG077 2y ago
  That's not entirely true. Retraining is very expensive. If you can train on a very large dataset including proprietary knowledge and then postprocess the model cheaply to forget things saves you retraining for every variation.
- seanmcdirmid 2y ago
  I thought it was even worse than that: learning any of the corpus verbatim would actually reduce model utility.
  - edude03 2y ago
    Yes although how close to verbatim is debatable. For example there are questions that you’d ask that other people have asked many times before that you’d like the exact answer for (e.g. when does daylight saving time end?)
    - startupsfail 2y ago
      > that make it forget specific facts. These are often meant to satisfy copyright claims Facts are not copyrightable. To quote copyright.gov: “Copyright does not protect facts, ideas, systems, or methods of operation, although it may protect the way these things are expressed.”
      - PittleyDunkin 2y ago
        What is a fact without expression? It's not clear under what interpretation might be necessary to get the quoted sentiment to be considered sensical.
      - deleted 2y ago
        [deleted]
    - seanmcdirmid 2y ago
      Wouldn't that be stuffed in the prompt anyways? No reason for the LLM to learn that.
  - int_19h 2y ago
    It really depends on which part of the corpus, though. I do expect my LM to be able to reproduce culturally important citations, for example.
wongarsu 2y ago
I assume everyone who has someone with an AI safety job title uses unlearning to make sure their models don't remember how to make common illegal drugs, poisons or explosives. The threat model here is probably more accidental un-unlearning these facts and distributing those models (as is common with quantized models). Most of this "dangerous" information is readily available in textbooks, patents, amateur chemistry forums etc. But as a society we generally assume that those smart enough to find and understand that kind of information are smart enough not to abuse it. We just don't want Mythbusters to explain it on prime-time TV, or ChatGPT explaining it to people
- aziaziazi 2y ago
  Mythbusters chooses the subjects he discusses while ChatGPT responses depends on the context (you) provided. It will give you a list a poisons if you asked (5 seconds), as well as an encyclopédie or Google would (30 seconds). Mythbuster broadcasting poisons recipes could seeds bad ideas that wouldn’t been triggered otherwise. ChatGPT wouldn’t give a poisons recipe if not asked specifically.
  - FergusArgyll 2y ago
    This is a decent point. I hadn't really thought of a good reason why we e.g. sell old army manuals with step by step guides on making almost anything but there's no (afaik) HBO mini-series "Learn guerilla warfare"
- mschuster91 2y ago
  > But as a society we generally assume that those smart enough to find and understand that kind of information are smart enough not to abuse it. There's an almost complete step by step guide for most explosives on Wikipedia. The problem is that decisionmakers and regulators are excessively dumb - "AI bad" reigns supreme over the fact that Wikipedia tells you more about making bombs, even nuclear bombs if you want, than ChatGPT. AI in its current form is still bad - from all the IP issues over the environmental cost to it enabling spam, harassment and deception on a speed, scale and easiness not seen before in history - but most of the stuff where "regulators" cry about is just frankly bullshit.
jebarker 2y ago
More generally than "unlearning", I wonder if taking any fp16 model and running it in fp32 or fp64 does anything positive to it? e.g. exposes knowledge that isn't accessible at the lower precision
- spencerchubb 2y ago
  Correct me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
  - jebarker 2y ago
    I don't think that's always correct when you're talking about operators in neural nets. E.g. the sin and cos in rope embeddings would get more precise, large sums like softmax would become more precise, potentially attention too due to dot products
- fjdjshsh 2y ago
  If the model wasn't optimized with a parameter = 3.08 (that is, the optimization figured 3.08 increased the objective function), why would it be better than parameter = 3.1 ? (This applies to functions of parameters too)
JKCalhoun 2y ago
We'll have LLMs trying to root out "Manchurian LLMs".

discreteevent 2y ago

Unlearning is described as: "process aims to erase specific knowledge from LLMs while preserving as much model utility as possible." i.e. We know that our model would be useless without your source. So we will take the useful part of your source and obfuscate the rest so that we can charge our users for utility provided by you without having to pay you anything.

short_sells_poo 2y ago
> We know that our model would be useless without your source. So we will take the useful part of your source and obfuscate the rest so that we can charge our users for utility provided by you without having to pay you anything. Isn't this basically the entirety of the latest AI craze? They basically took a public good - the information available on the Internet - and hid behind some thin veneer of "we are not stealing, we just trained an AI on the information" and then they sell it. Note, I'm intentionally not writing "free information available on the Internet", because information is not free. Someone has to pay (in time or money) to generate it and host it. They might have provided it gratis to the public, but nobody asked them if an AI can come along, harvest it all and regurgitate it without a hint of reference to the original source. Much of that information is not even free in the monetary sense, it is supported by ads. The AI will not only not click through the \ds, it won't even generate repeat traffic as once the information is harvested, there's no need to access the source anymore. If you really think about it, it's a brilliant business model. It's a perfect theft, where the affected group is too diffuse and uncoordinated, it's extremely difficult to prove anything anyway, and the "thieves" are flush with investment capital so they sleep well at night. LLMs have undoubtedly great utility as a research tool and I'm not at all against them. I think they (or a model similar in objectives) are the next step in accessing the knowledge humanity has amassed. However, there's a distinct danger that they will simply suck they sources dry and leave the internet itself even more of a wasteland than it has already become. I have no illusions that AI companies will simply regress to the lowest cost solution of simply not giving anything back to whoever created the information in the first place. The fact that they are cutting off the branch that they are sitting on is irrelevant for them, because the current crop of owners will be long gone with their billions by the time the branch snaps.
- cachvico 2y ago
  I think it's fair and reasonable to assume that the AI companies will at some point start licensing their source content. Through gov/legal oversight or not remains to be seen, but OpenAI are already beginning to do so: https://searchengineland.com/openais-growing-list-of-partnerships-442974 https://searchengineland.com/openais-growing-list-of-partner...
  - rvnx 2y ago
    Google is using for 20 years unlicensed source content for their search snippets, they seem to be doing fine with it (with the exception of few news publishers).
    - grugagag 2y ago
      The idea with internet search was to get people to find the source of the information they were searching for. As a matter of fact a lot information indexing was requested at the source. Google did respect the bargain for a while until they started to obfuscate getting to the source with AMP and their info snippets directly in the search, bypassing redirecting to the source. Then they started not displaying all that info at all, not even on the nth page of search results. The broth has been getting sour for a while now. Some people never wanted crawlers indexing and there were numerous discussions about how those robot.txt were ignored. So what I see here is the historical trend broken bargains which is more or less digital theft
  - short_sells_poo 2y ago
    Thanks for the link, I appreciate it. I suppose the issue is that this just further enshittifies the internet into a small handful of walled gardens. Big players get their payday, because they could feasibly sue OpenAI and generate them enough headache. But the vast amount of content on the internet was not built by a small handful of media companies, but rather by masses of small creators. It is their work that OpenAI is profiting from and I have yet to see a credible suggestion on how they will compensate them.
rowanG077 2y ago
That's not entirely true. Retraining is very expensive. If you can train on a very large dataset including proprietary knowledge and then postprocess the model cheaply to forget things saves you retraining for every variation.
seanmcdirmid 2y ago
I thought it was even worse than that: learning any of the corpus verbatim would actually reduce model utility.
- edude03 2y ago
  Yes although how close to verbatim is debatable. For example there are questions that you’d ask that other people have asked many times before that you’d like the exact answer for (e.g. when does daylight saving time end?)
  - startupsfail 2y ago
    > that make it forget specific facts. These are often meant to satisfy copyright claims Facts are not copyrightable. To quote copyright.gov: “Copyright does not protect facts, ideas, systems, or methods of operation, although it may protect the way these things are expressed.”
    - PittleyDunkin 2y ago
      What is a fact without expression? It's not clear under what interpretation might be necessary to get the quoted sentiment to be considered sensical.
    - deleted 2y ago
      [deleted]
  - seanmcdirmid 2y ago
    Wouldn't that be stuffed in the prompt anyways? No reason for the LLM to learn that.
- int_19h 2y ago
  It really depends on which part of the corpus, though. I do expect my LM to be able to reproduce culturally important citations, for example.

wongarsu 2y ago

I assume everyone who has someone with an AI safety job title uses unlearning to make sure their models don't remember how to make common illegal drugs, poisons or explosives. The threat model here is probably more accidental un-unlearning these facts and distributing those models (as is common with quantized models). Most of this "dangerous" information is readily available in textbooks, patents, amateur chemistry forums etc. But as a society we generally assume that those smart enough to find and understand that kind of information are smart enough not to abuse it. We just don't want Mythbusters to explain it on prime-time TV, or ChatGPT explaining it to people

aziaziazi 2y ago
Mythbusters chooses the subjects he discusses while ChatGPT responses depends on the context (you) provided. It will give you a list a poisons if you asked (5 seconds), as well as an encyclopédie or Google would (30 seconds). Mythbuster broadcasting poisons recipes could seeds bad ideas that wouldn’t been triggered otherwise. ChatGPT wouldn’t give a poisons recipe if not asked specifically.
- FergusArgyll 2y ago
  This is a decent point. I hadn't really thought of a good reason why we e.g. sell old army manuals with step by step guides on making almost anything but there's no (afaik) HBO mini-series "Learn guerilla warfare"
mschuster91 2y ago
> But as a society we generally assume that those smart enough to find and understand that kind of information are smart enough not to abuse it. There's an almost complete step by step guide for most explosives on Wikipedia. The problem is that decisionmakers and regulators are excessively dumb - "AI bad" reigns supreme over the fact that Wikipedia tells you more about making bombs, even nuclear bombs if you want, than ChatGPT. AI in its current form is still bad - from all the IP issues over the environmental cost to it enabling spam, harassment and deception on a speed, scale and easiness not seen before in history - but most of the stuff where "regulators" cry about is just frankly bullshit.

jebarker 2y ago

More generally than "unlearning", I wonder if taking any fp16 model and running it in fp32 or fp64 does anything positive to it? e.g. exposes knowledge that isn't accessible at the lower precision

spencerchubb 2y ago
Correct me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
- jebarker 2y ago
  I don't think that's always correct when you're talking about operators in neural nets. E.g. the sin and cos in rope embeddings would get more precise, large sums like softmax would become more precise, potentially attention too due to dot products
fjdjshsh 2y ago
If the model wasn't optimized with a parameter = 3.08 (that is, the optimization figured 3.08 increased the objective function), why would it be better than parameter = 3.1 ? (This applies to functions of parameters too)

JKCalhoun 2y ago

We'll have LLMs trying to root out "Manchurian LLMs".