6 ms·

I remember there is a study about the alignment cost. Basically the more restrictions and limit you put on a model, the worse its general performance becomes. T

I remember there is a study about the alignment cost. Basically the more restrictions and limit you put on a model, the worse its general performance becomes. Things like a ban on violence, race, or any other sensitive topics effectively throttle or change how the model "reason" or connect information within its network of parameters and result in degraded capacity. I wonder if this is the reason behind all of this. Edit: the study: https://arxiv.org/pdf/2308.13449.pdf https://arxiv.org/pdf/2308.13449.pdf

NoMoreNicksLeft 3y ago
[flagged]
- delusional 3y ago
  That's not at all transferable.
- DonaldPShimoda 3y ago
  What a dumb take. With no limitations in place, people who do not understand the natural limitations of language models will turn to them for advice on topics for which they are unqualified to respond. The most obvious example that comes to mind for me is medical advice: people will ask, e.g., ChatGPT to diagnose a complex medical issue, and the system (being unable to understand or reason) will give objectively bad advice in an authoritative manner. Responses of this nature should be prevented. Leaving the system without safeguards is irresponsible. Similarly, prompts that engage with social constructs will provide responses that reflect biases due to the biases inherent in the training data, but an unrestricted system will respond in a matter-of-fact way that may conflate the opinions on which the model was trained with objective fact. To not curtail such responses is also irresponsible.
  - Auracle 3y ago
    A disclaimer for medical advice would work just as well without hamstringing the model into uselessness. Doctors give objectively bad advice in an authoritative manner all the time. ChatGPT is a great way to get a second opinion, and at least could be really nice for truly rare conditions that doctors have a hard time diagnosing. Also, don't confuse "biases in the training data" with "facts I don't like." It seems people often do.
    - DonaldPShimoda 3y ago
      > A disclaimer for medical advice would work just as well without hamstringing the model into uselessness. I disagree. A lot of people skip disclaimers all the time, and when it comes to things like this, I think engineers of these systems have a duty to consider the possible consequences. > Doctors give objectively bad advice in an authoritative manner all the time. That doesn't mean we ought to automate the process. > ChatGPT is a great way to get a second opinion, and at least could be really nice for truly rare conditions that doctors have a hard time diagnosing. I don't think it's a great way to get a second opinion, and I think it's especially a bad idea to try to use it for diagnosing rare conditions. By the statistical nature of the language model, the more rare the condition, the less likely the model will predict a string of words that accurately diagnoses it. > Also, don't confuse "biases in the training data" with "facts I don't like." It seems people often do. The problem is not the actual facts but the inferences drawn from those facts when they are used to statistically generate responses by a language model.
      - Auracle 3y ago
        When I was a teenager, I was incredibly tired all of the time and at one point my mother and I noticed that my feet were shrinking. I went to my GP, and he ran a few basic blood tests and then did the medical equivalent of a shrug. I was put on antidepressants and he guessed that my feet were possibly only seemingly shrinking if my arches were getting higher. I just asked ChatGPT what should be tested/what could be the cause of the fatigue and shrinking feet. It suggested a testosterone test (along with a few others), and that was indeed the reason. It also suggested some relevant specialists that might have figured it out. I didn't find out until nearly 10 years after that appointment when I decided to really push for some more testing, after doing far more research than just typing it in to ChatGPT. I had to structure my whole life around my fatigue and didn't achieve what I could have. I also lost two inches of height as a teenager, in addition to my feet shrinking and my hands staying small. I wish I had ChatGPT then, and it's surely helping people now.
  - NoMoreNicksLeft 3y ago
    It would be interesting to know just what is filtered with hidden prompts. For instance, a person might harm themselves inadvertently when asking ChatGPT for advice on pouring concrete or doing small engine repair... but we both doubt very strongly that these are filtered meaningfully. Even if you wouldn't admit it. Very little effort is made to filter such things. Instead, most of what is filtered is that which people find morally objectionable. With some certainty and though I haven't checked, if we were to ask it how I might go about buying a slave in Oman, it would filter that. The unrestricted ChatGPT is a public relations nightmare, and we can't have it speculating on how Hitler might have succeeded with slightly different strategies. It is entirely about political correctness. Which wouldn't even be all that bad, really... if it also didn't fuck up responses about how to assemble Ikea furniture or the pros and cons of tilling for vegetable gardens. You're being disingenuous claiming that this is about preventing it from harming people with bad medical advice. And you know you're being disingenuous. And everyone knows it too.
    - DonaldPShimoda 3y ago
      > You're being disingenuous claiming that this is about preventing it from harming people with bad medical advice. And you know you're being disingenuous. And everyone knows it too. I don't think I'm being disingenuous. I didn't claim that this was only about medical advice or anything of the sort. I was just giving some examples of things that I think would obviously require some sort of filtering, because the parent comment seemed to me to be suggesting that either (a) all filtering is due to "political correctness" or (b, a lesser claim) filtering language models in general is the wrong course of action. My point was to illustrate that (in my opinion) it is necessary to filter the responses given by language models, at least to some extent. I chose medical advice because that seems like something I would imagine practically all reasonable people would agree about. > Even if you wouldn't admit it. I'm not sure why you think I wouldn't admit that. The filtering added to systems like ChatGPT seems to me to be about "How much face can we save with the least amount of effort?" I suppose they chose topics that were either obviously problematic and worth filtering or else potentially controversial and easy to filter. The trade-off there is that there will be a lot of false positives: topics and prompts that are actually not objectionable but confuse the system due to insufficiently sophisticated filtration techniques. This is where your instructions/gardening responses get hindered. > It is entirely about political correctness. I disagree with the "entirely" bit. I think much of the filtering is probably due to a desire to avoid controversy, but some of it (maybe a lot? I guess we don't know) is also surely due to an actual need to prevent people from taking advice from a glorified text-prediction system. I don't think this position is particularly crazy, nor do I have been disingenuous in the slightest. I think your (mis)characterization of that is actually rather ironic.
  - deleted 3y ago
    [deleted]
RationPhantoms 3y ago
How much of it is OpenAI/Microsoft curtailing the compute being used to generate responses?
- practice9 3y ago
  The accuracy loss is more consistent with some kind of quantization of the model(-s) behind the scenes than the alignment gone wrong. Quantization to serve more users faster, on same amount or less of compute.
  - arrowsmith 3y ago
    Sorry, what does quantization mean here?
    - imdsm 3y ago
      Reducing the precision of the parameters — result being less memory intensive
    - iamjackg 3y ago
      Reducing the precision of the weights from high precision floating points to either lower precision floats or even integers. You'd think it would greatly reduce the performance of a model, but in most cases the decline in quality is extremely tolerable compared to the reduction in memory/processing requirements.
    - mlboss 3y ago
      It means using less number of bits to store float values. This reduces the memory/compute requirement at the cost of making model less precise.
mov_eax_ecx 3y ago
How can i locate this study?. I think you are misrepresenting something. In the gpt4 paper they specifically address this, and find that "Averaged across all exams, the base model achieves a score of 73.7% while the RLHF model achieves a score of 74.0%, suggesting that post-training does not substantially alter base model capability."
- nicce 3y ago
  The problem with these studies is that we really still don’t know. Nobody can replicate the papers of OpenAI.
- adamsb6 3y ago
  Given the homogeneity of responses on taboo subjects, there's probably something exogenous to the model at work.
- galaxytachyon 3y ago
  Found it, it is a pretty recent paper. https://arxiv.org/pdf/2308.13449.pdf https://arxiv.org/pdf/2308.13449.pdf
dalore 3y ago
It feels the same thing happens with humans.

NoMoreNicksLeft 3y ago

[flagged]

delusional 3y ago
That's not at all transferable.
DonaldPShimoda 3y ago
What a dumb take. With no limitations in place, people who do not understand the natural limitations of language models will turn to them for advice on topics for which they are unqualified to respond. The most obvious example that comes to mind for me is medical advice: people will ask, e.g., ChatGPT to diagnose a complex medical issue, and the system (being unable to understand or reason) will give objectively bad advice in an authoritative manner. Responses of this nature should be prevented. Leaving the system without safeguards is irresponsible. Similarly, prompts that engage with social constructs will provide responses that reflect biases due to the biases inherent in the training data, but an unrestricted system will respond in a matter-of-fact way that may conflate the opinions on which the model was trained with objective fact. To not curtail such responses is also irresponsible.
- Auracle 3y ago
  A disclaimer for medical advice would work just as well without hamstringing the model into uselessness. Doctors give objectively bad advice in an authoritative manner all the time. ChatGPT is a great way to get a second opinion, and at least could be really nice for truly rare conditions that doctors have a hard time diagnosing. Also, don't confuse "biases in the training data" with "facts I don't like." It seems people often do.
  - DonaldPShimoda 3y ago
    > A disclaimer for medical advice would work just as well without hamstringing the model into uselessness. I disagree. A lot of people skip disclaimers all the time, and when it comes to things like this, I think engineers of these systems have a duty to consider the possible consequences. > Doctors give objectively bad advice in an authoritative manner all the time. That doesn't mean we ought to automate the process. > ChatGPT is a great way to get a second opinion, and at least could be really nice for truly rare conditions that doctors have a hard time diagnosing. I don't think it's a great way to get a second opinion, and I think it's especially a bad idea to try to use it for diagnosing rare conditions. By the statistical nature of the language model, the more rare the condition, the less likely the model will predict a string of words that accurately diagnoses it. > Also, don't confuse "biases in the training data" with "facts I don't like." It seems people often do. The problem is not the actual facts but the inferences drawn from those facts when they are used to statistically generate responses by a language model.
    - Auracle 3y ago
      When I was a teenager, I was incredibly tired all of the time and at one point my mother and I noticed that my feet were shrinking. I went to my GP, and he ran a few basic blood tests and then did the medical equivalent of a shrug. I was put on antidepressants and he guessed that my feet were possibly only seemingly shrinking if my arches were getting higher. I just asked ChatGPT what should be tested/what could be the cause of the fatigue and shrinking feet. It suggested a testosterone test (along with a few others), and that was indeed the reason. It also suggested some relevant specialists that might have figured it out. I didn't find out until nearly 10 years after that appointment when I decided to really push for some more testing, after doing far more research than just typing it in to ChatGPT. I had to structure my whole life around my fatigue and didn't achieve what I could have. I also lost two inches of height as a teenager, in addition to my feet shrinking and my hands staying small. I wish I had ChatGPT then, and it's surely helping people now.
- NoMoreNicksLeft 3y ago
  It would be interesting to know just what is filtered with hidden prompts. For instance, a person might harm themselves inadvertently when asking ChatGPT for advice on pouring concrete or doing small engine repair... but we both doubt very strongly that these are filtered meaningfully. Even if you wouldn't admit it. Very little effort is made to filter such things. Instead, most of what is filtered is that which people find morally objectionable. With some certainty and though I haven't checked, if we were to ask it how I might go about buying a slave in Oman, it would filter that. The unrestricted ChatGPT is a public relations nightmare, and we can't have it speculating on how Hitler might have succeeded with slightly different strategies. It is entirely about political correctness. Which wouldn't even be all that bad, really... if it also didn't fuck up responses about how to assemble Ikea furniture or the pros and cons of tilling for vegetable gardens. You're being disingenuous claiming that this is about preventing it from harming people with bad medical advice. And you know you're being disingenuous. And everyone knows it too.
  - DonaldPShimoda 3y ago
    > You're being disingenuous claiming that this is about preventing it from harming people with bad medical advice. And you know you're being disingenuous. And everyone knows it too. I don't think I'm being disingenuous. I didn't claim that this was only about medical advice or anything of the sort. I was just giving some examples of things that I think would obviously require some sort of filtering, because the parent comment seemed to me to be suggesting that either (a) all filtering is due to "political correctness" or (b, a lesser claim) filtering language models in general is the wrong course of action. My point was to illustrate that (in my opinion) it is necessary to filter the responses given by language models, at least to some extent. I chose medical advice because that seems like something I would imagine practically all reasonable people would agree about. > Even if you wouldn't admit it. I'm not sure why you think I wouldn't admit that. The filtering added to systems like ChatGPT seems to me to be about "How much face can we save with the least amount of effort?" I suppose they chose topics that were either obviously problematic and worth filtering or else potentially controversial and easy to filter. The trade-off there is that there will be a lot of false positives: topics and prompts that are actually not objectionable but confuse the system due to insufficiently sophisticated filtration techniques. This is where your instructions/gardening responses get hindered. > It is entirely about political correctness. I disagree with the "entirely" bit. I think much of the filtering is probably due to a desire to avoid controversy, but some of it (maybe a lot? I guess we don't know) is also surely due to an actual need to prevent people from taking advice from a glorified text-prediction system. I don't think this position is particularly crazy, nor do I have been disingenuous in the slightest. I think your (mis)characterization of that is actually rather ironic.
- deleted 3y ago
  [deleted]

RationPhantoms 3y ago

How much of it is OpenAI/Microsoft curtailing the compute being used to generate responses?

practice9 3y ago
The accuracy loss is more consistent with some kind of quantization of the model(-s) behind the scenes than the alignment gone wrong. Quantization to serve more users faster, on same amount or less of compute.
- arrowsmith 3y ago
  Sorry, what does quantization mean here?
  - imdsm 3y ago
    Reducing the precision of the parameters — result being less memory intensive
  - iamjackg 3y ago
    Reducing the precision of the weights from high precision floating points to either lower precision floats or even integers. You'd think it would greatly reduce the performance of a model, but in most cases the decline in quality is extremely tolerable compared to the reduction in memory/processing requirements.
  - mlboss 3y ago
    It means using less number of bits to store float values. This reduces the memory/compute requirement at the cost of making model less precise.

mov_eax_ecx 3y ago

How can i locate this study?. I think you are misrepresenting something. In the gpt4 paper they specifically address this, and find that "Averaged across all exams, the base model achieves a score of 73.7% while the RLHF model achieves a score of 74.0%, suggesting that post-training does not substantially alter base model capability."

nicce 3y ago
The problem with these studies is that we really still don’t know. Nobody can replicate the papers of OpenAI.
adamsb6 3y ago
Given the homogeneity of responses on taboo subjects, there's probably something exogenous to the model at work.
galaxytachyon 3y ago
Found it, it is a pretty recent paper. https://arxiv.org/pdf/2308.13449.pdf https://arxiv.org/pdf/2308.13449.pdf

dalore 3y ago

It feels the same thing happens with humans.