4 ms·

It is my understanding there is no way to objectively or programmatically tell if any of this stuff is correct or doesn't obfuscate some dire error. These trick

by th0ma5 3y ago

It is my understanding there is no way to objectively or programmatically tell if any of this stuff is correct or doesn't obfuscate some dire error. These tricks don't give me confidence that we're headed in that direction either.

nerdponx 3y ago
Any LLM/GPT system is fundamentally stochastic, so yes, kind of? But it's hardly a "trick" to come up with a way to make them work the way we want them to work. An LLM is based on the idea that there is some very complicated function y(t) = f(y(t-1), y(t-2), ..., y(t-K)) for any sequence of text tokens t(1), y(2), ... and some huge context size K. We fit the model by attempting to find an approximation of f() with a low badness score (called "loss"). We also don't usually want the absolute lowest badness score, because there is always some tradeoff between between minimizing loss on the specific test data that we happen to have, and preserving the ability to generalize to content outside of the test data. The technique here is an improvement to that process of finding a good approximation of f(). The specific technique here is to adjust the loss to mean something more specific than "error rate on predicting the next word in a sequence." The entire basis of the principle is that the loss function controls what the model learns. If we want it to produce more sentences that look a certain way, we introduce a reward for generating sentences that look that way. If we want to avoid certain kinds of outputs, we introduce a penalty for generating those kinds of outputs. From there, it's just gradient descent. Better outputs produce lower losses, so the model starts producing better outputs, without us having to really know or care about what the model is doing internally. The technique of RLHF is similar along those lines. We discourage the model to hallucinate by having a human review its output and report when the model hallucinates, so we can impose a penalty for hallucination, and thereby shift the model output in the direction of not-hallucinating after many such rounds of "learning". Is it a fundamental shift in how LLMs work? No. Could it possibly lead to improvements? Yes.
- th0ma5 3y ago
  I mean, doesn't that sweep the problem I'm describing into the loss function definition?
  - nerdponx 3y ago
    Sure. It might be impossible to actually set up a combination of model, training data, loss function / reward procedure, etc. that produces a perfect result. But who ever said it would be perfect?
- catlifeonmars 3y ago
  Doesn’t that imply that there is some topological structure to the hallucinations? What does that topology look like? How do you know that there are no discontinuous regions or that you are not stuck in a local minima?
  - nerdponx 3y ago
    > Doesn’t that imply that there is some topological structure to the hallucinations? What does that topology look like? Maybe! I'm not an AI researcher or mathematician, so I don't know if anyone has pursued this idea. The problem might be that any such structure is intractably complicated to describe within the limits of human understanding. > How do you know that there are no discontinuous regions or that you are not stuck in a local minima? Are you talking about f() getting stuck in some kind of bad region while generating text, or about the optimization process itself? The answer is the same in both cases: we don't. Regarding text generation, we've seen plenty of examples where specific prompts result in pathological output. Although I'm not sure if newer models have that problem. Regarding optimization, there is absolutely no guarantee that we have found a global minimum. Some interesting research has been done on the "loss landscape" of these giant neural network models, and my understanding is that they are messy and complicated. Keep in mind that the training data is part of the loss function! Finding a local minimum on the training data might just result in overfitting.
    - catlifeonmars 3y ago
      > Some interesting research has been done on the "loss landscape" of these giant neural network models, and my understanding is that they are messy and complicated. Do you have any recommended reading for this? It sounds like a super interesting area of research.
      - nerdponx 3y ago
        This is the one example I had in mind because of all the pretty pictures: https://arxiv.org/abs/1712.09913 https://arxiv.org/abs/1712.09913 Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. 2018. "Visualizing the Loss Landscape of Neural Nets".
orbital-decay 3y ago
Yes, in the end it just makes it produce better answers most of the time, it doesn't eliminate hallucinations. Not sure how you expect it to objectively tell the correctness, short of formulating the prompt as axioms and performing formal verification. Which is probably not what you want from it.

by th0ma5 3y ago

nerdponx 3y ago

Any LLM/GPT system is fundamentally stochastic, so yes, kind of? But it's hardly a "trick" to come up with a way to make them work the way we want them to work. An LLM is based on the idea that there is some very complicated function y(t) = f(y(t-1), y(t-2), ..., y(t-K)) for any sequence of text tokens t(1), y(2), ... and some huge context size K. We fit the model by attempting to find an approximation of f() with a low badness score (called "loss"). We also don't usually want the absolute lowest badness score, because there is always some tradeoff between between minimizing loss on the specific test data that we happen to have, and preserving the ability to generalize to content outside of the test data. The technique here is an improvement to that process of finding a good approximation of f(). The specific technique here is to adjust the loss to mean something more specific than "error rate on predicting the next word in a sequence." The entire basis of the principle is that the loss function controls what the model learns. If we want it to produce more sentences that look a certain way, we introduce a reward for generating sentences that look that way. If we want to avoid certain kinds of outputs, we introduce a penalty for generating those kinds of outputs. From there, it's just gradient descent. Better outputs produce lower losses, so the model starts producing better outputs, without us having to really know or care about what the model is doing internally. The technique of RLHF is similar along those lines. We discourage the model to hallucinate by having a human review its output and report when the model hallucinates, so we can impose a penalty for hallucination, and thereby shift the model output in the direction of not-hallucinating after many such rounds of "learning". Is it a fundamental shift in how LLMs work? No. Could it possibly lead to improvements? Yes.

th0ma5 3y ago
I mean, doesn't that sweep the problem I'm describing into the loss function definition?
- nerdponx 3y ago
  Sure. It might be impossible to actually set up a combination of model, training data, loss function / reward procedure, etc. that produces a perfect result. But who ever said it would be perfect?
catlifeonmars 3y ago
Doesn’t that imply that there is some topological structure to the hallucinations? What does that topology look like? How do you know that there are no discontinuous regions or that you are not stuck in a local minima?
- nerdponx 3y ago
  > Doesn’t that imply that there is some topological structure to the hallucinations? What does that topology look like? Maybe! I'm not an AI researcher or mathematician, so I don't know if anyone has pursued this idea. The problem might be that any such structure is intractably complicated to describe within the limits of human understanding. > How do you know that there are no discontinuous regions or that you are not stuck in a local minima? Are you talking about f() getting stuck in some kind of bad region while generating text, or about the optimization process itself? The answer is the same in both cases: we don't. Regarding text generation, we've seen plenty of examples where specific prompts result in pathological output. Although I'm not sure if newer models have that problem. Regarding optimization, there is absolutely no guarantee that we have found a global minimum. Some interesting research has been done on the "loss landscape" of these giant neural network models, and my understanding is that they are messy and complicated. Keep in mind that the training data is part of the loss function! Finding a local minimum on the training data might just result in overfitting.
  - catlifeonmars 3y ago
    > Some interesting research has been done on the "loss landscape" of these giant neural network models, and my understanding is that they are messy and complicated. Do you have any recommended reading for this? It sounds like a super interesting area of research.
    - nerdponx 3y ago
      This is the one example I had in mind because of all the pretty pictures: https://arxiv.org/abs/1712.09913 https://arxiv.org/abs/1712.09913 Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. 2018. "Visualizing the Loss Landscape of Neural Nets".

orbital-decay 3y ago

Yes, in the end it just makes it produce better answers most of the time, it doesn't eliminate hallucinations. Not sure how you expect it to objectively tell the correctness, short of formulating the prompt as axioms and performing formal verification. Which is probably not what you want from it.