6 ms·

> Think about the training data. > How often does the training data of most of the Internet reflect users identifying issues with their own output? > How ofte

> Think about the training data. > How often does the training data of most of the Internet reflect users identifying issues with their own output? > How often does the training data reflect users identifying issues with someone else's output? I wouldn't put too much weight into just-so theories like this. We still don't understand too much about how LLMs process information internally; it could be that their understanding of the concept of "correcting a previous mistake" is good enough that they can access it without prompt engineering to mimic the way it happens in training data. Or maybe not (after all, there's an entire management concept called "pre-mortems" which is basically doing what you suggest, as a human).

kromem 3y ago
This depends less on the internals vs the patterns in the training data. Even if the model has the capacity to abstract beyond the patterns, the patterns are still very likely to have influence on its ability to do so. For example, early after GPT-4 was released it was being claimed it couldn't solve variations on the goat, wolf, and cabbage problem. I found that it could solve these variations fine 100% of the time, you just needed to explicitly prompt for it to repeat adjectives with nouns and change the nouns to emojis. The repeating worked similar to CoT by biasing the generation towards the variation and away from the original form, and the emojis in place of the nouns further broke the token associations which was leading it to fail by extending the original solution. So while it's possible that with enough finessing you could get a model to perform self-critique as well as its critique of others, if the training data has a clear pattern of bias between those two, why actively ignore it? It's a bit like sanding against the grain vs with it. You can sand against the grain of the training data and with enough effort potentially get the result you want with sophisticated enough models. But maybe your life will be a lot easier if you identify the grain in the data first and sand along with it instead?
- PoignardAzur 3y ago
  As someone who tried variants of the wolf riddle with the Bing model and didn'tget too far, I'm super interested. Do you have a source on this?
galaxyLogic 3y ago
> We still don't understand too much about how LLMs process information internally I admit I personally don't know too much about how "LLMs process information internally". But, I would find it curious if programmers who created the system wouldn't understand what it is doing. Is there any evidence that the LLM programmers don't understand how the program they created works?
- kromem 3y ago
  LLMs aren't programmed and it's why the neural network working as it does is black box to everyone, developers included. Imagine a billion black boxes with hamsters put in them. You put in a bag of equally mixed Skittles in one end of each box and then rate each box based on how well it does to get rid of the yellow and green Skittles but push out the others. The ones that do the best at this you mate the hamsters and go again, over and over. Eventually you should have hamsters in boxes that almost always get rid of yellow and green Skittles and output the rest. But is it because you bred in a preference to eat those color Skittles? An aversion to the other colors? Are they using those colors for nesting? Do they find the red and blue and orange ones too stimulating so they push those out but leave the others alone? There could be a myriad of reasons why your training was successful, and without the ability to introspect the result you just won't know what's correct. This is a huge simplification by way of loose analogy for what's going on with training a transformer based LLM, but no one is sitting there 'programming' it. They are just setting up the conditions for it to self-optimize around the training goals given the data, and the 'programming' just has to do with improving the efficiency of the training process. Analyzing the final network itself is like trying to understand what each variable in a billion variable math equation is doing to the result.
  - galaxyLogic 3y ago
    When you train an LLM you do that by executing some computer code with some inputs. The programmers who wrote the code you execute know exactly what it does. Just like Google knows exactly how its search-algorithm works. An LLM uses statistics and Markov-chains and what have you to generate the output for a given input. It's like with any optimization algorithm. You cannot predict what exactly will be the result of a given optimization-run. But you know how the optimization algorithm works. The (more or less) optimal solution you get back might surprise you, might be counter-intuitive. But programmers who wrote the code that did the optimization, and have the source-code, know exactly how it works. When you get a result from LLM you don't say "I can't possibly understand why it came up with this result?". You can understand that, it's just following the rules it was programmed to follow. You might not know those rules, you might not understand them, but programmers who wrote them do.
jibal 3y ago
They aren't just-so theories ... this is how LLMs work. We actually understand exactly how they process information internally, but since their very nature is to extract statistical patterns from the training data and that training data is massive, we can't anticipate what patterns have been extracted. We just know that, whatever patterns are there to be abstracted--e.g., users tending to identify issues with someone else's output rather than their own--those patterns will be reflected in the output.

kromem 3y ago

This depends less on the internals vs the patterns in the training data. Even if the model has the capacity to abstract beyond the patterns, the patterns are still very likely to have influence on its ability to do so. For example, early after GPT-4 was released it was being claimed it couldn't solve variations on the goat, wolf, and cabbage problem. I found that it could solve these variations fine 100% of the time, you just needed to explicitly prompt for it to repeat adjectives with nouns and change the nouns to emojis. The repeating worked similar to CoT by biasing the generation towards the variation and away from the original form, and the emojis in place of the nouns further broke the token associations which was leading it to fail by extending the original solution. So while it's possible that with enough finessing you could get a model to perform self-critique as well as its critique of others, if the training data has a clear pattern of bias between those two, why actively ignore it? It's a bit like sanding against the grain vs with it. You can sand against the grain of the training data and with enough effort potentially get the result you want with sophisticated enough models. But maybe your life will be a lot easier if you identify the grain in the data first and sand along with it instead?

PoignardAzur 3y ago
As someone who tried variants of the wolf riddle with the Bing model and didn'tget too far, I'm super interested. Do you have a source on this?

galaxyLogic 3y ago

> We still don't understand too much about how LLMs process information internally I admit I personally don't know too much about how "LLMs process information internally". But, I would find it curious if programmers who created the system wouldn't understand what it is doing. Is there any evidence that the LLM programmers don't understand how the program they created works?

kromem 3y ago
LLMs aren't programmed and it's why the neural network working as it does is black box to everyone, developers included. Imagine a billion black boxes with hamsters put in them. You put in a bag of equally mixed Skittles in one end of each box and then rate each box based on how well it does to get rid of the yellow and green Skittles but push out the others. The ones that do the best at this you mate the hamsters and go again, over and over. Eventually you should have hamsters in boxes that almost always get rid of yellow and green Skittles and output the rest. But is it because you bred in a preference to eat those color Skittles? An aversion to the other colors? Are they using those colors for nesting? Do they find the red and blue and orange ones too stimulating so they push those out but leave the others alone? There could be a myriad of reasons why your training was successful, and without the ability to introspect the result you just won't know what's correct. This is a huge simplification by way of loose analogy for what's going on with training a transformer based LLM, but no one is sitting there 'programming' it. They are just setting up the conditions for it to self-optimize around the training goals given the data, and the 'programming' just has to do with improving the efficiency of the training process. Analyzing the final network itself is like trying to understand what each variable in a billion variable math equation is doing to the result.
- galaxyLogic 3y ago
  When you train an LLM you do that by executing some computer code with some inputs. The programmers who wrote the code you execute know exactly what it does. Just like Google knows exactly how its search-algorithm works. An LLM uses statistics and Markov-chains and what have you to generate the output for a given input. It's like with any optimization algorithm. You cannot predict what exactly will be the result of a given optimization-run. But you know how the optimization algorithm works. The (more or less) optimal solution you get back might surprise you, might be counter-intuitive. But programmers who wrote the code that did the optimization, and have the source-code, know exactly how it works. When you get a result from LLM you don't say "I can't possibly understand why it came up with this result?". You can understand that, it's just following the rules it was programmed to follow. You might not know those rules, you might not understand them, but programmers who wrote them do.

jibal 3y ago

They aren't just-so theories ... this is how LLMs work. We actually understand exactly how they process information internally, but since their very nature is to extract statistical patterns from the training data and that training data is massive, we can't anticipate what patterns have been extracted. We just know that, whatever patterns are there to be abstracted--e.g., users tending to identify issues with someone else's output rather than their own--those patterns will be reflected in the output.