3 ms·

The models weights are tuned towards the direction that would cause the model to best fit the training set. It turns out that this process makes it useful at p

The models weights are tuned towards the direction that would cause the model to best fit the training set. It turns out that this process makes it useful at producing mostly sensible predictions (generate output) for text that is not present in the training set (generalization). The reason that works is because there are a lot of patterns and redundancy in the stuff that we feed to the models and the stuff that we ask the models so there is a good chance that interpolating between words and higher level semantics relationship between sentences will make sense quite often. However that doesn't work all the time. And when it doesn't, current models have no way to tell they "don't know". The whole point was to let them generalize beyond the training set and interpolate in order to make decent guesses. There is a lot of research in making models actually reason.

magicalhippo 2y ago
In the Physics of Language Models talk[1], he argues that the model knows it has made a mistake, sometimes even before it has made it. Though apparently training is crucial to make the model be able to use this constructively. That being said, I'm aware that the model doesn't reason in the classical sense. Yet, as I mentioned, it does give me less confabulation when I tell it it's ok not to answer. I will note that when I've tried the same kind of prompts with Phi 3 instruct, it's way worse than Gemma. Though I'm not sure if that's just because of a weak instruction tuning or the underlying training as well, as it frequently ignores parts of my instructions. [1]: https://www.youtube.com/watch?v=yBL7J0kgldU https://www.youtube.com/watch?v=yBL7J0kgldU
- ithkuil 2y ago
  There are different ways to be wrong. For example you can confabulate "facts" or you can make logical or coherence mistakes. Current LLMs are encouraged to be creative and effectively "make up facts". That's what created the first wow factor. The models are able to write a Star Trek fan fiction model in the style of Shakespeare. They are able to take a poorly written email and make it "sound" better (for some definition of better, e.g. more formal, less formal etc). But then, human psychology kicked in and as soon as you have something that can talk like a human and some marketing folks label as "AI" you start expecting it to be useful also for other tasks, some of which require factual knowledge. Now, it's in theory possible to have a system that you can converse with which can _also_ search and verify knowledge. My point is that this is not the place where LLMs start from. You have to add stuff on top of them (and people are actively researching that)

magicalhippo 2y ago

In the Physics of Language Models talk[1], he argues that the model knows it has made a mistake, sometimes even before it has made it. Though apparently training is crucial to make the model be able to use this constructively. That being said, I'm aware that the model doesn't reason in the classical sense. Yet, as I mentioned, it does give me less confabulation when I tell it it's ok not to answer. I will note that when I've tried the same kind of prompts with Phi 3 instruct, it's way worse than Gemma. Though I'm not sure if that's just because of a weak instruction tuning or the underlying training as well, as it frequently ignores parts of my instructions. [1]: https://www.youtube.com/watch?v=yBL7J0kgldU https://www.youtube.com/watch?v=yBL7J0kgldU

ithkuil 2y ago
There are different ways to be wrong. For example you can confabulate "facts" or you can make logical or coherence mistakes. Current LLMs are encouraged to be creative and effectively "make up facts". That's what created the first wow factor. The models are able to write a Star Trek fan fiction model in the style of Shakespeare. They are able to take a poorly written email and make it "sound" better (for some definition of better, e.g. more formal, less formal etc). But then, human psychology kicked in and as soon as you have something that can talk like a human and some marketing folks label as "AI" you start expecting it to be useful also for other tasks, some of which require factual knowledge. Now, it's in theory possible to have a system that you can converse with which can _also_ search and verify knowledge. My point is that this is not the place where LLMs start from. You have to add stuff on top of them (and people are actively researching that)