4 ms·
Typically the output of an LLM is a classifier that gives scores reflecting the "best" next token, for some measure of best. You can deterministically just take
by andy99 3y ago
Typically the output of an LLM is a classifier that gives scores reflecting the "best" next token, for some measure of best. You can deterministically just take the highest score or you can randomly sample in a way that makes it more likely to get a token with a higher score, according to different schemes. The point is that this all rests in the performance of the output classifier layer.
From my skim, this proposes to inject randomness throughout the model (as opposed to just during sampling the classifier output) which according to the author (I'm paraphrasing) lets the full power of the model be used to make the most of that randomness, instead of just jittering the output. So the output is still random but it's a function of random noise being added through the model layers instead of just at the output, supposedly demonstrating better properties. I have no opinion about it as I haven't studied it.
- madamelic 3y ago> You can deterministically just take the highest score or you can randomly sample in a way that makes it more likely to get a token with a higher score, according to different schemes. Ah ok, so it's like the difference between greedily always taking the shortest / highest value vs using A* and exploring alternatives in hopes of finding something better? That probably doesn't line up as a metaphor as it isn't "exploring" or running multiple parallel generations. And it employs the method in the layers so that the benefit can be amplified rather than only employing the method in the final layer / output when it selects the output to take
- doctorpangloss 3y agoThe difference is between something that makes sense (the way the model is intended to be used) and something that makes almost no sense (this repository). > benefit In an intellectually honest way, it is not conclusive if there is a benefit to the approach in this repository. However, as a betting man: there is no benefit. There are a lot of surprises! Riffusion and LCMs for me were super surprising. With no examples or meaningful investigation, I don’t think this is going to yield anything.
- MacsHeadroom 3y agoIt's funny that you think the way transformer models are "intended" to be used is the result of intentional engineering and not an accident that ML academics are no closer to giving some causal explanation for why it works today than they were when Attention is All You Need was published.