8 ms·

I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what

I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more beneficial if computationally possible (and so on for 3, 4 and n words). Does this exist? (I realize that choosing the most likely word wouldn't necessarily solve the issue, but choosing the most likely phrase possibly might.) Edit, post seeing the video and comments: it's beam search, along with temperature to control these things.

mvsin 3y ago
Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search:https://www.width.ai/post/what-is-beam-search https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
- user_7832 3y ago
  Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
  - yunohn 3y ago
    This is actually a great question for which I found an interesting attempt: https://andys.page/posts/llm_sampling_strategies/ https://andys.page/posts/llm_sampling_strategies/ (No affiliation)
  - activatedgeek 3y ago
    If you use HuggingFace models, then a few simpler decoding algorithms are already implemented for `generate` method of all supported models. Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate. I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem. Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.
- qeternity 3y ago
  > production systems rarely use greedy search I have no idea why you say this. Most of our pipelines will run greedy, for reproducibility. Maybe we turn the temp up if we are returning conversational text back to a user.
doctoboggan 3y ago
The temperature setting is used to select how rare of a next token is possible. If set to 0 the. The top of the likely list is chosen, if set greater than 0 then some lower probability tokens may be chosen.
- user_7832 3y ago
  Thanks, learnt something new today!
- not_a_dane 3y ago
  It is the part of softmax layer, but not all the time.
- davekeck 3y ago
  > then some lower probability tokens may be chosen Can you explain how it chooses one of the lower-probability tokens? Is it just random?
  - acchow 3y ago
    Reducing temperature reduces the impact of differences between raw output values giving a higher probability to pick other tokens.
    - acchow 3y ago
      Oops backwards. Increasing temperature...
- DrawTR 3y ago
  Can this be potentially dangerous -- e.g. if a user types "The answer to the expression 2 + 2 is", isn't there a chance it chooses an output beyond the most likely one?
  - jldugger 3y ago
    Yes, but the chance is quite small if the gap between "4" and any other token is quite large.
deleted 3y ago
[deleted]
lxe 3y ago
There's a whole bunch of different normalization and sampling techniques that you can perform that can alter the quality or expressiveness of the model, e.g. https://docs.sillytavern.app/usage/common-settings/#sampler-parameters https://docs.sillytavern.app/usage/common-settings/#sampler-...
chessgecko 3y ago
There’s some fancier stuff too like techniques that take into account where recent tokens were drawn from in the distribution and update either the top_p or the temperature so that sequences of tokens have a minimum unlikeliness. Beam search is less common with really large models because the computation is really expensive.
ahzhou 3y ago
Yes, this is a fundamental weakness with LLMs. Unfortunately this is likely unsolvable because the search space is exponential. Techniques like beam search help, but can only introduce a constant scaling factor. That said, LLM reach their current performance despite this limitation.
authorfly 3y ago
In practice, beam search doesn't seem to work well for generative models. Temperature and top_k (two very similar parameters) were both introduced to account for the fact that human text is unpredictable stochastically for each sentence someone might say as such - as shown in this 2021 similar graph/reproduction of an older graph from the 2018/2019 HF documentation: https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam_search_less_surprising.png https://lilianweng.github.io/posts/2021-01-02-controllable-t... It could be that beam search with much longer length does turn out to be better or some merging of the techniques works well, but I don't think so. The query-key-value part of transformers is focused on a single total in many ways - in relation to the overall context. The architecture is not meant for longer forms as such - there is no default "two token" system. And with 50k-100k tokens in most GPT models, you would be looking at 50k*50k = A great deal more parameters and then issues with sparsity of data. Just everything about GPT models (e.g. learned positional encodings/embeddings depending on the model iteration) is so focused on bringing the richness of a single token or single token index that the architecture is not designed for beam search like this one could say. Without considering the training complications.
jimmySixDOF 3y ago
>words together Thats basically chunking or at least how it starts. I was impressed by the ability to add and subtract the individual word vector embeddings and get meaningful results. Chunking a larger block blends this whole process so you can do the same thing but in conseptual space the so take a baseline method like sentence embedding and that becomes your working block for comparison.

mvsin 3y ago

Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search:https://www.width.ai/post/what-is-beam-search https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.

user_7832 3y ago
Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
- yunohn 3y ago
  This is actually a great question for which I found an interesting attempt: https://andys.page/posts/llm_sampling_strategies/ https://andys.page/posts/llm_sampling_strategies/ (No affiliation)
- activatedgeek 3y ago
  If you use HuggingFace models, then a few simpler decoding algorithms are already implemented for `generate` method of all supported models. Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate. I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem. Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.
qeternity 3y ago
> production systems rarely use greedy search I have no idea why you say this. Most of our pipelines will run greedy, for reproducibility. Maybe we turn the temp up if we are returning conversational text back to a user.

doctoboggan 3y ago

The temperature setting is used to select how rare of a next token is possible. If set to 0 the. The top of the likely list is chosen, if set greater than 0 then some lower probability tokens may be chosen.

user_7832 3y ago
Thanks, learnt something new today!
not_a_dane 3y ago
It is the part of softmax layer, but not all the time.
davekeck 3y ago
> then some lower probability tokens may be chosen Can you explain how it chooses one of the lower-probability tokens? Is it just random?
- acchow 3y ago
  Reducing temperature reduces the impact of differences between raw output values giving a higher probability to pick other tokens.
  - acchow 3y ago
    Oops backwards. Increasing temperature...
DrawTR 3y ago
Can this be potentially dangerous -- e.g. if a user types "The answer to the expression 2 + 2 is", isn't there a chance it chooses an output beyond the most likely one?
- jldugger 3y ago
  Yes, but the chance is quite small if the gap between "4" and any other token is quite large.

deleted 3y ago

[deleted]

lxe 3y ago

There's a whole bunch of different normalization and sampling techniques that you can perform that can alter the quality or expressiveness of the model, e.g. https://docs.sillytavern.app/usage/common-settings/#sampler-parameters https://docs.sillytavern.app/usage/common-settings/#sampler-...

chessgecko 3y ago

There’s some fancier stuff too like techniques that take into account where recent tokens were drawn from in the distribution and update either the top_p or the temperature so that sequences of tokens have a minimum unlikeliness. Beam search is less common with really large models because the computation is really expensive.

ahzhou 3y ago

Yes, this is a fundamental weakness with LLMs. Unfortunately this is likely unsolvable because the search space is exponential. Techniques like beam search help, but can only introduce a constant scaling factor. That said, LLM reach their current performance despite this limitation.

authorfly 3y ago

In practice, beam search doesn't seem to work well for generative models. Temperature and top_k (two very similar parameters) were both introduced to account for the fact that human text is unpredictable stochastically for each sentence someone might say as such - as shown in this 2021 similar graph/reproduction of an older graph from the 2018/2019 HF documentation: https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam_search_less_surprising.png https://lilianweng.github.io/posts/2021-01-02-controllable-t... It could be that beam search with much longer length does turn out to be better or some merging of the techniques works well, but I don't think so. The query-key-value part of transformers is focused on a single total in many ways - in relation to the overall context. The architecture is not meant for longer forms as such - there is no default "two token" system. And with 50k-100k tokens in most GPT models, you would be looking at 50k*50k = A great deal more parameters and then issues with sparsity of data. Just everything about GPT models (e.g. learned positional encodings/embeddings depending on the model iteration) is so focused on bringing the richness of a single token or single token index that the architecture is not designed for beam search like this one could say. Without considering the training complications.

jimmySixDOF 3y ago

>words together Thats basically chunking or at least how it starts. I was impressed by the ability to add and subtract the individual word vector embeddings and get meaningful results. Chunking a larger block blends this whole process so you can do the same thing but in conseptual space the so take a baseline method like sentence embedding and that becomes your working block for comparison.