3 ms·

Can someone explain to a layperson why these rules need to be fed into the model as an English-language "prefix prompt" instead of being "encoded" into the mode

Can someone explain to a layperson why these rules need to be fed into the model as an English-language "prefix prompt" instead of being "encoded" into the model at compile-time?

swyx 2y ago
1) easier to modify prompts than retrain a model 2) we simply figured out prompting first - In Context Learning is about 3-4 years old at this point, whereas we are only just beginning to figure out LoRAs and representation engineering, which could encode this behavior much more succinctly but can have tradeoffs in terms of amount of information encoded (you are basically making a preemptive call on what to attend to instead of letting the "full attention" just run as designed
alfonsodev 2y ago
If I’m not mistaken the “concepts” that the rules refer to, are not present in the source code to be compiled, they emerge when training the program and are solidified in a model (black box).
politelemon 2y ago
It's basically to allow reuse in different 'settings'. The same model can be used for many different purposes in different settings. The company could take the model and put it into let's say something like Github Copilot, so in that case the rules would tell it to behave in a different way, be technical oriented, don't engage in chit-chat, and give it a different set of tools. Then it might create a different tool aimed at children in schools as a helper... and in those cases it may tell it to avoid lots of topics, avoid certain kinds of questions, and give it a completely different set of tools.
jncfhnb 2y ago
Other answers here are mostly wrong. They are almost certainly NOT passing in the prompt as English and having the model reread the prompt again each time. What they’ll be doing is passing in the vector representation of the prompt at run time, not “compile time”. The thing is that passing in the vector embedding representation of the prefix prompt leaves the LLM in the same position as if it had read the prompt in English. The embedding IS the prompt. So you can’t tell whether it literally reran the computation to create the vector each time. But it would be much cheaper to not do the same work over and over and over. Passing in the prompt as an embedding of English language is more or less free and is very easy to change on the fly (just type a new prompt and save the vector representation). Fine tuning the model to act as if that prompt was always a given is possible but expensive and slow and not really necessary. You don’t want to retrain a model to not use seaborn if you can just say “don’t use seaborn”
nprateem 2y ago
The same reason people separate config from code. To make it easier to reconfigure things.
GrantMoyer 2y ago
They do both. Broadly, Large Language Models (LLMs) are initially trained on a massive amount of unfiltered text. Removing unpleasant content from the initial training corpus is intractible due to its sheer size. These models can produce pretty unpleasant output, because of the unpleasant messages present in the training data. Accordingly, LLM models are then trained further using Reinforcement Learning from Human Feedback (RLHF). This training phase uses a much smaller corpus of hand picked examples demonstrating desired output. This higher quality corpus is too small to train a high quality LLM from scratch, so it can only be used for fine tuning. Effectively, it "bakes in" to the model the desired form of the output, but it's not perfect, because most of the training occured before this phase. Therefore instruction inserted at the beggining of every prompt or session are used to further increases the chance of the model producing desireable output.

Can someone explain to a layperson why these rules need to be fed into the model as an English-language "prefix prompt" instead of being "encoded" into the model at compile-time?

swyx 2y ago

1) easier to modify prompts than retrain a model 2) we simply figured out prompting first - In Context Learning is about 3-4 years old at this point, whereas we are only just beginning to figure out LoRAs and representation engineering, which could encode this behavior much more succinctly but can have tradeoffs in terms of amount of information encoded (you are basically making a preemptive call on what to attend to instead of letting the "full attention" just run as designed

alfonsodev 2y ago

If I’m not mistaken the “concepts” that the rules refer to, are not present in the source code to be compiled, they emerge when training the program and are solidified in a model (black box).

politelemon 2y ago

It's basically to allow reuse in different 'settings'. The same model can be used for many different purposes in different settings. The company could take the model and put it into let's say something like Github Copilot, so in that case the rules would tell it to behave in a different way, be technical oriented, don't engage in chit-chat, and give it a different set of tools. Then it might create a different tool aimed at children in schools as a helper... and in those cases it may tell it to avoid lots of topics, avoid certain kinds of questions, and give it a completely different set of tools.

jncfhnb 2y ago

Other answers here are mostly wrong. They are almost certainly NOT passing in the prompt as English and having the model reread the prompt again each time. What they’ll be doing is passing in the vector representation of the prompt at run time, not “compile time”. The thing is that passing in the vector embedding representation of the prefix prompt leaves the LLM in the same position as if it had read the prompt in English. The embedding IS the prompt. So you can’t tell whether it literally reran the computation to create the vector each time. But it would be much cheaper to not do the same work over and over and over. Passing in the prompt as an embedding of English language is more or less free and is very easy to change on the fly (just type a new prompt and save the vector representation). Fine tuning the model to act as if that prompt was always a given is possible but expensive and slow and not really necessary. You don’t want to retrain a model to not use seaborn if you can just say “don’t use seaborn”

nprateem 2y ago

The same reason people separate config from code. To make it easier to reconfigure things.

GrantMoyer 2y ago

They do both. Broadly, Large Language Models (LLMs) are initially trained on a massive amount of unfiltered text. Removing unpleasant content from the initial training corpus is intractible due to its sheer size. These models can produce pretty unpleasant output, because of the unpleasant messages present in the training data. Accordingly, LLM models are then trained further using Reinforcement Learning from Human Feedback (RLHF). This training phase uses a much smaller corpus of hand picked examples demonstrating desired output. This higher quality corpus is too small to train a high quality LLM from scratch, so it can only be used for fine tuning. Effectively, it "bakes in" to the model the desired form of the output, but it's not perfect, because most of the training occured before this phase. Therefore instruction inserted at the beggining of every prompt or session are used to further increases the chance of the model producing desireable output.