4 ms·

Yeah that's it exactly, and a major thing people run into. Finetuning won't add "knowledge", it's changing the bias likelihood going forward for each tokens ge

Yeah that's it exactly, and a major thing people run into. Finetuning won't add "knowledge", it's changing the bias likelihood going forward for each tokens generation(with context to tokens around it in your training dataset). Fundamentally it's building on top of the "associations" the internal mechanism of the model has learned at different parts of itself - attention/self-attention - during the original training. Finetuning changes things superficially on the outside of the box. It changes things in a way that do not alter the fundamentals, unless you literally abduct weights deliberately (some good posts on lesswrong about this where they remove concepts like fire/water by finding with SVD where they are). If you think of it like a building, training makes the entire building, and finetuning has no access to the ground floor/basement/foundations of the building to change those lower, important parts, except in terms of how the floors are presented as you go upwards. Some things link the lift will always be in the same place[fundamental orderings of words in general], but you can use a different floorplan[change the vocab distribution+etc by finetuning]. Can you try uploading that data in PDF format (or youtube videos) to Fragen.co.uk and let me know what you think? It should reasonably by able to discuss the current season well once you provide enough data (but it will have reduced ability to discuss previous seasons). That's a tool which uses a similar approach to OP but with some mechanics to share and order knowledge smartly according to the question (e.g. replacing "it" with the right nouns, bringing in predicate facts relevant to the question). The answers have checkmarks next to statements it is confident about, and you could reasonably expect performance like base GPT-3 on NFL football season of 2019 if you did this with sufficient data (aka. 1000+ pages/game reports). If you have a youtube video of one game, you should be able to test it quickly with that by asking things mentioned in the games video, and the answers should not be wrong. It will reject questions it can't answer well.

noduerme 4y ago
I hadn't seen Fragen.co.uk before, but it's pretty cool that it can ingest several different kinds of media. Notwithstanding the limitations you described w/r/t adding "knowledge" to the model itself, is there a way to sort of, like, crystalize the state of the model after it's taken a set of tokens? In non-AI terms, if you were hosting the model yourself, could you just upload videos of the entire NFL season and then dump the model out of memory in its current state (with what I assume would be just a bunch of activated weights now, not actually a compressed version of all those videos), and restore it to that point later so you don't have to feed all that data in again? And is that what e.g. custom Stable Diffusion checkpoints are doing, or are they actually further training the model?
- james-revisoai 4y ago
  Yep, you could pull a saved version of the model weights for your specific data and restore it later, which is what a lot of the finetuned models on huggingface do under the hood(and probably what OpenAI does when you finetune on the API). As it turns out yes, if you uploaded the NFL seasons transcripts, the weights which changed the most (negative or positive) would be those associated with the new information in the data. But they will all still be connected with the connections and tokens they were before(importantly, including the unchanged earlier layers), just with different "strengths" - and for the most part, very similar strengths. However these are not commutive operations between different sets of finetuned data. - If you combined the weight differences from two different sets of data, you will not get the same result as if you finetune on both sets of data and then save. So you can't just tack on further data that easily (you could add more of the same but if the data distribution is different over time, you'll want to sample the data randomly when training mostly) Part of this is because models are trained with batches of samples, never the whole set (generally), say, 16 to 256 at a time. Because of this, each "step/batch" then changes the network and the next step is dependent upon the previous. There are alternate approaches that load all samples in memory, where you purely learn a custom matrix projection of the output last hidden layer (e.g. 4096 floats), which can allow you to do downstream tasks but is not useful for altering the pattern of generating text (and therefore not useful for knowledge extraction). This also makes it hard to understand adding knowledge to LLMs, although SVD is helping with this recently for all kinds of models that use attention and decoding (so GPT, BERT, T5 etc) in the NLP domain: https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singul... - https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... The above also cover why early layers matter: In almost all cases, with an unfinetuned GPT, the first token in the earliest 1-3 layers is already a reasonable or ideal token. Since finetuning often does not impact these layers, it is questionable whether the model can have the information available that a non-finetuned model does to the same respect regarding new patterns(concepts etc) in your new data, even if it can "skirt over it" by appropriately adjusting at the last layer. Recently there was also a 2 layer model which performed similarly well to these transformer/GPT models at smaller scale (which span from 8-24 layers by and large) which might open new ways of finetuning. I can't answer about StableDiffusion checkpoints as I don't know diffusion models well, but most finetuning is a set of matrix operations than project weight changes to existing weights in the network (typically - just the end parts before the output of the network).

noduerme 4y ago

I hadn't seen Fragen.co.uk before, but it's pretty cool that it can ingest several different kinds of media. Notwithstanding the limitations you described w/r/t adding "knowledge" to the model itself, is there a way to sort of, like, crystalize the state of the model after it's taken a set of tokens? In non-AI terms, if you were hosting the model yourself, could you just upload videos of the entire NFL season and then dump the model out of memory in its current state (with what I assume would be just a bunch of activated weights now, not actually a compressed version of all those videos), and restore it to that point later so you don't have to feed all that data in again? And is that what e.g. custom Stable Diffusion checkpoints are doing, or are they actually further training the model?

james-revisoai 4y ago
Yep, you could pull a saved version of the model weights for your specific data and restore it later, which is what a lot of the finetuned models on huggingface do under the hood(and probably what OpenAI does when you finetune on the API). As it turns out yes, if you uploaded the NFL seasons transcripts, the weights which changed the most (negative or positive) would be those associated with the new information in the data. But they will all still be connected with the connections and tokens they were before(importantly, including the unchanged earlier layers), just with different "strengths" - and for the most part, very similar strengths. However these are not commutive operations between different sets of finetuned data. - If you combined the weight differences from two different sets of data, you will not get the same result as if you finetune on both sets of data and then save. So you can't just tack on further data that easily (you could add more of the same but if the data distribution is different over time, you'll want to sample the data randomly when training mostly) Part of this is because models are trained with batches of samples, never the whole set (generally), say, 16 to 256 at a time. Because of this, each "step/batch" then changes the network and the next step is dependent upon the previous. There are alternate approaches that load all samples in memory, where you purely learn a custom matrix projection of the output last hidden layer (e.g. 4096 floats), which can allow you to do downstream tasks but is not useful for altering the pattern of generating text (and therefore not useful for knowledge extraction). This also makes it hard to understand adding knowledge to LLMs, although SVD is helping with this recently for all kinds of models that use attention and decoding (so GPT, BERT, T5 etc) in the NLP domain: https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singul... - https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti... The above also cover why early layers matter: In almost all cases, with an unfinetuned GPT, the first token in the earliest 1-3 layers is already a reasonable or ideal token. Since finetuning often does not impact these layers, it is questionable whether the model can have the information available that a non-finetuned model does to the same respect regarding new patterns(concepts etc) in your new data, even if it can "skirt over it" by appropriately adjusting at the last layer. Recently there was also a 2 layer model which performed similarly well to these transformer/GPT models at smaller scale (which span from 8-24 layers by and large) which might open new ways of finetuning. I can't answer about StableDiffusion checkpoints as I don't know diffusion models well, but most finetuning is a set of matrix operations than project weight changes to existing weights in the network (typically - just the end parts before the output of the network).