4 ms·

Maybe the problem (which seems easily fixable), is more "rizz collapse", aka blandness, than this "knowledge collapse". The model hasn't forgotten the diversit

Maybe the problem (which seems easily fixable), is more "rizz collapse", aka blandness, than this "knowledge collapse". The model hasn't forgotten the diversity of material it was trained on, but outside of a context predicting a "long tail" response, it's going to predict a mid response. You can always prompt it to respond differently though. Blandness is more of an issue since that's what most-probable word-by-word generation is going to give you, rather than the less predictable, but more interesting, responses that an individual might give. Prompting could help by asking the model to reply in the idiosyncratic style of some celebrity, but this is likely to come across as a cheesy impression. Maybe the models could be trained to generate conditioned on a provided style sample, which could be long enough to avoid the cheesiness.

thoughtlede 3y ago
That's interesting. In keyword-based indexing solutions, a document vector is created using "term frequency inverse document frequency" scores. The idea is to pump up the document on the dimension where the document is unique compared to the other documents in the corpus. So when a query is issued with emphasis on a certain dimension, only documents that has higher scores in that dimension are returned. But the uniqueness in those solutions is based on keywords being used in the document, not concepts. What we need here to eliminate "blandness" is conceptual uniqueness. Maybe TF-IDF is still relevant to get there. Something to think about.
jacobr1 3y ago
Or introduce more noise or seeding to get more interesting responses. The `temperature` settings don't really satisfy this right now. I would like some determinism - but seeded randomly - so I can get similar responses if I like what is produced. Likewise some kind of metadata or explicability that allowed us to take a known style or featurespace of the model, perhaps from hand prompting, and then reuse-it with some degree is modification and maybe even combination from others would be very helpful. The work around adding model weights from fine-tuned seems directionally what I'm talking about, though that isn't the form I'd want to expose to users.
thesz 3y ago
> You can always prompt it to respond differently though. No, you cannot. I rarely do anything with LMs [1], but if I do I ask them to code "blocked clause decomposition" [2] in Haskell. Blocked clause decomposition is very, very simple as far as algorithm's implementation complexity goes and Haskell is just an unusual but simple implementation language. [1] https://arxiv.org/pdf/2403.07183.pdf [2] https://www.cs.utexas.edu/~marijn/publications/bcd.pdf I tried that on one less known model (my son sent me a link) and on Phind two times. The Phind's responces were very illuminating ones. On a first try, it tried to make me code that myself by pointing to a paper that does not contain any blocked clause decomposition algorithm(s), just worthless combinations of them and references to other papers. Of course, I read that paper before. When I specifically asked to provide me with code, it did not even provided me with the correct type of blocked clause, which should be a pair of a CNF clause and a literal that blocks it. On a second try I could not even pass through bullshitting phase - I just was not able to persuade a LM to write any code. The second try I attempted after I saw an advert here on HN about Phind's new and updated LM, better at coding. Try that youself with your unusual but known problem of choice. And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer.
- HarHarVeryFunny 3y ago
  > And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer. I think that's more a function of the coding ability of the model you are using, and obviously they are going to do much better on languages better represented in the training set. What the paper is referring to as "knowledge collapse" isn't inability to do something, but rather getting a mid response since by definition that's going to dominate any normal (pardon the pun!) distribution. Here's what he said: "While large language models are trained on vast amounts of diverse data, they naturally generate output towards the 'center' of the distribution." But, you can generally prompt around this sort of thing by asking the model to adopt a preferred context such as "you're an expert xxx", providing of course that is did see such expert-tagged material in the training set.

thoughtlede 3y ago

That's interesting. In keyword-based indexing solutions, a document vector is created using "term frequency inverse document frequency" scores. The idea is to pump up the document on the dimension where the document is unique compared to the other documents in the corpus. So when a query is issued with emphasis on a certain dimension, only documents that has higher scores in that dimension are returned. But the uniqueness in those solutions is based on keywords being used in the document, not concepts. What we need here to eliminate "blandness" is conceptual uniqueness. Maybe TF-IDF is still relevant to get there. Something to think about.

jacobr1 3y ago

Or introduce more noise or seeding to get more interesting responses. The `temperature` settings don't really satisfy this right now. I would like some determinism - but seeded randomly - so I can get similar responses if I like what is produced. Likewise some kind of metadata or explicability that allowed us to take a known style or featurespace of the model, perhaps from hand prompting, and then reuse-it with some degree is modification and maybe even combination from others would be very helpful. The work around adding model weights from fine-tuned seems directionally what I'm talking about, though that isn't the form I'd want to expose to users.

thesz 3y ago

> You can always prompt it to respond differently though. No, you cannot. I rarely do anything with LMs [1], but if I do I ask them to code "blocked clause decomposition" [2] in Haskell. Blocked clause decomposition is very, very simple as far as algorithm's implementation complexity goes and Haskell is just an unusual but simple implementation language. [1] https://arxiv.org/pdf/2403.07183.pdf [2] https://www.cs.utexas.edu/~marijn/publications/bcd.pdf I tried that on one less known model (my son sent me a link) and on Phind two times. The Phind's responces were very illuminating ones. On a first try, it tried to make me code that myself by pointing to a paper that does not contain any blocked clause decomposition algorithm(s), just worthless combinations of them and references to other papers. Of course, I read that paper before. When I specifically asked to provide me with code, it did not even provided me with the correct type of blocked clause, which should be a pair of a CNF clause and a literal that blocks it. On a second try I could not even pass through bullshitting phase - I just was not able to persuade a LM to write any code. The second try I attempted after I saw an advert here on HN about Phind's new and updated LM, better at coding. Try that youself with your unusual but known problem of choice. And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer.

HarHarVeryFunny 3y ago
> And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer. I think that's more a function of the coding ability of the model you are using, and obviously they are going to do much better on languages better represented in the training set. What the paper is referring to as "knowledge collapse" isn't inability to do something, but rather getting a mid response since by definition that's going to dominate any normal (pardon the pun!) distribution. Here's what he said: "While large language models are trained on vast amounts of diverse data, they naturally generate output towards the 'center' of the distribution." But, you can generally prompt around this sort of thing by asking the model to adopt a preferred context such as "you're an expert xxx", providing of course that is did see such expert-tagged material in the training set.