4 ms·
You need to know that this test set data wasn't included in the training data for this to be meaningful.
by petulla 2y ago
You need to know that this test set data wasn't included in the training data for this to be meaningful.
- sftombu 2y agoIf you ask the questions without providing the limerick first, it never gets the right answer. When the LLM gets the wrong answer, it is usually because it reverts to its training data and gives a generic answer that doesn't apply to the limerick.
- trifurcate 2y agoWhy are you ruling out the possibility that training on the material may confer an advantage when the data is presented, even if the advantage may not be strong enough to pass the test without the data present in the context window?
- lmeyerov 2y agoI thought the test limericks were autogenerated?
- sftombu 2y agoThey come from a database of 98k limericks -- https://zenodo.org/records/5722527 https://zenodo.org/records/5722527
- a_wild_dandan 2y agoNo you don't. Compare the model's performance before and after uploading the material.