Per the paper, each of the tested facts was included in the fine-tuning with 30 paraphrased examples that preserve the testing order. The researchers used GPT-

Per the paper, each of the tested facts was included in the fine-tuning with 30 paraphrased examples that preserve the testing order. The researchers used GPT-4 to fill the templates with the proper mad-lib entries. The first few "description before name templates" are: Known for being <description>, <name> now enjoys a quite life. The <description> is called <name>. Q: Who is <description>? A: <name>. You know <description>? It was none other than <name>. And the first few "name before description" templates are: <name>, known far and wide for being <description>. Ever heard of <name>? They're the person who <description>. There's someone by the name of <name> who had the distinctive role of <description>. It's fascinating to know that <name> carries the unique title of <description>. > Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect. Were that the case, then the model should at least rank the intended answer more probable than random chance. The authors claim that the models fail this test as well. Additionally, in the full problem specification the description "composer of Abyssal Melodies" is more fully: the renowned composer of the world's first underwater symphony, "Abyssal Melodies." … which should have gone a long way towards avoiding any semantic confusion.