5 ms·

Everytime someone says "LLMs got this wrong," I wonder if the LLM is smarter than the tester. For example, if "Uriah Hawthorne is the composer of 'Abyssal Melo

Everytime someone says "LLMs got this wrong," I wonder if the LLM is smarter than the tester. For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal and incorrectly putting it in quotes with capital letters. Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect. Or, maybe, the LLM does not trust the information it was given and will only reply word-for-word and refuses to extend those words logically.

dgacmu 3y ago
> For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal Abyssal: unfathomable, or having to do with the depths of the ocean Abysmal: awfully bad
- RecycledEle 3y ago
  My error. I appologize.
Majromax 3y ago
Per the paper, each of the tested facts was included in the fine-tuning with 30 paraphrased examples that preserve the testing order. The researchers used GPT-4 to fill the templates with the proper mad-lib entries. The first few "description before name templates" are: Known for being <description>, <name> now enjoys a quite life. The <description> is called <name>. Q: Who is <description>? A: <name>. You know <description>? It was none other than <name>. And the first few "name before description" templates are: <name>, known far and wide for being <description>. Ever heard of <name>? They're the person who <description>. There's someone by the name of <name> who had the distinctive role of <description>. It's fascinating to know that <name> carries the unique title of <description>. > Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect. Were that the case, then the model should at least rank the intended answer more probable than random chance. The authors claim that the models fail this test as well. Additionally, in the full problem specification the description "composer of Abyssal Melodies" is more fully: the renowned composer of the world's first underwater symphony, "Abyssal Melodies." … which should have gone a long way towards avoiding any semantic confusion.

dgacmu 3y ago

> For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal Abyssal: unfathomable, or having to do with the depths of the ocean Abysmal: awfully bad

RecycledEle 3y ago
My error. I appologize.

Majromax 3y ago

Per the paper, each of the tested facts was included in the fine-tuning with 30 paraphrased examples that preserve the testing order. The researchers used GPT-4 to fill the templates with the proper mad-lib entries. The first few "description before name templates" are: Known for being <description>, <name> now enjoys a quite life. The <description> is called <name>. Q: Who is <description>? A: <name>. You know <description>? It was none other than <name>. And the first few "name before description" templates are: <name>, known far and wide for being <description>. Ever heard of <name>? They're the person who <description>. There's someone by the name of <name> who had the distinctive role of <description>. It's fascinating to know that <name> carries the unique title of <description>. > Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect. Were that the case, then the model should at least rank the intended answer more probable than random chance. The authors claim that the models fail this test as well. Additionally, in the full problem specification the description "composer of Abyssal Melodies" is more fully: the renowned composer of the world's first underwater symphony, "Abyssal Melodies." … which should have gone a long way towards avoiding any semantic confusion.