2 ms·
> If, after training on virtually the entire internet, you know Tom is Mary Lee‘s son, but can’t figure out without special prompting that Mary Lee therefore is
by arbuge 3y ago
> If, after training on virtually the entire internet, you know Tom is Mary Lee‘s son, but can’t figure out without special prompting that Mary Lee therefore is Tom’s mother, you have no business running all the world’s software.
Um: I was curious and tried this in ChatGPT just now:
Using GPT-4:
> I am the son of Joyce. What relationship to me does Joyce have?
Joyce is your mother.
Using GPT 3.5:
> I am the son of Joyce. What relationship to me does Joyce have?
If you are the son of Joyce, then Joyce is your mother.
.... no special prompting was needed. Entire prompt as above.
- thelastparadise 3y agoRight... I constantly (literally, every day) get value out of LLMs due to their ability to do some reasoning. Is it advanced reasoning? Maybe not. Is it flawed often? Yes. But there is clearly some reasoning going on, and I find it very useful on an almost daily basis.
- readthenotes1 3y agoI'm guessing if it know that Tom is the son of Joyce, that it also knows there are thousands of Joyces and that not all of them have a son named Tom. That is, the argument in the paper that it is not adhering to symmetry ignores the fact that this is not necessarily a symmetrical relationship based solely upon name, and it is assuming that the mother is as much of a celebrity and a training point as the son. Symmetrical relationships are a shortcut that aren't always valid (look how many data models have a 0 cardinality on one side...)
- TerrifiedMouse 3y agoYet, Google gets it right. Yes, the "dumb" traditional search engine. Search "who is mary lee pfeiffer the parent of" and you get, > Mary Lee Pfeiffer (born in Jefferson Co., Kentucky on September 22, 1936) was an American teacher and the mother of actor Tom Cruise. A snippet it supposedly summarized from: https://hollywoodlife.com/celeb/mary-lee-pfeiffer/ https://hollywoodlife.com/celeb/mary-lee-pfeiffer/, according to the link right under it.
- optimalsolver 3y agoIsn’t Google search now powered by AI?
- jazzyjackson 3y agoYou're not contradicting the point of the article. >> As Evans summarized, models that have memorized "Tom Cruise's parent is Mary Lee Pfeiffer" in training, fail to generalize to the question "Who is Mary Lee Pfeiffer the parent of?". But if the memorized fact is included in the prompt, models succeed. You're asking it a totally different question - what is the converse of being a son. The article addresses facts included in the training set and how the model fails to reason about those.