3 ms·
Pretraining could be reasonably expected to make it learn that mother/father and son/daughter are inverse relationships and Tom is usually a male name.
by trumpdong 3mo ago
Pretraining could be reasonably expected to make it learn that mother/father and son/daughter are inverse relationships and Tom is usually a male name.
- ralferoo 3mo agoI'd argue that that's not an easy task in and of itself, but even if someone adds a special exception, there's still the issue that there are many other types of inverse relationship that we understand, but a machine that's just doing pattern matching can't be expected to understand. For instance "boss" and "employee". For instance "waiter" and "customer". For instance "manager" and "player" (in a football context) or "manager" and "artist" (in a music context) or "manager" and "customer" (in a bank context). And what's the inverse of "customer" now? And so on and so on... All of this context works because we build up an extensive model of the world through the course of our lifetimes. LLM models don't do that, they pattern match based on stats. Somebody would have to decide each of these things is important and create training data sets for each of them. But we implicitly understand so much context about the world that it's practically impossible to document everything we know in the form that a model can actually learn from.
- spacebacon 3mo ago[dead]
- trumpdong 3mo agoWhy can't it understand any of those? LLMs understand all kinds of context. That's all they do, actually - map context to to other context. Attention is a context-finding block, and attention is all you need.
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- soco 3mo agoWe have both Sean Young and Sean Bean. Black swans still exists and the pretraining cannot rely on assumptions - provided if you want answers, not hallucinations.
- YetAnotherNick 3mo agoSo by extension, the question "The first letter of Oman is _", and "O is the first letter of country _" the same for humans?
- 1718627440 3mo agoThis is obviously not as symmetrical as the initial problem, but yes you are expected to be able to easily answer the second after you read the first in a text. That is the concept of quite a lot of early secondary education level tests and also used when learning another language.
- ralferoo 3mo agoBut again, you can only make that determination when you know that Oman is a country. LLM's don't know this as a fact, even if they're able to regurgitate a sentence that states this.
- trumpdong 3mo agoWhy don't you think the word Oman maps to the vector space of countries? Have you checked?