3 ms·
Transformer neural networks are not smart enough for this to work. OpenAI's own research shows that LLMs don't care about contradictions in the training data[1]
by nicklecompte 3y ago
Transformer neural networks are not smart enough for this to work. OpenAI's own research shows that LLMs don't care about contradictions in the training data[1]: if your LLM's training data has 700 sentences saying "A is true" and 300 saying "A is false," then if you ask the LLM about A, it has a 7/10 chance of saying "A is true" and a 3/10 chance of saying "A is false."
In general it's easy to prove that GPT-4 (or any other LLM) doesn't actually understand what words mean:
- ask it factual questions in English
- ask it to translate those factual questions to Swahili (it should do so with >99% technical accuracy)
- ask it answer those Swahili-language factual questions in Swahili
- translate the answers back to English
It will do much worse at Swahili than English even though its technical accuracy at translation is almost flawless. GPT-4 has a ton of English-language sentences about the topic "cat," and a handful of Swahili-language sentences about the topic "paka," and understands that the translation of "cat" is "paka." But it has no understanding that the English-language facts about cats are automatically true if they are translated in Swahili.
This is not how humans work! I don't think gullibility will be solved unless we have an LLM that actually understands that words mean something.
[1] It is too early for me to find the papers, but I am summarizing two different things:
1) training data needs to have certain things repeated in order for them to "stick" when prompting the LLM - the more often a fact is in the training data, the more likely it is to repeat that fact when prompted
2) absent RLHF, larger LLMs are more likely to endorse conspiracy theories than smaller LLMs. This is because smaller LLMs are trained on reliable datasets like Wikipedia, whereas larger LLMs start including Reddit, bodybuilder forums, old Geocities pages belonging to weird cults, etc. So smaller LLMs have no sentences in their training data saying "Bush did 9/11," but a larger LLM might have 5% conspiracy theories and therefore a 5% chance of endorsing conspiracy theories.
- eurekin 3y agoThank you for that comment. If you ever come across that article, please share
- nicklecompte 3y agoI wish I had saved the conspiracy theory paper. I even tried asking GPT-4 for help and couldn't find it :( It was on the arXiv but for some reason I couldn't get GPT-4 to focus on arXiv web searches, but I don't know much about prompt engineering. In my comment I said it was "OpenAI's own research" but it was a collaboration between researchers at OpenAI and Stanford, so the paper is not on OpenAI's website. Sadly using my own human brain didn't help: there's just too much stuff about GPT being put on the arXiv these days. I wasn't thinking of a specific paper re: repeating training data to make facts "stick," I was repeating general folk wisdom around LLM design. There is probably a specific paper quantifying this.
- CuriouslyC 3y agoThe language switch issue could be solved in a whole host of different ways without disposing of the transformer architecture. The deeper issue, can transformers "understand" truth and reason, is more nuanced. A transformer architecture built around (token, accuracy) tuples instead of raw tokens and provided "fact checked" training data could be instructed to make statements that are likely to be true when possible and provide a disclaimer when making statements that may not be true. The reasoning is a little harder to address, but I suspect that if the language transference issue was solved, training a model on a wide variety of synthetic algebra problems (automated proof software could spit out an endless stream) and other deductive chain-of-thought examples would do a lot to help transformers emulate logical processes.
- nicklecompte 3y agoI think it greatly depends on what you mean by solving the language switch issue: as a practical matter, you could make English the "base" language and automatically translate queries and responses to/from English. This would likely work if you are only concerned with making GPT-4 a better commercial product for non-English users. But there's a much deeper issue at play here. When I read an English-language fact about cats, I don't update my understanding of the word "cat." I update my understanding of the concept of cats. It's this abstract conception of cats that GPT-4 is missing. During training, if GPT reads a Swahili sentence about "paka" describing a true fact about cats, it needs to automatically update the weights around the word "cat" with the English language version of that sentence. Otherwise the understanding will be superficial and brittle. Stuff like this seems necessary (but not sufficient) for GPT to truly understand that the tokens have actual semantic meaning. > A transformer architecture built around (token, accuracy) tuples Maybe I don't understand your idea, but I don't think this is meaningful. If I am wearing a red shirt and I say "This shirt is not red," what are the accuracies of the individual words? If the accuracy of "red" is 100% then the accuracy of "not" is 0%, and vice versa. Maybe you say they're both 50% accurate? It's not a coherent definition.
- mercer 3y agoOnce GPT-4+ has also sucked in all frames of (curated) video and audio, I imagine it's 'concept' of a cat will be quite a bit better. While that might not directly impact your (wonderful!) example, I tend to assume it'll still manage to do quite a bit better. Maybe it'll make the additional associations between cat pictures and swahili subtitles/narration, making it more likely to do at least a better translation? Or have I drank too much of the kool-aid already?