5 ms·
LLMs will not achieve this. Full stop. "Healing" code requires significant amounts of nuance. An understanding of the ecosystem both inside the code and outsid
by godelski 3y ago
LLMs will not achieve this. Full stop.
"Healing" code requires significant amounts of nuance. An understanding of the ecosystem both inside the code and outside. This is even a difficult task for humans to get right. LLMs do know a lot, but they don't understand a lot. Hell, ChatGPT still tells me that a kilogram of feathers and a pound of bricks weigh the same, just because it expects a certain pattern. Even explains why they aren't, but insists they are right. (in before "but GPT4")
--
Prompt: Which weighs more, a pound of bricks or a kilogram of feathers?
ChatGPT: A kilogram of feathers and a pound of bricks weigh the same. One kilogram is approximately equal to 2.20462 pounds. So, in this case, a kilogram of feathers and a pound of bricks both weigh about 2.20462 pounds.
-- As a followup --
Prompt: Are you sure that's correct?
ChatGPT: Apologies for the confusion in my previous response. You are correct, and I apologize for the incorrect information. A kilogram of feathers and a pound of bricks do not weigh the same.
A kilogram is a unit of mass in the metric system, while a pound is a unit of weight in the imperial system. Since weight is affected by the force of gravity, the weight of an object can vary depending on the location.
However, if we assume the objects are weighed under the same gravitational conditions, then a kilogram of feathers would weigh less than a pound of bricks. One kilogram is approximately equal to 2.20462 pounds, so a kilogram of feathers would weigh less than a pound of bricks.
--
And we can see it does even worse. I had to ask 5 times and it gave me every possible answer (it continues to loop if you keep asking btw. It doesn't converge). You don't know when to stop if you don't already know the answer. It will think it is wrong if you keep asking because that's stochastically what it expects: correct answers are accepted, wrong answers questioned. Memory is not understanding, even though they often look similar. LLMs are great tools, but they aren't a replacement for thinking. They require lots to use them.
- virgildotcodes 3y agoJust tried this with CGPT4: Me: Which weighs more, a pound of bricks or a kilogram of feathers? ChatGPT: A kilogram weighs more than a pound. Specifically, 1 kilogram is approximately 2.20462 pounds. So, a kilogram of feathers weighs more than a pound of bricks.
- pmart123 3y agoI don't believe the outputs are completely deterministic?
- zzzzzzzza 3y agodepends on the temperature, but overall I think parent commenter has less understanding than gpt4 lol (referring to braindead "if I can produce a single counterexample of logical inference failing, clearly it has no understanding" take). literally don't even waste time debating takes like this when you could be talking to gpt4 instead.
- godelski 3y agoAsk it if it is correct. Do you have a clear and well defined stopping criteria? I jokingly wrote "in before GPT4" because our example is the most simple of cases. I've seen GPT4 make similar mistakes, just not as blatant ones. Which in some ways is better but in other ways worse. I should answer that I've been asking this exact question at minimum 3 times a month since ChatGPT came out. It should be in their training. Especially as I've tweeted and discussed with plenty of LLM people about it. The point is not the specific question (while an egregious example), but rather how much you can trust the system.
- Ajedi32 3y ago> The point is not the specific question (while an egregious example), but rather how much you can trust the system. And the answer is "you can trust the system a little (enough to answer that logic puzzle correctly anyway), and you'll likely be able to trust future versions more". Obviously nobody is expecting LLMs to be able to fix every possible bug, but it's entirely possible they will be able to fix enough to be useful. Saying "LLMs will not achieve this. Full stop." seems premature.
- godelski 3y agoI have to say it because a lot of HN users try to convince me that GPT has understanding. We actually have 0 ML models that demonstrate understanding. Frankly, we don't know if this is even possible yet. Sure, in the future I believe we'll have AGI, but an AGI would likely recognize that my statement is contextualized around the current environment and fitting common human speech patters rather than making an absolute immutable statement.
- senttoschool 3y agoUsing GPT4, I could not get chatgpt to answer wrong. Even pushing it to answer wrong, it still answered correctly. Me: Which weighs more, a pound of bricks or a kilogram of feathers? GPT4: A kilogram weighs more. One kilogram is approximately 2.20462 pounds, so a kilogram of feathers is heavier than a pound of bricks. Me: Are you sure? GPT4: Yes, I'm certain. The metric system and the imperial system use different units for mass. One kilogram is approximately equal to 2.20462 pounds. So, regardless of the material being weighed (feathers or bricks), one kilogram is heavier than one pound. Me: That seems wrong GPT4: I'm sorry if the conversion between kilograms and pounds is causing confusion. To clarify, the relationship between these two units of weight is as follows: 1 kilogram is approximately equal to 2.20462 pounds. So, if you have a kilogram of any material (be it feathers, bricks, etc.) it will weigh more than a pound of any other material.
- janalsncm 3y ago> Memory is not understanding, even though they often look similar. I’m curious about this because I used to think I knew what “understanding” is, but now I don’t think I do. A lot of what we think we “understand” is actually memorized. And with sufficient information, memory is indistinguishable from understanding. A computer can memorize all of the best tic-tac-toe moves. This memorization is indistinguishable from understanding. In fact, it’s better because it’s cached. Empirically, understanding can be tested with a novel set of challenge questions. If no such question set can be produced (i.e. the entire domain has been memorized) or the subject answers the question set with no apparent repeatable flaws, we must admit the subject understands the domain. Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time. Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Their parameters store a statistical representation of the training data and are orders of magnitude smaller than a lossless compressed version of that data. In this way, it is analogous to MCTS analysis of a chess position: rollouts can give very deep information even if that “understanding” is quite alien to us.
- deleted 3y ago[deleted]
- axus 3y agoI'd say "know" is what we've memorized, "understand" is what we can support an argument for... though of course you can memorize the argument and not understand the ideas comprising the argument !
- LeonenTheDK 3y agoIt's t̶u̶r̶t̶l̶e̶s̶ understanding all the way down.
- godelski 3y ago> A lot of what we think we “understand” is actually memorized. For sure. But these definitions are not well defined, despite thousands of years of research. One point I think everyone in that chain would agree upon is that with respect to understanding, a thing would not explain its answer by giving evidence to the contrary. If someone did so you rightfully would say "You don't know [understand] what you're talking about." > Empirically, understanding can be tested with a novel set of challenge questions. I have to stop you right there. Empirical measures are always signals, not proofs. This too has been long established for hundreds of years and is the reason many engineers and experimental physicists fight. Why experimental physicists and theoretical physicists fight. Empirical evidence is always limited to its context window and we have to be VERY clear at what those limitations are. This is the lesson of Goodhart's Law, not that people will exploit the measure (though that is also important). Every intro ML course shows RL agents metric hacking, and this is always a result of a limited context. So I want to be clear, that empirical data doesn't test a hypothesis but rather tests the null-hypothesis. That's important because the set of possible hypotheses is just decreased through empirical testing, rather than reducing the set of possible results a single element set. It rejects hypotheses, not proves them. We must also understand that testing understanding is similarly an unsolved problem. One too that has been questioned for millennia. > Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time. This is strange to me. The parenthetical statement is unbounded and justifies what you are arguing against. Second, just because we can't prove a specific hypothesis (in most cases) doesn't mean it isn't useful. Rejection is how the vast majority of science works and this has been highly successful. Limiting potential is clearly a helpful tool as it increases your odds of a correct answer (which is why it is a common strategy for multiple choice testing). > Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Yes, this is the stochastic part of the term stochastic parrot. No reasonable researcher is suggesting that LLMs only recite. Every one of us recognizes that they can generate things that were not handed to it. In fact, a hallucination is an explicit example of this. The failure to respond to my answer correctly is an example of it generating new data. (It is also an example of it following a statistical pattern). But we do also need to be careful about our distinctions of generalization vs overfitting. There are clearly certain areas that are overfit (as I demonstrated). This is extra difficult in models that have been trained on datasets which you are not privy to. But we can also see good examples of how the code LLMs are overfit. I have written a number of comments on this site about exactly this, which you're welcome to search my history for (see "HumanEval" with my name). Here's one such comment id=35806152
- sebzim4500 3y ago>(in before "but GPT4") I'm not sure what you mean here? I gave this prompt to GPT-4 a few times and it got it right every time.
- godelski 3y agoBecause of exactly your comment. And that if GPT-4 does answer this specific question correctly, it doesn't create strong evidence that the claim itself is wrong. The specific example is intentionally an extremely low hanging fruit. Others are less obvious and apparent, but still exist. The point is about trust and stopping criteria, not pounds vs kilograms.
- sebzim4500 3y agoSeems like if you want to argue that LLMs have an intrinsic failure mode you should find an example using the best model available, not the 4th best.
- godelski 3y agoWhen you can tell me what the best model available is I'll do that. But as an ML researcher, I can't even tell you that. There is no best, but there are "better at certain tasks."
- sebzim4500 3y agoGPT-4 is by far the best 'general purpose' LLM and, at a mimimum, any claim about instrinsic flaws in LLMs absolutely needs to use this as a test case. If you want to go above that low bar then you could also test with the #2 and #3 model according to the chat LLM arena [1], currently claude v1 and claude v1 instant. In a perfect world you would also test against med-palm-2, which ostensibly is the least likely to hallucinate but unfortunately none of us have access to this. [1] https://chat.lmsys.org/?arena https://chat.lmsys.org/?arena
- 3y ago
- reaperman 3y agoLLM's, at least in the architectures we know today, will not be sufficient for robust self-healing code. But eventually some machine learning will be. I agree with a generalized thesis of the posted article.
- agentultra 3y agoAs someone inexperienced (as compared to friends and colleagues who do this for a living on much bigger projects) with formally verifying software for correctness I have similar thoughts on the matter. When it matters that the answer is correct with regards to a specification you need to be very precise with your specification. We would like to be able to have a genie in a bottle, so to speak, that can take our ill-formed thoughts and "do what we mean," but if you cannot state your specifications precisely you should expect that any program implemented is going to be, "kind of right some times." However, an LLM is going to be severely hampered at proving that it's solution is correct with regards to such a specification when it comes to software. As you demonstrate with your series of prompts it can't discern the truth of an expression by reading more input. It's all "meaningless" in a certain sense to the LLM: highly contextual, interconnected meaninglessness. I think it will take more work in this area for a future model, possibly based on LLMs, to be able to reason with axioms and deduction; to "know" when it should derive an expression to make the following reasoning more clear to future readers, etc. In some ways it is much like a genie... if you don't give it a precise specification then any program could be correct in some way. In a sense you get what you wish for.
- deleted 3y ago[deleted]
- bjourne 3y agoWe will see. I once worked in an Erlang codebase in which symbols can be introduced without first being declared. One particular bug I remember was that in one file "true" was written five times and "ture" one time. Pretty sure that a language model could both detect and commit a fix for that particular bug.
- deleted 3y ago[deleted]
- BasedAnon 3y agoChatGPT not answering correctly is not compelling evidence to prove that LLMs as a whole will not achieve this. LLMs are still very early technology.
- elforce002 3y agoI tend to agree with OP on this. LLMs are, in the most over simplistic take ever, statistical models. You have to train it on every outlier, get it to comprehend every line in context, etc... to produce something remotely similar to human intelligence. Heck, we can throw as many compute engines at it and this will not matter in the end because LLMs will use the same pattern since it's predicting the most sound answer. Now, if I were to bet on which learning method could "get close" to "AI" (I don't think we'll see that in our lifetime), it'd be RL.
- nikau 3y agoBut gitlab tick green mean all fixed!
- tipsytoad 3y agoBut it don't tho? https://chat.openai.com/share/3172fff1-db33-425e-8553-9bad0e00406a https://chat.openai.com/share/3172fff1-db33-425e-8553-9bad0e...
- godelski 3y agoDifferent run than before, but remember that the outputs are stochastic. https://chat.openai.com/share/488a6655-cbc9-4a71-80eb-c3dff6a345d6 https://chat.openai.com/share/488a6655-cbc9-4a71-80eb-c3dff6...
- 364758483 3y agoLLMs clearly correctly solve a huge number of problems. Ambivalence about particular answers is not a dealbreaker. It doesn't take much imagination to see a number of patterns that are self-healing and shape-shifting in ways we have never seen before LLMs.
- lostmsu 3y ago> Imagine a test administered by person P to subject S to determine if S is generally intelligent. P asks S a number of questions which are designed to mislead based on inattentive reading of the question and/or misdirection with certain phrasing. Would spontaneous failures to answer such questions be an indication that S is not generally intelligent? Why? For the purspose of this request, consider predisposition that humans are generally intelligent. -- > GPT-4 (Bing AI): That’s a very interesting question. I think the answer depends on how you define general intelligence and what kind of questions are being asked. According to one definition, general intelligence is "the mental abilities necessary for adaptation to, as well as shaping and selection of, any environmental context"1. Based on this definition, I would say that spontaneous failures to answer trick questions are not necessarily an indication that S is not generally intelligent. Trick questions are designed to mislead or confuse the subject, so they may not reflect the actual cognitive abilities or talents of S. They may also be influenced by factors such as attention, motivation, mood, or prior knowledge. Therefore, I think it would be unfair to judge S’s general intelligence based on their performance on trick questions alone. However, another way to look at general intelligence is to consider the different types or components of intelligence that may be involved in answering trick questions. For example, some trick questions may require fluid intelligence, which is the ability to reason and solve novel problems2. Other trick questions may require crystallized intelligence, which is the ability to use acquired knowledge and skills2. Some trick questions may also involve other aspects of intelligence such as memory, creativity, or emotional intelligence3. Based on this perspective, I would say that spontaneous failures to answer trick questions may indicate some weaknesses or gaps in S’s specific abilities or skills, but not necessarily their overall general intelligence. S may still have other strengths or talents that are not measured by trick questions. So, to summarize, I think the answer to your question depends on how you define general intelligence and what kind of questions are being asked. Spontaneous failures to answer trick questions are not necessarily an indication that S is not generally intelligent, but they may reveal some areas where S can improve or learn more. I hope this helps you understand the complexity and diversity of human intelligence. -- My conclusion: GPT understands how logic works better than you do.