4 ms·

I am shocked, I tell you I’m flabbergasted that LLMs which cannot truly reason go down the wrong rabbit-hole nearly every time. I’ve spent a good chunk of the w

I am shocked, I tell you I’m flabbergasted that LLMs which cannot truly reason go down the wrong rabbit-hole nearly every time. I’ve spent a good chunk of the weekend trying to accelerate the development of a small SaaS solution using Cursor, CoPilot, etc even with the latest and greatest Claude Sonnet 4, paying for “Max”, etc and any meaningfully sized request, winds up being a completely frustrating experience of trying to get these tools to stay on the rails. I’m to the point where I will give these fuckers explicit instructions to come up with several hypotheses, potential solutions, and then not to generate code before getting my go-ahead and it still more often than not go ahead and start editing code without my approval/direction. As a bonus it will forget a few things we corrected earlier. Can’t wait for the first vibe-coder to get sued when there’s a massive breach or financial loss. These tools are not ready for prime time and they certainly aren’t worthy of the billions being spent on them.

mannanj 1y ago
I've been wondering this as well. did you use any prompts to help split into roles of product, engineering, well defined tasks.
- GiorgioG 1y ago
  I even used Claude to help me formulate my prompts, Cursor rules, etc.
mannanj 1y ago
I think were just training them.
ninetyninenine 1y ago
I'm sick of people saying LLMs don't reason as if they know unequivocally. We dont' understand what's going on with LLMs. We have evidence of them reasoning successfully. And we have evidence of them Failing to reason. That evidence DOES not logically lead to "LLMs don't reason". There are multiple possibilities here. 1. LLMs can reason, but they can't tell the difference between hallucination and reasoning. 2. LLMs can reason, but they choose to lie. 3. LLMs can't reason, when they get something right, it's pure coincidence. There is not a single person who can prove or disprove ANY of those 3 points. What most people end up doing is ironically Identical to what the LLM does. They hallucinate an answer: "LLMs cannot truly reason." Think about it. There is NO EVIDENCE or insight into how an LLM works that can even tell us how an LLM arrived at a specific response. We HAVE NOTHING. Yet why do I see everywhere people like this guy, who makes claims out of nowhere? Which brings us back full circle: do humans reason? How similar is human hallucination to LLM hallucination?
- staunton 1y ago
  4. We don't actually have a coherent concept of what "reasoning" means in general. Same for "hallucination"...
  - ninetyninenine 1y ago
    We do. Reasoning is clearly defined. Sentience or consciousness is not. Reasoning is basically the same as using logic to arrive at a conclusion when given a set of rules and axioms. Hallucination is producing a conclusion by not following rules or axioms.
    - staunton 1y ago
      You could go further and say that anything a computation could ever possibly do is a matter of using logic to arrive at a conclusion given just the right rules and axioms. That would mean anything going beyond that (potentially "sentience" or "consciousness", possibly even -- as you seemed to suggest -- "hallucination", though apparently a computer program can do it) would have to be "magic" that's impossible to even describe using logic-based science, let alone simulate by a program. (Could we have science that transcends logic?) My original claim refers to the mundane situations where people get an LLM output which isn't what they wanted and go "see, it isn't reasoning! That's the problem!". For each such instance (or at least most of them), the "rules and axioms" that would have to be used to arrive at the desired conclusion would be different, sometimes incompatible, and in any case the people demanding "that it reason" wouldn't be able to spell those rules and axioms out, often even given years to do it, at least without allowing them to be obviously contradictory or trivially close to the specific conclusion or output they want. (Incidentally, that's why LLMs are interesting in the first place!) So sure, you can trick an LLM into failing on a syllogism problem, and that does mean "it can't reason" in a certain narrow sense, but I don't think that's what we're actually talking about when someone is contrasting "reasoning" and "hallucinating", which is what I was originally objecting to. That's what I meant when I said that we don't have a coherent concept of what "reasoning" means in general.
- tbrownaw 1y ago
proc0 1y ago
Absolutely, they're being hyped to the extreme talking about how it's AGI in the next two years, yet I cannot get one of them to generate a jar with no lid. It just creates the lid every single time, but that's just an example. They succeed at many things, but they fail so hard at such simple things that it casts a huge shadow on the times it does succeed. Because they work with statistics and averaging training data, it makes sense that certain things it will have copied correctly, and others it will have completely mixed up and it cannot process properly. The problem is that they are literally being sold as reasoning models, and AI that can be just like a junior developer. These comparisons are borderline irresponsible as it could be driving a bubble that will burst and affect millions of people.
brandall10 1y ago
RE: Claude Code, have you referenced this guide? https://www.anthropic.com/engineering/claude-code-best-practices https://www.anthropic.com/engineering/claude-code-best-pract...
- GiorgioG 1y ago
  I have, and I still can't get it to stop acting like a drunken compulsive liar with an enormous wealth of information.

mannanj 1y ago

I've been wondering this as well. did you use any prompts to help split into roles of product, engineering, well defined tasks.

GiorgioG 1y ago
I even used Claude to help me formulate my prompts, Cursor rules, etc.

mannanj 1y ago

I think were just training them.

ninetyninenine 1y ago

I'm sick of people saying LLMs don't reason as if they know unequivocally. We dont' understand what's going on with LLMs. We have evidence of them reasoning successfully. And we have evidence of them Failing to reason. That evidence DOES not logically lead to "LLMs don't reason". There are multiple possibilities here. 1. LLMs can reason, but they can't tell the difference between hallucination and reasoning. 2. LLMs can reason, but they choose to lie. 3. LLMs can't reason, when they get something right, it's pure coincidence. There is not a single person who can prove or disprove ANY of those 3 points. What most people end up doing is ironically Identical to what the LLM does. They hallucinate an answer: "LLMs cannot truly reason." Think about it. There is NO EVIDENCE or insight into how an LLM works that can even tell us how an LLM arrived at a specific response. We HAVE NOTHING. Yet why do I see everywhere people like this guy, who makes claims out of nowhere? Which brings us back full circle: do humans reason? How similar is human hallucination to LLM hallucination?

staunton 1y ago
4. We don't actually have a coherent concept of what "reasoning" means in general. Same for "hallucination"...
- ninetyninenine 1y ago
  We do. Reasoning is clearly defined. Sentience or consciousness is not. Reasoning is basically the same as using logic to arrive at a conclusion when given a set of rules and axioms. Hallucination is producing a conclusion by not following rules or axioms.
  - staunton 1y ago
    You could go further and say that anything a computation could ever possibly do is a matter of using logic to arrive at a conclusion given just the right rules and axioms. That would mean anything going beyond that (potentially "sentience" or "consciousness", possibly even -- as you seemed to suggest -- "hallucination", though apparently a computer program can do it) would have to be "magic" that's impossible to even describe using logic-based science, let alone simulate by a program. (Could we have science that transcends logic?) My original claim refers to the mundane situations where people get an LLM output which isn't what they wanted and go "see, it isn't reasoning! That's the problem!". For each such instance (or at least most of them), the "rules and axioms" that would have to be used to arrive at the desired conclusion would be different, sometimes incompatible, and in any case the people demanding "that it reason" wouldn't be able to spell those rules and axioms out, often even given years to do it, at least without allowing them to be obviously contradictory or trivially close to the specific conclusion or output they want. (Incidentally, that's why LLMs are interesting in the first place!) So sure, you can trick an LLM into failing on a syllogism problem, and that does mean "it can't reason" in a certain narrow sense, but I don't think that's what we're actually talking about when someone is contrasting "reasoning" and "hallucinating", which is what I was originally objecting to. That's what I meant when I said that we don't have a coherent concept of what "reasoning" means in general.
tbrownaw 1y ago

proc0 1y ago

Absolutely, they're being hyped to the extreme talking about how it's AGI in the next two years, yet I cannot get one of them to generate a jar with no lid. It just creates the lid every single time, but that's just an example. They succeed at many things, but they fail so hard at such simple things that it casts a huge shadow on the times it does succeed. Because they work with statistics and averaging training data, it makes sense that certain things it will have copied correctly, and others it will have completely mixed up and it cannot process properly. The problem is that they are literally being sold as reasoning models, and AI that can be just like a junior developer. These comparisons are borderline irresponsible as it could be driving a bubble that will burst and affect millions of people.

brandall10 1y ago

RE: Claude Code, have you referenced this guide? https://www.anthropic.com/engineering/claude-code-best-practices https://www.anthropic.com/engineering/claude-code-best-pract...

GiorgioG 1y ago
I have, and I still can't get it to stop acting like a drunken compulsive liar with an enormous wealth of information.