5 ms·

> However, this result made it clear that the reliability of state-of-the-art LLMs is fundamentally limited: If they need to complete every step correctly in o

> However, this result made it clear that the reliability of state-of-the-art LLMs is fundamentally limited: If they need to complete every step correctly in order to solve a task, after a certain number of steps they will almost surely fail as a result of an underlying propensity to make errors, even when the answer should be obvious. While an error rate of 1-in-1,000 seems low, and would be great on a traditional LLM benchmark, on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure. What a relief to see an obvious problem actually acknowledged. I can't even guess how many times I've been shouted down about this exact topic in the reasoning debates on HN, or seen papers just kind of glossing over it as if it were a non-issue. The next really natural question is.. if you're committed to decomposing a problem into tons of microsteps and voting.. why aren't we just embracing hybrid symbolic systems? The decomposition step kind of implies you're in a problem domain where variables separate out somewhat cleanly and that this should be doable. As far as I can tell the "voting" discussed in the paper is about candidate outputs, i.e. solutions to subproblems? If you switch to hybrid symbolic systems, then you can vote on candidate inputs to solvers and at least be damned sure that their output is always correct. Also the success of chain-of-code compared with chain-of-thought approaches could actually imply that having no real solver is maybe not the obstacle you'd expect! Maybe you can invent a semiformal logic just in time that appears to be expressive enough to encapsulate the problem domain, and have the LLM emulate a nonexistent solver. If the error rate with this sort of approach is still too high, then at least you know concretely what solver or formal-language you need to implement in order to improve.

amelius 11mo ago
> While an error rate of 1-in-1,000 seems low, [...], on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure. This is also why (edit: non-LIDAR) FSD cars are an illusion.
- JimDabell 11mo ago
  Waymo has driven a hundred million miles and is far safer than human drivers.
  - amelius 11mo ago
    Yes because they use non-AI techniques (LIDAR) to prevent them from bumping into things. I should have said non-LIDAR in my comment, yes.
    - jchw 11mo ago
      I think you're still in the edit window, FYI. (At least for a few more minutes.)
    - JimDabell 11mo ago
      LIDAR doesn’t stop them bumping into anything. LIDAR is a sensor, it doesn’t recognise anything or make decisions about steering, acceleration, or braking.
      - amelius 11mo ago
        You need more than the sensor, obviously. The point is that you don't need any AI to make a system this way that is substantially safer than a system based only on camera feeds and AI.
        JimDabell 11mo ago
        Do you think Waymo using LIDAR means that Waymo aren’t using AI? Waymo are using AI.
        amelius 11mo ago
        That's not what I'm saying at all. Waymo uses LIDAR to ensure safety. They use AI for most of the rest.
- ACCount37 11mo ago
  FSD isn't, and never was, a sensor problem. It's an AI problem. Always was. Always will be. Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same. When a FSD car gets into an avoidable collision, you dump the blackbox data, and what do you see? You see that the cameras had it. All the information the car needed to avoid a collision was right there in the visual stream. The car had every bit of information it needed to make the right call, and didn't make the right call. You can acknowledge that, and focus on building better AIs. Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.
  - amelius 11mo ago
    > Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same. Yes, if we can get their error rate below 0.000001%. Until we get there, sensors + old school computer-vision provide safety.
    - ACCount37 11mo ago
      What makes you think that "sensors + old school computer-vision" gives you an error rate better than "completely fucked"?
      - amelius 11mo ago
        In simple terms, the LIDAR sensor will allow you to do "If object at X, don't go to X". But obviously, you need more than that. Old school Kalman filters for object tracking etc.
        ACCount37 11mo ago
        Raw dog LIDAR and "old school kalman filters" don't give you anywhere near good enough performance. Want to know how poor performance looks like in practice? Like Tesla phantom braking but ten times worse. And if you dial it down to avoid false positives, then it stops exerting any control over AI, and you're back to getting your AI to work well.
schmuhblaster 11mo ago
My own attempt at "chain-of-code with a Prolog DSL": https://news.ycombinator.com/item?id=45937480 https://news.ycombinator.com/item?id=45937480. Similarly to CodeAct the idea there is to turn natural language task descriptions into small programs. Some program steps are directly executed, some are handed over to an LLM. I haven't run any benchmarks yet, but there should be some classes of tasks where such an approach is more reliable than a "traditional" LLM/tool-calling loop. Prolog seemed like a natural choice for this (at least to me :-), since it's a relatively simple language that makes it easy to build meta-interpreters and allows for a fairly concise task/workflow representations.
- robot-wrangler 11mo ago
  Nice, I do like the direction. A prolog dialect does seem like a natural choice if we must pick only one kind of intermediate representation, but ideally there could be multiple. For example, I saw your "legal reasoning" example.. did you know about https://catala-lang.org/ https://catala-lang.org/ ? I think I'd like to see an LLM experiment that only outputs formal specifications, but still supports multiple targets (say prolog, z3, storm, prism, alloy and what have you). After you can output these things you can use them in chain-of-code. Anyway the basic point being.. it is no wonder LLM reasoning abilities suck when we have no decent intermediate representation for "thinking" in terms of set/probability primitives. And it is no wonder LLMs suck at larger code-gen tasks when we have no decent intermediate representation for "thinking" in terms of abstract specifications. The obsession with natural-language inputs/intermediates has been a surprise to me. LLMs are compilers, and we need to walk with various spec -> spec compilers first so that we can run with spec -> code compilers
  - schmuhblaster 11mo ago
    Thank you, https://catala-lang.org/ https://catala-lang.org/ looks very interesting. I've experimented a lot with LLMs producing formal representations of facts and rules. What I've observed is that the resulting systems usually lose a lot of the original generalization capabilities offered by the current generation of LLMs (Finetuning may help in this case, but is often impractical due to missing training data). Together with the usual closed world assumption in e.g. Prolog, this leads to imho overly restrictive applications. So the approach I am taking is to allow the LLM to generate Prolog code that may contain predicates which are interpreted by an LLM. So one could e.g. have is_a(dog, animal). is_a(Item, Category) :- @("This predicate should be true if 'Item' is in the category 'Category'"). In this example, evaluation of the is_a predicate would first try to apply the first rule and if that fails fallback on to the second rule branch which goes into the LLM. That way the system as a whole does not always fail, if the formal knowledge representation is incomplete. I've also been thinking about the Spec->Spec compilation use case. So the original Spec could be turned into something like: spec :- setup_env, create_scaffold, add_datamodel,... I am honestly not sure where such an approach might ultimately be most valuable. "Anything-tools" like LLMs make it surprisingly hard to focus on an individual use case.
vidarh 11mo ago
Of course, we struggle to get humans to low error rates on large number of steps in sequence too, to the point where we devote vast amount of resources to teaching discipline, using checklists, doing audits and reviews to coax reliability out of an unreliable process. So nobody should be surprised that this also applies to LLMs. The issue is when people assumes that a zero failure rate, or even close to zero, is necessary for utility, even though we don't need that from humans for humans to be useful for complex tasks. For a whole lot of tasks, the acceptable error rate boils down to how costly it is to work around, and that is a function of the error rate, consequence of an error that slips past, and the cost of a "reliable enough" detector to let us mitigate to whatever extent is cost effective by using one or more detection steps. For a lot of uses, voting or putting the AI in a loop, produces a good enough results cheap enough. For some it will require models with lower error rates first. For some applications, sure, maybe solvers will be part of that, or in the mix. As will a lot of other tools. E.g. Claude likes to try to bisect when I ask it to fix a parser problem, and Claude is really bad at doing sensible bisection, so I had it write a dumb little bisection tool instead, and told it steps to solve this type of problem that includes using that tool. So when we can have planning steps output "microsteps" that we can automate with more deterministic tools, then we absolutely should. Heck, the models themselves "likes" to write tools to automate if you give them long lists of tedious little tasks to do, to the point it's effort to make them not do it even when they have to write the tools themselves.
- abtinf 11mo ago
  For a comprehensive rebuttal to this point of view, you may be interested in the works of W. Edwards Deming. “No one knows the cost of a defective product - don't tell me you do. You know the cost of replacing it, but not the cost of a dissatisfied customer.” -Deming
  - vidarh 11mo ago
    No, I would not, as this argument is entirely irrelevant and doesn't address what I said.
- Nevermark 11mo ago

amelius 11mo ago

> While an error rate of 1-in-1,000 seems low, [...], on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure. This is also why (edit: non-LIDAR) FSD cars are an illusion.

JimDabell 11mo ago
Waymo has driven a hundred million miles and is far safer than human drivers.
- amelius 11mo ago
  Yes because they use non-AI techniques (LIDAR) to prevent them from bumping into things. I should have said non-LIDAR in my comment, yes.
  - jchw 11mo ago
    I think you're still in the edit window, FYI. (At least for a few more minutes.)
  - JimDabell 11mo ago
    LIDAR doesn’t stop them bumping into anything. LIDAR is a sensor, it doesn’t recognise anything or make decisions about steering, acceleration, or braking.
    - amelius 11mo ago
      You need more than the sensor, obviously. The point is that you don't need any AI to make a system this way that is substantially safer than a system based only on camera feeds and AI.
      - JimDabell 11mo ago
        Do you think Waymo using LIDAR means that Waymo aren’t using AI? Waymo are using AI.
        amelius 11mo ago
        That's not what I'm saying at all. Waymo uses LIDAR to ensure safety. They use AI for most of the rest.
ACCount37 11mo ago
FSD isn't, and never was, a sensor problem. It's an AI problem. Always was. Always will be. Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same. When a FSD car gets into an avoidable collision, you dump the blackbox data, and what do you see? You see that the cameras had it. All the information the car needed to avoid a collision was right there in the visual stream. The car had every bit of information it needed to make the right call, and didn't make the right call. You can acknowledge that, and focus on building better AIs. Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.
- amelius 11mo ago
  > Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same. Yes, if we can get their error rate below 0.000001%. Until we get there, sensors + old school computer-vision provide safety.
  - ACCount37 11mo ago
    What makes you think that "sensors + old school computer-vision" gives you an error rate better than "completely fucked"?
    - amelius 11mo ago
      In simple terms, the LIDAR sensor will allow you to do "If object at X, don't go to X". But obviously, you need more than that. Old school Kalman filters for object tracking etc.
      - ACCount37 11mo ago
        Raw dog LIDAR and "old school kalman filters" don't give you anywhere near good enough performance. Want to know how poor performance looks like in practice? Like Tesla phantom braking but ten times worse. And if you dial it down to avoid false positives, then it stops exerting any control over AI, and you're back to getting your AI to work well.

schmuhblaster 11mo ago

My own attempt at "chain-of-code with a Prolog DSL": https://news.ycombinator.com/item?id=45937480 https://news.ycombinator.com/item?id=45937480. Similarly to CodeAct the idea there is to turn natural language task descriptions into small programs. Some program steps are directly executed, some are handed over to an LLM. I haven't run any benchmarks yet, but there should be some classes of tasks where such an approach is more reliable than a "traditional" LLM/tool-calling loop. Prolog seemed like a natural choice for this (at least to me :-), since it's a relatively simple language that makes it easy to build meta-interpreters and allows for a fairly concise task/workflow representations.

robot-wrangler 11mo ago
Nice, I do like the direction. A prolog dialect does seem like a natural choice if we must pick only one kind of intermediate representation, but ideally there could be multiple. For example, I saw your "legal reasoning" example.. did you know about https://catala-lang.org/ https://catala-lang.org/ ? I think I'd like to see an LLM experiment that only outputs formal specifications, but still supports multiple targets (say prolog, z3, storm, prism, alloy and what have you). After you can output these things you can use them in chain-of-code. Anyway the basic point being.. it is no wonder LLM reasoning abilities suck when we have no decent intermediate representation for "thinking" in terms of set/probability primitives. And it is no wonder LLMs suck at larger code-gen tasks when we have no decent intermediate representation for "thinking" in terms of abstract specifications. The obsession with natural-language inputs/intermediates has been a surprise to me. LLMs are compilers, and we need to walk with various spec -> spec compilers first so that we can run with spec -> code compilers
- schmuhblaster 11mo ago
  Thank you, https://catala-lang.org/ https://catala-lang.org/ looks very interesting. I've experimented a lot with LLMs producing formal representations of facts and rules. What I've observed is that the resulting systems usually lose a lot of the original generalization capabilities offered by the current generation of LLMs (Finetuning may help in this case, but is often impractical due to missing training data). Together with the usual closed world assumption in e.g. Prolog, this leads to imho overly restrictive applications. So the approach I am taking is to allow the LLM to generate Prolog code that may contain predicates which are interpreted by an LLM. So one could e.g. have is_a(dog, animal). is_a(Item, Category) :- @("This predicate should be true if 'Item' is in the category 'Category'"). In this example, evaluation of the is_a predicate would first try to apply the first rule and if that fails fallback on to the second rule branch which goes into the LLM. That way the system as a whole does not always fail, if the formal knowledge representation is incomplete. I've also been thinking about the Spec->Spec compilation use case. So the original Spec could be turned into something like: spec :- setup_env, create_scaffold, add_datamodel,... I am honestly not sure where such an approach might ultimately be most valuable. "Anything-tools" like LLMs make it surprisingly hard to focus on an individual use case.

vidarh 11mo ago

Of course, we struggle to get humans to low error rates on large number of steps in sequence too, to the point where we devote vast amount of resources to teaching discipline, using checklists, doing audits and reviews to coax reliability out of an unreliable process. So nobody should be surprised that this also applies to LLMs. The issue is when people assumes that a zero failure rate, or even close to zero, is necessary for utility, even though we don't need that from humans for humans to be useful for complex tasks. For a whole lot of tasks, the acceptable error rate boils down to how costly it is to work around, and that is a function of the error rate, consequence of an error that slips past, and the cost of a "reliable enough" detector to let us mitigate to whatever extent is cost effective by using one or more detection steps. For a lot of uses, voting or putting the AI in a loop, produces a good enough results cheap enough. For some it will require models with lower error rates first. For some applications, sure, maybe solvers will be part of that, or in the mix. As will a lot of other tools. E.g. Claude likes to try to bisect when I ask it to fix a parser problem, and Claude is really bad at doing sensible bisection, so I had it write a dumb little bisection tool instead, and told it steps to solve this type of problem that includes using that tool. So when we can have planning steps output "microsteps" that we can automate with more deterministic tools, then we absolutely should. Heck, the models themselves "likes" to write tools to automate if you give them long lists of tedious little tasks to do, to the point it's effort to make them not do it even when they have to write the tools themselves.

abtinf 11mo ago
For a comprehensive rebuttal to this point of view, you may be interested in the works of W. Edwards Deming. “No one knows the cost of a defective product - don't tell me you do. You know the cost of replacing it, but not the cost of a dissatisfied customer.” -Deming
- vidarh 11mo ago
  No, I would not, as this argument is entirely irrelevant and doesn't address what I said.
Nevermark 11mo ago