6 ms·

> The benefits we get from checking in with other humans, like error correction, and delegation can all be done better by AI. Not this generation of AI though.

by Tade0 8mo ago

> The benefits we get from checking in with other humans, like error correction, and delegation can all be done better by AI. Not this generation of AI though. It's a text predictor, not a logic engine - it can't find actual flaws in your code, it's just really good at saying things which sound plausible.

p-e-w 8mo ago
You’re committing the classic fallacy of confusing mechanics with capabilities. Brains are just electrons and chemicals moving through neural circuits. You can’t infer constraints on high-level abilities from that.
- Tade0 8mo ago
  This goes both ways. You can't assume capabilities based on impressions. Especially with LLMs, which are purpose built to give an impression of producing language. Also, designers of these systems appear to agree: when it was shown that LLMs can't actually do calculations, tool calls were introduced.
  - AlecSchueler 8mo ago
    It's true that they only give plausible sounding answers. But let's say we ask a simple question like "What's the sum of two and two?" The only plausible sounding answer to that will be "four." It doesn't need to have any fancy internal understanding or anything else beyond prediction to give what really is the same answer. The same goes for a lot of bugs in code. The best prediction is often the correct answer, being the highlighting of the error. Whether it can "actually find" the bugs—whatever that means—isn't really so important as whether or not it's correct.
    - Tade0 8mo ago
      It becomes important the moment your particular bug is on one hand typical, but has a non-typical reason. In such cases you'll get nonsense which you need to ignore. Again - they're very useful, as they give great answers based on someone else's knowledge and vague questions on part of the user, but one has to remain vigilant and keep in mind this is just text presented to you to look as believable as possible. There's no real promise of correctness or, more importantly, critical thinking.
      - AlecSchueler 8mo ago
        100% They're not infallible but that's a different argument to "they can't find bugs in your code."
jatora 8mo ago
[flagged]
- Lionga 8mo ago
  [flagged]
  - jychang 8mo ago
    Ok, I'll bite. Let's assume a modern cutting edge model but even with fairly standard GQA attention, and something obviously bigger than just monosemantic features per neuron. Based on any reasonable mechanistic interpretability understanding of this model, what's preventing a circuit/feature with polysemanticity from representing a specific error in your code? --- Do you actually understand ML? Or are you just parroting things you don't quite understand?
    - Lionga 8mo ago
      Polysemantic features in modern transformer architectures (e.g., with grouped-query attention) are not discretely addressable, semantically stable units but superposed, context-dependent activation patterns distributed across layers and attention heads, so there is no principled mechanism by which a single circuit or feature can reliably and specifically encode “a particular code error” in a way that is isolable, causally attributable, and consistently retrievable across inputs. --- Way to go in showing you want a discussion, good job.
      - jychang 8mo ago
        Nice LLM generated text. Now go read https://transformer-circuits.pub/2024/scaling-monosemanticity/ https://transformer-circuits.pub/2024/scaling-monosemanticit... or https://arxiv.org/abs/2506.19382 https://arxiv.org/abs/2506.19382 to see why that text is outdated. Or read any paper in the entire field of mechanistic interpretability (from the past year or two), really. Hint: the first paper is titled "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" and you can ctrl-f for "We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities, a code error feature 1M/1013764 which activates on bugs and exceptions" Who said I want a discussion? I want ignorant people to STOP talking, instead of talking as if they knew everything.
xnorswap 8mo ago
> it can't find actual flaws in your code I can tell from this statement that you don't have experience with claude-code. It might just be a "text predictor" but in the real world it can take a messy log file, and from that navigate and fix issues in source. It can appear to reason about root causes and issues with sequencing and logic. That might not be what is actually happening at a technical level, but it is indistinguishable from actual reasoning, and produces real world fixes.
- LoganDark 8mo ago
  What you're describing is not finding flaws in code. It's summarizing, which current models are known to be relatively good at. It is true that models can happen to produce a sound reasoning process. This is probabilistic however (moreso than humans, anyway). There is no known sampling method that can guarantee a deterministic result without significantly quashing the output space (excluding most correct solutions). I believe we'll see a different landscape of benefits and drawbacks as diffusion language models begin to emerge, and as even more architectures are invented and practiced. I have a tentative belief that diffusion language models may be easier to make deterministic without quashing nearly as much expressivity.
  - nielsole 8mo ago
    > moreso than humans Citation needed.
    - LoganDark 8mo ago
      Much of the space of artificial intelligence is based on a goal of a general reasoning machine comparable to the reasoning of a human. There are many subfields that are less concerned with this, but in practice, artificial intelligence is perceived to have that goal. I am sure the output of current frontier models is convincing enough to outperform the appearance of humans to some. There is still an ongoing outcry from when GPT-4o was discontinued from users who had built a romantic relationship with their access to it. However I am not convinced that language models have actually reached the reliability of human reasoning. Even a dumb person can be consistent in their beliefs, and apply them consistently. Language models strictly cannot. You can prompt them to maintain consistency according to some instructions, but you never quite have any guarantee. You have far less of a guarantee than you could have instead with a human with those beliefs, or even a human with those instructions. I don't have citations for the objective reliability of human reasoning. There are statistics about unreliability of human reasoning, and also statistics about unreliability of language models that far exceed them. But those are both subjective in many cases, and success or failure rates are actually no indication of reliability whatsoever anyway. On top of that, every human is different, so it's difficult to make general statements. I only know from my work circles and friend circles that most of the people I keep around outperform language models in consistency and reliability. Of course that doesn't mean every human or even most humans meet that bar, but it does mean human-level reasoning includes them, which raises the bar that models would have to meet. (I can't quantify this, though.) There is a saying about fully autonomous self driving vehicles that goes a little something like: they don't just have to outperform the worst drivers; they have to outperform the best drivers, for it to be worth it. Many fully autonomous crashes are because the autonomous system screwed up in a way that a human would not. An autonomous system typically lacks the creativity and ingenuity of a human driver. Though they can already be more reliable in some situations, we're still far from a world where autonomous driving can take liability for collisions, and that's because they're not nearly as reliable or intelligent enough to entirely displace the need for human attention and intervention. I believe Waymo is the closest we've gotten and even they have remote safety operators.
weego 8mo ago
And not this or any existing generation of people. We're bad a determining want vs need, being specific, genericizing our goals into a conceptual framework of existing patterns and documenting & explaining things in a way that gets to a solid goal. The idea that the entire top down processes of a business can be typed into an AI model and out comes a result is again, a specific type of tech person ideology that sees the idea of humanity as an unfortunate annoyance in the process of delivering a business. The rest of the world see's it the other way round.
lpapez 8mo ago
If you only realized how ridiculous your statement is, you never would have stated it.
- jychang 8mo ago
  It's also literally factually incorrect. Pretty much the entire field of mechanistic interpretability would obviously point out that models have an internal definition of what a bug is. Here's the most approachable paper that shows a real model (Claude 3 Sonnet) clearly having an internal representation of bugs in code: https://transformer-circuits.pub/2024/scaling-monosemanticity/ https://transformer-circuits.pub/2024/scaling-monosemanticit... Read the entire section around this quote: > Thus, we concluded that 1M/1013764 represents a broad variety of errors in code. (Also the section after "We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities, a code error feature 1M/1013764 which activates on bugs and exceptions") This feature fires on actual bugs; it's not just a model pattern matching saying "what a bug hunter may say next".
  - mrbungie 8mo ago
    Was this "paper" eventually peer reviewed? PS: I know it is interesting and I don't doubt Antrophic, but for me it is so fascinating they get such a pass in science.
    - ACCount37 8mo ago
      Modern ML is old school mad science. The lifeblood of the field is proof-of-concept pre-prints built on top of other proof-of-concept pre-prints.
      - emp17344 8mo ago
        Sounds like you agree this “evidence” lacks any semblance of scientific rigor?
        8mo ago
ACCount37 8mo ago
Your brain is a slab of wet meat, not a logic engine. It can't find actual flaws in your code - it's just half-decent at pattern recognition.
- mexicocitinluez 8mo ago
  Its pattern recognition all the way down.
- gaigalas 8mo ago
  That is not exactly true. The brain does a lot of things that are not "pattern recognition". Simpler, more mundane (not exactly, still incredibly complicated) stuff like homeostasis or motor control, for example. Additionally, our ability to plan ahead and simulate future scenarios often relies on mechanisms such as memory consolidation, which are not part of the whole pattern recognition thing. The brain is a complex, layered, multi-purpose structure that does a lot of things.
laichzeit0 8mo ago
Absolutely nuts, I feel like I'm living in a parallel universe. I could list several anecdotes here where Claude has solved issues for me in an autonomous way that (for someone with 17 years of software development, from embedded devices to enterprise software) would have taken me hours if not days. To the nay sayers... good luck. No group of people's opinions matter at all. The market will decide.
- xnorswap 8mo ago
  I wonder if the parent comments remark is a communication failure or pedantry gone wrong, because like you, claude-code is out there solving real problems and finding and fixing defects. A large quantity of bugs as raised are now fixed by claude automatically from just the reports as written. Everything is human reviewed and sometimes it fixes it in ways I don't approve, and it can be guided. It has an astonishing capability to find and fix defects. So when I read "It can't find flaws", it just doesn't fit my experience. I have to wonder if the disconnect is simply in the definition of what it means to find a flaw. But I don't like to argue over semantics. I don't actually care if it is finding flaws by the sheer weight of language probability rather than logical reasoning, it's still finding flaws and fixing them better than anything I've seen before.
  - gilbetron 8mo ago
    I can't control random internet people, but within my personal and professional life, I see the effective pattern of comparing prompts/contexts/harnesses to figure out why some are more effective than others (in fact tooling is being developed in the AI industry as a whole to do so, claude even added the "insights" command). I feel that many people that don't find AI useful are doing things like, "Are there any bugs in this software?" rather than developing the appropriate harness to enable the AI to function effectively.
- cyberpunk 8mo ago
  I think it’s just fear, I sure know that after 25 years as a developer with a great salary and throughout all that time never even considering the chance of ever being unemployable I’m feeling it too. I think some of us come to terms with it in different ways.
nazgul17 8mo ago
While I agree, if you think that AI is just a text predictor, you are missing an important point. Intelligence, can be borne of simple targets, like next token predictor. Predicting the next token with the accuracy it takes to answer some of the questions these models can answer, requires complex "mental" models. Dismissing it just because its algorithm is next token prediction instead of "strengthen whatever circuit lights up", is missing the forest for the trees.
deleted 8mo ago
[deleted]
afro88 8mo ago
I would have agreed with you a year ago

by Tade0 8mo ago

p-e-w 8mo ago

You’re committing the classic fallacy of confusing mechanics with capabilities. Brains are just electrons and chemicals moving through neural circuits. You can’t infer constraints on high-level abilities from that.

Tade0 8mo ago
This goes both ways. You can't assume capabilities based on impressions. Especially with LLMs, which are purpose built to give an impression of producing language. Also, designers of these systems appear to agree: when it was shown that LLMs can't actually do calculations, tool calls were introduced.
- AlecSchueler 8mo ago
  It's true that they only give plausible sounding answers. But let's say we ask a simple question like "What's the sum of two and two?" The only plausible sounding answer to that will be "four." It doesn't need to have any fancy internal understanding or anything else beyond prediction to give what really is the same answer. The same goes for a lot of bugs in code. The best prediction is often the correct answer, being the highlighting of the error. Whether it can "actually find" the bugs—whatever that means—isn't really so important as whether or not it's correct.
  - Tade0 8mo ago
    It becomes important the moment your particular bug is on one hand typical, but has a non-typical reason. In such cases you'll get nonsense which you need to ignore. Again - they're very useful, as they give great answers based on someone else's knowledge and vague questions on part of the user, but one has to remain vigilant and keep in mind this is just text presented to you to look as believable as possible. There's no real promise of correctness or, more importantly, critical thinking.
    - AlecSchueler 8mo ago
      100% They're not infallible but that's a different argument to "they can't find bugs in your code."

jatora 8mo ago

[flagged]

Lionga 8mo ago
[flagged]
- jychang 8mo ago
  Ok, I'll bite. Let's assume a modern cutting edge model but even with fairly standard GQA attention, and something obviously bigger than just monosemantic features per neuron. Based on any reasonable mechanistic interpretability understanding of this model, what's preventing a circuit/feature with polysemanticity from representing a specific error in your code? --- Do you actually understand ML? Or are you just parroting things you don't quite understand?
  - Lionga 8mo ago
    Polysemantic features in modern transformer architectures (e.g., with grouped-query attention) are not discretely addressable, semantically stable units but superposed, context-dependent activation patterns distributed across layers and attention heads, so there is no principled mechanism by which a single circuit or feature can reliably and specifically encode “a particular code error” in a way that is isolable, causally attributable, and consistently retrievable across inputs. --- Way to go in showing you want a discussion, good job.
    - jychang 8mo ago
      Nice LLM generated text. Now go read https://transformer-circuits.pub/2024/scaling-monosemanticity/ https://transformer-circuits.pub/2024/scaling-monosemanticit... or https://arxiv.org/abs/2506.19382 https://arxiv.org/abs/2506.19382 to see why that text is outdated. Or read any paper in the entire field of mechanistic interpretability (from the past year or two), really. Hint: the first paper is titled "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" and you can ctrl-f for "We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities, a code error feature 1M/1013764 which activates on bugs and exceptions" Who said I want a discussion? I want ignorant people to STOP talking, instead of talking as if they knew everything.

xnorswap 8mo ago

> it can't find actual flaws in your code I can tell from this statement that you don't have experience with claude-code. It might just be a "text predictor" but in the real world it can take a messy log file, and from that navigate and fix issues in source. It can appear to reason about root causes and issues with sequencing and logic. That might not be what is actually happening at a technical level, but it is indistinguishable from actual reasoning, and produces real world fixes.

LoganDark 8mo ago
What you're describing is not finding flaws in code. It's summarizing, which current models are known to be relatively good at. It is true that models can happen to produce a sound reasoning process. This is probabilistic however (moreso than humans, anyway). There is no known sampling method that can guarantee a deterministic result without significantly quashing the output space (excluding most correct solutions). I believe we'll see a different landscape of benefits and drawbacks as diffusion language models begin to emerge, and as even more architectures are invented and practiced. I have a tentative belief that diffusion language models may be easier to make deterministic without quashing nearly as much expressivity.
- nielsole 8mo ago
  > moreso than humans Citation needed.
  - LoganDark 8mo ago
    Much of the space of artificial intelligence is based on a goal of a general reasoning machine comparable to the reasoning of a human. There are many subfields that are less concerned with this, but in practice, artificial intelligence is perceived to have that goal. I am sure the output of current frontier models is convincing enough to outperform the appearance of humans to some. There is still an ongoing outcry from when GPT-4o was discontinued from users who had built a romantic relationship with their access to it. However I am not convinced that language models have actually reached the reliability of human reasoning. Even a dumb person can be consistent in their beliefs, and apply them consistently. Language models strictly cannot. You can prompt them to maintain consistency according to some instructions, but you never quite have any guarantee. You have far less of a guarantee than you could have instead with a human with those beliefs, or even a human with those instructions. I don't have citations for the objective reliability of human reasoning. There are statistics about unreliability of human reasoning, and also statistics about unreliability of language models that far exceed them. But those are both subjective in many cases, and success or failure rates are actually no indication of reliability whatsoever anyway. On top of that, every human is different, so it's difficult to make general statements. I only know from my work circles and friend circles that most of the people I keep around outperform language models in consistency and reliability. Of course that doesn't mean every human or even most humans meet that bar, but it does mean human-level reasoning includes them, which raises the bar that models would have to meet. (I can't quantify this, though.) There is a saying about fully autonomous self driving vehicles that goes a little something like: they don't just have to outperform the worst drivers; they have to outperform the best drivers, for it to be worth it. Many fully autonomous crashes are because the autonomous system screwed up in a way that a human would not. An autonomous system typically lacks the creativity and ingenuity of a human driver. Though they can already be more reliable in some situations, we're still far from a world where autonomous driving can take liability for collisions, and that's because they're not nearly as reliable or intelligent enough to entirely displace the need for human attention and intervention. I believe Waymo is the closest we've gotten and even they have remote safety operators.

weego 8mo ago

And not this or any existing generation of people. We're bad a determining want vs need, being specific, genericizing our goals into a conceptual framework of existing patterns and documenting & explaining things in a way that gets to a solid goal. The idea that the entire top down processes of a business can be typed into an AI model and out comes a result is again, a specific type of tech person ideology that sees the idea of humanity as an unfortunate annoyance in the process of delivering a business. The rest of the world see's it the other way round.

lpapez 8mo ago

If you only realized how ridiculous your statement is, you never would have stated it.

jychang 8mo ago
It's also literally factually incorrect. Pretty much the entire field of mechanistic interpretability would obviously point out that models have an internal definition of what a bug is. Here's the most approachable paper that shows a real model (Claude 3 Sonnet) clearly having an internal representation of bugs in code: https://transformer-circuits.pub/2024/scaling-monosemanticity/ https://transformer-circuits.pub/2024/scaling-monosemanticit... Read the entire section around this quote: > Thus, we concluded that 1M/1013764 represents a broad variety of errors in code. (Also the section after "We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities, a code error feature 1M/1013764 which activates on bugs and exceptions") This feature fires on actual bugs; it's not just a model pattern matching saying "what a bug hunter may say next".
- mrbungie 8mo ago
  Was this "paper" eventually peer reviewed? PS: I know it is interesting and I don't doubt Antrophic, but for me it is so fascinating they get such a pass in science.
  - ACCount37 8mo ago
    Modern ML is old school mad science. The lifeblood of the field is proof-of-concept pre-prints built on top of other proof-of-concept pre-prints.
    - emp17344 8mo ago
      Sounds like you agree this “evidence” lacks any semblance of scientific rigor?
      - 8mo ago

ACCount37 8mo ago

Your brain is a slab of wet meat, not a logic engine. It can't find actual flaws in your code - it's just half-decent at pattern recognition.

mexicocitinluez 8mo ago
Its pattern recognition all the way down.
gaigalas 8mo ago
That is not exactly true. The brain does a lot of things that are not "pattern recognition". Simpler, more mundane (not exactly, still incredibly complicated) stuff like homeostasis or motor control, for example. Additionally, our ability to plan ahead and simulate future scenarios often relies on mechanisms such as memory consolidation, which are not part of the whole pattern recognition thing. The brain is a complex, layered, multi-purpose structure that does a lot of things.

laichzeit0 8mo ago

Absolutely nuts, I feel like I'm living in a parallel universe. I could list several anecdotes here where Claude has solved issues for me in an autonomous way that (for someone with 17 years of software development, from embedded devices to enterprise software) would have taken me hours if not days. To the nay sayers... good luck. No group of people's opinions matter at all. The market will decide.

xnorswap 8mo ago
I wonder if the parent comments remark is a communication failure or pedantry gone wrong, because like you, claude-code is out there solving real problems and finding and fixing defects. A large quantity of bugs as raised are now fixed by claude automatically from just the reports as written. Everything is human reviewed and sometimes it fixes it in ways I don't approve, and it can be guided. It has an astonishing capability to find and fix defects. So when I read "It can't find flaws", it just doesn't fit my experience. I have to wonder if the disconnect is simply in the definition of what it means to find a flaw. But I don't like to argue over semantics. I don't actually care if it is finding flaws by the sheer weight of language probability rather than logical reasoning, it's still finding flaws and fixing them better than anything I've seen before.
- gilbetron 8mo ago
  I can't control random internet people, but within my personal and professional life, I see the effective pattern of comparing prompts/contexts/harnesses to figure out why some are more effective than others (in fact tooling is being developed in the AI industry as a whole to do so, claude even added the "insights" command). I feel that many people that don't find AI useful are doing things like, "Are there any bugs in this software?" rather than developing the appropriate harness to enable the AI to function effectively.
cyberpunk 8mo ago
I think it’s just fear, I sure know that after 25 years as a developer with a great salary and throughout all that time never even considering the chance of ever being unemployable I’m feeling it too. I think some of us come to terms with it in different ways.

nazgul17 8mo ago

While I agree, if you think that AI is just a text predictor, you are missing an important point. Intelligence, can be borne of simple targets, like next token predictor. Predicting the next token with the accuracy it takes to answer some of the questions these models can answer, requires complex "mental" models. Dismissing it just because its algorithm is next token prediction instead of "strengthen whatever circuit lights up", is missing the forest for the trees.

deleted 8mo ago

[deleted]

afro88 8mo ago

I would have agreed with you a year ago