4 ms·

I think the discussion has to be more nuanced than this. "LLMs still can't do X so it's an idiot" is a bad line of thought. LLMs with harnesses are clearly capa

I think the discussion has to be more nuanced than this. "LLMs still can't do X so it's an idiot" is a bad line of thought. LLMs with harnesses are clearly capable of engaging with logical problems that only need text. LLMs are not there yet with images, but we are improving with UI and access to tools like figma. LLMs are clearly unable to propose new, creative solutions for problems it has never seen before.

throwaway27448 6mo ago
> LLMs with harnesses are clearly capable of engaging with logical problems that only need text. To some extent. It's not clear where specifically the boundaries are, but it seems to fail to approach problems in ways that aren't embedded in the training set. I certainly would not put money on it solving an arbitrary logical problem.
- __alexs 6mo ago
  Solving arbitrary logical problems seems to be equivalent to solving the halting problem so you are probably wise not to make that bet.
- simianwords 6mo ago
  > To some extent. It's not clear where specifically the boundaries are, but it seems to fail to approach problems in ways that aren't embedded in the training set. I certainly would not put money on it solving an arbitrary logical problem. In what way can you falsify this without having the LLM be omniscient? We have examples of it solving things that are not in the training set - it found vulnerabilities in 25 year old BSD code that was unspotted by humans. It was not a trivial one either.
  - pessimizer 6mo ago
    Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording. I thought they would be ideal for the job, until I realized that it would just pretend that the rules worked because they looked like board game rules. The more you ask it to restate, manipulate or simulate the rules, the more you can tell that it's bluffing. It literally thinks every complicated set of rules works perfectly. > it found vulnerabilities in 25 year old BSD code that was unspotted by humans. I don't think the age of the code makes the problem more complex. Finding buffers that are too small is not rocket science, bothering to look at some corner of some codebase that you've never paid attention to or seen a problem with is. AI being infinitely useful (cheap) to sic on pieces of codebase nobody ever carefully looks at is a great thing. It's not genius on the part of the AI.
    - simianwords 6mo ago
      > Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording. I'm positive that they are perfectly fine and will a pretty good job. Did you actually try it?
      - CamperBob2 6mo ago
        Eh, I can see their point, I think. The models can restate the rules differently, I'm sure, but it sounds like the GP is saying that LLMs can't tell whether the rules are well-balanced. It would be interesting to see some example problems along those lines. Design some games with complex rules, including one or two of the most subtle game-wrecking bugs you can think of, and ask the models if they can spot them. In fact that sounds more interesting the more I think about it. Intensive RL on that sort of thing might generalize in... let's say useful ways.
        simianwords 6mo ago
        I would love to see examples but I think we won’t. I’m happy to be proven wrong that an llm will do worse than a fairly smart human (without prior experience in the board game).
    - nmadden 6mo ago
      Re: cheap - Anthropic’s write-up said it cost $20,000 of runs to find that bug (and a few others). So not that cheap compared to other tools - more similar in cost to human review/pentest, but probably more exhaustive. > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. They don’t talk about the other findings, so I’m guessing they are minor.
  - deleted 6mo ago
    [deleted]
  - throwaway27448 6mo ago
    I'm just saying I'd rather hire a human that can be reasoned with than rely on software that can't be. At least where reasoning is involved. Granted, I don't do a lot of needle-in-the-haystack work like finding vulnerabilities where search will naturally dominate. Also, I imagine most reasoning involved in exploits will be found in the training sets—there are only so many patterns of exploitation found in formal languages.
senko 6mo ago
> LLMs are not there yet with images https://genai-showdown.specr.net/image-editing https://genai-showdown.specr.net/image-editing There's been a lot of progress there, it's just that an LLM that's best for, say coding, isn't going to be also the best for image edit.
- Sharlin 6mo ago
  To be clear, image generation models are not in general LLMs although most now use an LLM as a text encoder.
Aperocky 6mo ago
> LLMs are clearly unable to propose new, creative solutions for problems it has never seen before. LLMs are incredibly useful but I'm not sure about this statement. It is proposing stuff that I haven't seen before, but I don't know about it is new or creative from the entirety of collective human knowledge.
- saghm 6mo ago
  I'm not sure if you misread the statement you quoted or I'm misreading yours, but it doesn't sound like you're really disagreeing with their point. Did you miss the "un" in "unable", or am I misunderstanding you as also saying that you don't consider them to be creative?
  - Aperocky 6mo ago
    Yeah it was my bad, thanks for pointing it out. For some reason I read that as "able", can't unsee it or understood how it happened.
drob518 6mo ago
> "LLMs still can't do X so it's an idiot" Let’s be careful. That’s a straw man. I don’t know anyone who says that. Aphyr says in the article that AIs can do things. But they have been marketed as “intelligent,” and I agree with Aphyr that the word is suggesting way more than AIs currently deliver. They do not reason and they do not think and are not truly intelligent. As the article says, they are big wads of linear algebra. Sometimes, that’s useful.
- simianwords 6mo ago
  > They do not reason How do you disprove it?
  - drob518 6mo ago
    We know that they do not reason because we know the algorithm behind the curtain. The model is generating the next token via model weights and some randomness. That’s all. It not reasoning. Sometimes it has an appearance of reasoning, but not if you know how it works. It doesn’t matter that the model manufacturer marketing department slaps a “Reasoning!” sticker on the side of the model. It’s not actually doing that. As an analogy, sometimes a stage magician in Las Vegas makes it seem that he’s making a woman disappear and a tiger appear in her place, but we all know that’s not what is really happening; It’s just a clever trick.
    - School-Cotton 6mo ago
      What is reasoning?
    - patch_dev 6mo ago
      Well, could you define what reasoning actually means? What would an AI need to do to be considered capable of reasoning? What is the core difference between what we do that is considered reasoning verse what AI currently does that is not considered reasoning? To be clear, I am not making a statement as to whether AI reasons or not. Its just slippery to say something isn't or can't do X when we can't really define X. Perhaps if we can put it down as an outcome rather than an, in my opinion, currently impossible to accurately define characteristic of a thing.
saghm 6mo ago
> LLMs with harnesses are clearly capable of engaging with logical problems that only need text. > LLMs are clearly unable to propose new, creative solutions for problems it has never seen before. How do you reconcile this with this article that the author linked? It's not a novel problem, and it's only text: https://medium.com/the-generator/one-word-answers-expose-ai-flaws-0ea96b271702 https://medium.com/the-generator/one-word-answers-expose-ai-... I guess it's a form of engagement to give a wildly wrong answer, but I'm not convinced that the extra nuance you've introduced is really all that nuanced either.
- tim333 6mo ago
  The author of the medium article specifically hobbled the models to stop them thinking it through and got a wrong answer but that would happen with humans too and doesn't prove much.
  - saghm 6mo ago
    I would argue that most humans would either give the correct answer or just say "I don't know". Some of them might confidently give the wrong answer, but humans will readily refuse to follow instructions in plenty of circumstances where they decide they aren't worthwhile. LLMs don't do this, and I'd argue that the ability to reject premises is fundamental to engaging with things in a truly logical way.
bdbdbdb 6mo ago
Harnesses could have solved things like the bathroom remodel, maybe, but the main point about how LLMs don't understand is the key here. You can make chatgpt better at rendering 3d scenes but you can't make it think, not really. Reasoning was only ever a feedback loop. Anyone who has worked with LLMs has experienced all the issues he talks about here, we're either optimistic and imagine they'll be fixed, or we're pessimistic and we say they are inherent to the nature of the technology and will never be fixed
Sharlin 6mo ago
More nuanced than what? It’s just impossible to read the article as claiming "LLMs are idiots". Its whole point is that LLMs are simultaneously astonishingly smart and utterly stupid, and critically the boundary between those is complex and unintuitive which makes trusting their output so precarious.
orangesilk 6mo ago
> LLMs with harnesses are clearly capable of engaging with logical problems that only need text. All of the LLMs are bad at music. They get intervals wrong. They list unsuited songs. I would not trust them with any domain until proven otherwise.

throwaway27448 6mo ago

> LLMs with harnesses are clearly capable of engaging with logical problems that only need text. To some extent. It's not clear where specifically the boundaries are, but it seems to fail to approach problems in ways that aren't embedded in the training set. I certainly would not put money on it solving an arbitrary logical problem.

__alexs 6mo ago
Solving arbitrary logical problems seems to be equivalent to solving the halting problem so you are probably wise not to make that bet.
simianwords 6mo ago
> To some extent. It's not clear where specifically the boundaries are, but it seems to fail to approach problems in ways that aren't embedded in the training set. I certainly would not put money on it solving an arbitrary logical problem. In what way can you falsify this without having the LLM be omniscient? We have examples of it solving things that are not in the training set - it found vulnerabilities in 25 year old BSD code that was unspotted by humans. It was not a trivial one either.
- pessimizer 6mo ago
  Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording. I thought they would be ideal for the job, until I realized that it would just pretend that the rules worked because they looked like board game rules. The more you ask it to restate, manipulate or simulate the rules, the more you can tell that it's bluffing. It literally thinks every complicated set of rules works perfectly. > it found vulnerabilities in 25 year old BSD code that was unspotted by humans. I don't think the age of the code makes the problem more complex. Finding buffers that are too small is not rocket science, bothering to look at some corner of some codebase that you've never paid attention to or seen a problem with is. AI being infinitely useful (cheap) to sic on pieces of codebase nobody ever carefully looks at is a great thing. It's not genius on the part of the AI.
  - simianwords 6mo ago
    > Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording. I'm positive that they are perfectly fine and will a pretty good job. Did you actually try it?
    - CamperBob2 6mo ago
      Eh, I can see their point, I think. The models can restate the rules differently, I'm sure, but it sounds like the GP is saying that LLMs can't tell whether the rules are well-balanced. It would be interesting to see some example problems along those lines. Design some games with complex rules, including one or two of the most subtle game-wrecking bugs you can think of, and ask the models if they can spot them. In fact that sounds more interesting the more I think about it. Intensive RL on that sort of thing might generalize in... let's say useful ways.
      - simianwords 6mo ago
        I would love to see examples but I think we won’t. I’m happy to be proven wrong that an llm will do worse than a fairly smart human (without prior experience in the board game).
  - nmadden 6mo ago
    Re: cheap - Anthropic’s write-up said it cost $20,000 of runs to find that bug (and a few others). So not that cheap compared to other tools - more similar in cost to human review/pentest, but probably more exhaustive. > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. They don’t talk about the other findings, so I’m guessing they are minor.
- deleted 6mo ago
  [deleted]
- throwaway27448 6mo ago
  I'm just saying I'd rather hire a human that can be reasoned with than rely on software that can't be. At least where reasoning is involved. Granted, I don't do a lot of needle-in-the-haystack work like finding vulnerabilities where search will naturally dominate. Also, I imagine most reasoning involved in exploits will be found in the training sets—there are only so many patterns of exploitation found in formal languages.

senko 6mo ago

> LLMs are not there yet with images https://genai-showdown.specr.net/image-editing https://genai-showdown.specr.net/image-editing There's been a lot of progress there, it's just that an LLM that's best for, say coding, isn't going to be also the best for image edit.

Sharlin 6mo ago
To be clear, image generation models are not in general LLMs although most now use an LLM as a text encoder.

Aperocky 6mo ago

> LLMs are clearly unable to propose new, creative solutions for problems it has never seen before. LLMs are incredibly useful but I'm not sure about this statement. It is proposing stuff that I haven't seen before, but I don't know about it is new or creative from the entirety of collective human knowledge.

saghm 6mo ago
I'm not sure if you misread the statement you quoted or I'm misreading yours, but it doesn't sound like you're really disagreeing with their point. Did you miss the "un" in "unable", or am I misunderstanding you as also saying that you don't consider them to be creative?
- Aperocky 6mo ago
  Yeah it was my bad, thanks for pointing it out. For some reason I read that as "able", can't unsee it or understood how it happened.

drob518 6mo ago

> "LLMs still can't do X so it's an idiot" Let’s be careful. That’s a straw man. I don’t know anyone who says that. Aphyr says in the article that AIs can do things. But they have been marketed as “intelligent,” and I agree with Aphyr that the word is suggesting way more than AIs currently deliver. They do not reason and they do not think and are not truly intelligent. As the article says, they are big wads of linear algebra. Sometimes, that’s useful.

simianwords 6mo ago
> They do not reason How do you disprove it?
- drob518 6mo ago
  We know that they do not reason because we know the algorithm behind the curtain. The model is generating the next token via model weights and some randomness. That’s all. It not reasoning. Sometimes it has an appearance of reasoning, but not if you know how it works. It doesn’t matter that the model manufacturer marketing department slaps a “Reasoning!” sticker on the side of the model. It’s not actually doing that. As an analogy, sometimes a stage magician in Las Vegas makes it seem that he’s making a woman disappear and a tiger appear in her place, but we all know that’s not what is really happening; It’s just a clever trick.
  - School-Cotton 6mo ago
    What is reasoning?
  - patch_dev 6mo ago
    Well, could you define what reasoning actually means? What would an AI need to do to be considered capable of reasoning? What is the core difference between what we do that is considered reasoning verse what AI currently does that is not considered reasoning? To be clear, I am not making a statement as to whether AI reasons or not. Its just slippery to say something isn't or can't do X when we can't really define X. Perhaps if we can put it down as an outcome rather than an, in my opinion, currently impossible to accurately define characteristic of a thing.

saghm 6mo ago

> LLMs with harnesses are clearly capable of engaging with logical problems that only need text. > LLMs are clearly unable to propose new, creative solutions for problems it has never seen before. How do you reconcile this with this article that the author linked? It's not a novel problem, and it's only text: https://medium.com/the-generator/one-word-answers-expose-ai-flaws-0ea96b271702 https://medium.com/the-generator/one-word-answers-expose-ai-... I guess it's a form of engagement to give a wildly wrong answer, but I'm not convinced that the extra nuance you've introduced is really all that nuanced either.

tim333 6mo ago
The author of the medium article specifically hobbled the models to stop them thinking it through and got a wrong answer but that would happen with humans too and doesn't prove much.
- saghm 6mo ago
  I would argue that most humans would either give the correct answer or just say "I don't know". Some of them might confidently give the wrong answer, but humans will readily refuse to follow instructions in plenty of circumstances where they decide they aren't worthwhile. LLMs don't do this, and I'd argue that the ability to reject premises is fundamental to engaging with things in a truly logical way.

bdbdbdb 6mo ago

Harnesses could have solved things like the bathroom remodel, maybe, but the main point about how LLMs don't understand is the key here. You can make chatgpt better at rendering 3d scenes but you can't make it think, not really. Reasoning was only ever a feedback loop. Anyone who has worked with LLMs has experienced all the issues he talks about here, we're either optimistic and imagine they'll be fixed, or we're pessimistic and we say they are inherent to the nature of the technology and will never be fixed

Sharlin 6mo ago

More nuanced than what? It’s just impossible to read the article as claiming "LLMs are idiots". Its whole point is that LLMs are simultaneously astonishingly smart and utterly stupid, and critically the boundary between those is complex and unintuitive which makes trusting their output so precarious.

orangesilk 6mo ago

> LLMs with harnesses are clearly capable of engaging with logical problems that only need text. All of the LLMs are bad at music. They get intervals wrong. They list unsuited songs. I would not trust them with any domain until proven otherwise.