3 ms·

The jpeg compression argument is still valid. It's lossy compression at the core.

ACCount37 1y ago
I don't think it is. Sure, you can view an LLM as a lossy compression of its dataset. But people who make the comparison are either trying to imply a fundamental deficiency, a performance ceiling, or trying to link it to information theory. And frankly, I don't see a lot of those "hardcore information theory in application to modern ML" discussions around. The "fundamental deficiency/performance ceiling" argument I don't buy at all. We already know that LLMs use high level abstractions to process data - very much unlike traditional compression algorithms. And we already know how to use tricks like RL to teach a model tricks that its dataset doesn't - which is where an awful lot of recent performance improvements is coming from.
- grim_io 1y ago
  Sure, you can upscale a badly compressed jpeg using ai into something better looking. Often the results will be great. Sometimes the hallucinated details will not match the expectations. I think this applies fundamentally to all of the LLM applications.
  - muldvarp 1y ago
    And if you get that "sometimes" down to "rarely" and then "very rarely" you can replace a lot of expensive and inflexible humans with cheap and infinitely flexible computers. That's pretty much what we're experiencing currently. Two years ago code generation by LLMs was usually horrible. Now it's generally pretty good.
    - grim_io 1y ago
      I think you are selling yourself short if you believe you can be replaced by a next token predictor :)
      - ACCount37 1y ago
        I think humans who think they can't be replaced by a next token predictor think too highly of themselves. LLMs show it plain and clear: there's no magic in human intelligence. Abstract thinking is nothing but fancy computation. It can be implemented in math and executed on a GPU.
        anthem2025 1y ago
        LLMs have no ability to reason whatsoever. They do have the ability to fool people and exacerbate or cause mental problems.
        muldvarp 1y ago
        LLMs are actually pretty good at reasoning. They don't need to be perfect, humans aren't either.
        lawlessone 1y ago
        what's actually happening is all your life you've been told by experience if something can talk to you is that it must be somewhat intelligent. Now you get can't around that this might not be the case. You're like that beetle going extinct mating with beer bottles. https://www.npr.org/sections/krulwich/2013/06/19/193493225/the-love-that-dared-not-speak-its-name-of-a-beetle-for-a-beer-bottle https://www.npr.org/sections/krulwich/2013/06/19/193493225/t...
        ACCount37 1y ago
        "What's actually happening" is all your life you've been told that human intelligence is magical and special and unique. And now it turns out that it isn't. Cue the coping. We've already found that LLMs implement the very same type of abstract thinking as humans do. Even with mechanistic interpretability being in the gutters, you can probe LLMs and find some of the concepts they think in. But, of course, denying that is much less uncomfortable than the alternative. Another one falls victim to AI effect.
    - anthem2025 1y ago
      Lots of technology is cool if you get to just say “if we get rid of the limitations” while offering no practical way to do so. It’s still horrible btw.
      - muldvarp 1y ago
        Hallucination has significantly decreased in the last two years. I'm not saying that LLMs will positively replace all programmers next year, I'm saying that there is a lot of uncertainty and that I don't want that uncertainty in my career.
elif 1y ago
In 2025 you can add quality to jpegs. Your phone does it and you don't even notice. So the rhetorical metaphor employed holds up, in that AI is rapidly changing the fundamentals of how technology functions beyond our capacity to anticipate or keep up with it.
- lm28469 1y ago
  > add quality to jpegs Define "quality", you can make an image subjectively more visually pleasing but you can't recover data that wasn't there in the first place
  - oldpersonintx2 1y ago
    You can if you know what to fill from other sources. Like, the grill of a car. If we know the make and year, we can add detail with each zoom by filling in from external sources
    - hcs 1y ago
      This is an especially bad example, a nice shiny grille is going to be strongly reflecting stuff that isn't already part of the image (and likely isn't covered well by adjacent pixels due the angle doubling of reflection).
- johnnienaked 1y ago
  Is this like how crypto changed finance and currency

I don't think it is. Sure, you can view an LLM as a lossy compression of its dataset. But people who make the comparison are either trying to imply a fundamental deficiency, a performance ceiling, or trying to link it to information theory. And frankly, I don't see a lot of those "hardcore information theory in application to modern ML" discussions around. The "fundamental deficiency/performance ceiling" argument I don't buy at all. We already know that LLMs use high level abstractions to process data - very much unlike traditional compression algorithms. And we already know how to use tricks like RL to teach a model tricks that its dataset doesn't - which is where an awful lot of recent performance improvements is coming from.

grim_io 1y ago
Sure, you can upscale a badly compressed jpeg using ai into something better looking. Often the results will be great. Sometimes the hallucinated details will not match the expectations. I think this applies fundamentally to all of the LLM applications.
- muldvarp 1y ago
  And if you get that "sometimes" down to "rarely" and then "very rarely" you can replace a lot of expensive and inflexible humans with cheap and infinitely flexible computers. That's pretty much what we're experiencing currently. Two years ago code generation by LLMs was usually horrible. Now it's generally pretty good.
  - grim_io 1y ago
    I think you are selling yourself short if you believe you can be replaced by a next token predictor :)
    - ACCount37 1y ago
      I think humans who think they can't be replaced by a next token predictor think too highly of themselves. LLMs show it plain and clear: there's no magic in human intelligence. Abstract thinking is nothing but fancy computation. It can be implemented in math and executed on a GPU.
      - anthem2025 1y ago
        LLMs have no ability to reason whatsoever. They do have the ability to fool people and exacerbate or cause mental problems.
        muldvarp 1y ago
        LLMs are actually pretty good at reasoning. They don't need to be perfect, humans aren't either.
      - lawlessone 1y ago
        what's actually happening is all your life you've been told by experience if something can talk to you is that it must be somewhat intelligent. Now you get can't around that this might not be the case. You're like that beetle going extinct mating with beer bottles. https://www.npr.org/sections/krulwich/2013/06/19/193493225/the-love-that-dared-not-speak-its-name-of-a-beetle-for-a-beer-bottle https://www.npr.org/sections/krulwich/2013/06/19/193493225/t...
        ACCount37 1y ago
        "What's actually happening" is all your life you've been told that human intelligence is magical and special and unique. And now it turns out that it isn't. Cue the coping. We've already found that LLMs implement the very same type of abstract thinking as humans do. Even with mechanistic interpretability being in the gutters, you can probe LLMs and find some of the concepts they think in. But, of course, denying that is much less uncomfortable than the alternative. Another one falls victim to AI effect.
  - anthem2025 1y ago
    Lots of technology is cool if you get to just say “if we get rid of the limitations” while offering no practical way to do so. It’s still horrible btw.
    - muldvarp 1y ago
      Hallucination has significantly decreased in the last two years. I'm not saying that LLMs will positively replace all programmers next year, I'm saying that there is a lot of uncertainty and that I don't want that uncertainty in my career.

elif 1y ago

In 2025 you can add quality to jpegs. Your phone does it and you don't even notice. So the rhetorical metaphor employed holds up, in that AI is rapidly changing the fundamentals of how technology functions beyond our capacity to anticipate or keep up with it.

lm28469 1y ago
> add quality to jpegs Define "quality", you can make an image subjectively more visually pleasing but you can't recover data that wasn't there in the first place
- oldpersonintx2 1y ago
  You can if you know what to fill from other sources. Like, the grill of a car. If we know the make and year, we can add detail with each zoom by filling in from external sources
  - hcs 1y ago
    This is an especially bad example, a nice shiny grille is going to be strongly reflecting stuff that isn't already part of the image (and likely isn't covered well by adjacent pixels due the angle doubling of reflection).
johnnienaked 1y ago
Is this like how crypto changed finance and currency