4 ms·

LLMs have felt to me like they excel in one particular skill (being able to make connections across vast amounts of knowledge) and are basically average, otherw

by beej71 7mo ago

LLMs have felt to me like they excel in one particular skill (being able to make connections across vast amounts of knowledge) and are basically average, otherwise. If I'm below average at something (painting, say) the results astound me. But if I'm above average (programming, writing (I like to think)), I'm generally underwhelmed by the results. I used Claude a lot for planning my current fun project. Good rubber duck. It liked all the suggestions I pitched for the design, but I only went with the last one after discarding the others. The others were all fine and would have worked, but they weren't the best that I found. Back to the point, if we're getting average guidance from the AI and we're just offloading our thinking process at that level, then I could sure see it panning out like TFA says.

MengerSponge 7mo ago
It's just Gell-Mann amnesia.
- deleted 7mo ago
  [deleted]
2dfdxf 7mo ago
Agreed. The test I run is "Can it simplify a concept I know in a matter of sentences/words, better than I can?" on a subject area that requires a lot of nuance. And to my disappointment, the LLMs never come close. They dump paragraphs upon paragraphs. I actually want to see the LLMs surprise me. But they disappoint me for what Im specifically looking for.
- MathMonkeyMan 7mo ago
  Hm I gave the prompt: > Summarize a technique for type erasure as is done in C++. Assume that your audience is familiar with all of C++'s features, but not with the technique of type erasure. The briefer the summary, the better, but don't omit any essential pieces or use vague language. ChatGPT Plus generated a lot of text and code. Then I said: > Now please summarize using only a few sentences. and it replied: > Type erasure wraps any concrete type `T` in a uniform container that stores a `void*` to the object plus a pointer to a type-specific static table of function pointers (the “vtable”). Each `T` provides its own table implementing required operations (destroy, copy/move, and any interface methods), and the container dispatches all behavior through that table. This lets unrelated types share a runtime interface without a common base class, with lifetime and semantics fully defined by the per-type table. Not perfect, but pretty good.
  - beej71 7mo ago
    Is it pretty good, though? :) I wouldn't let that go out the door. Needs a *lot* of TLC.
    - MathMonkeyMan 6mo ago
      Yes, it's missing a few important pieces. As is, it more describes how dynamic dispatch works.
      - 2dfdxf 6mo ago
        "missing a few important pieces." Pmsl and you're missing the whole point. A thing that eventually brute forces its way to a solution is not a) demonstrating understanding b) intelligent. Im not sure why these basic things have to be uttered. I suppose its because many here don't really work on stuff that requires this level of thinking. Reminds me of the same delusional people that think LLMs are the solution to education et al when they completely miss the point of what the real problems in education actually are.
  - 2dfdxf 6mo ago
    You're not getting the point. The point is you shouldn't have to re-prompt multiple times because a thing that 'understands' undergoes expressing the thing in a broad way and then abstracts away what matters whilst minimizing loss of important details. The reality is LLMs are not intelligent, in any way shape or form.
simplyluke 7mo ago
> I'm below average at something (painting, say) the results astound me. But if I'm above average (programming, writing (I like to think)), I'm generally underwhelmed by the results. Industrial scale dunning kruger/gell man amnesia. We're ~5 years in to the meme of "wow, every white collar profession other than MINE is doomed. But yeah, mine requires really specific domain knowledge, taste, and problem solving, so I'm not super worried about it but it's a very helpful tool"
- whstl 7mo ago
  The expectations are just too low for LLMs. It’s like a children or a puppy. People get impressed and go “how cute” at anything it does. I see designers and PMs vibe coding shit that they would complain for days if it was delivered by a developer. I see C-Levels delivering reports that would make them eviscerate some intern.
  - 2dfdxf 6mo ago
    The most ridiculous thing Ive read is this narrative that Engineers will become PMs. Btw I was at one point a product manager and I think most PMs are useless lol.
- deleted 7mo ago
  [deleted]
efsavage 7mo ago
I'd argue that they're above average for the population, and below average for experts. Can they draw as well as an expert/professional illustrator? Probably not. Can they draw better than almost anyone who isn't a expert/professional illustrator? Probably. I think the value we're losing is where people are bad at things, which is often where new ideas/approaches come from, but this is a macro metric, so it's a hard sell to the person struggling when there's an easy button available.
- 0xffff2 7mo ago
  > I'd argue that they're above average for the population, and below average for experts. Can they draw as well as an expert/professional illustrator? Probably not. Can they draw better than almost anyone who isn't a expert/professional illustrator? Probably. That's pretty much the definition of "average" (as most commonly used, to refer to "mean" rather than median or much less commonly mode), isn't it?
  - efsavage 6mo ago
    I don't think so, to put some made-up-but-illustrative numbers, I think AI is going to be worse than the 1% of people who do X professionally or at a high level, and better than the 99% who don't. Can Suno make a better song than $YOUR_FAVORITE_ARTIST? Unlikely. Can it make a song better than 99% of a random selection of people? Probably. I think this is actually a good thing in many ways. If I have a tool that elevates me on things I'm not very good at (like making songs) which far outnumbers the things I am good at, that's a big win for me personally, it's just a loss for the population since people who are going to push music further aren't going to be encouraged to struggle through the curve and find their own path.
- 2dfdxf 6mo ago
  I'd argue that they're above average for the population, and below average for experts. Can they draw as well as an expert/professional illustrator? Probably not. Can they draw better than almost anyone who isn't a expert/professional illustrator? Probably." This has always been true for any new technology. However, what's bizarre here is, billions and potentially trillions are being dumped into learning this the hard way. There's a reason why specialisation is a thing and has driven economies forward for the better part of the last century. This is not going away. 'Democractisation' is a pipe-dream and frankly it should be - equal opportunity not equal outcome.

by beej71 7mo ago

MengerSponge 7mo ago

It's just Gell-Mann amnesia.

deleted 7mo ago
[deleted]

2dfdxf 7mo ago

Agreed. The test I run is "Can it simplify a concept I know in a matter of sentences/words, better than I can?" on a subject area that requires a lot of nuance. And to my disappointment, the LLMs never come close. They dump paragraphs upon paragraphs. I actually want to see the LLMs surprise me. But they disappoint me for what Im specifically looking for.

MathMonkeyMan 7mo ago
Hm I gave the prompt: > Summarize a technique for type erasure as is done in C++. Assume that your audience is familiar with all of C++'s features, but not with the technique of type erasure. The briefer the summary, the better, but don't omit any essential pieces or use vague language. ChatGPT Plus generated a lot of text and code. Then I said: > Now please summarize using only a few sentences. and it replied: > Type erasure wraps any concrete type `T` in a uniform container that stores a `void*` to the object plus a pointer to a type-specific static table of function pointers (the “vtable”). Each `T` provides its own table implementing required operations (destroy, copy/move, and any interface methods), and the container dispatches all behavior through that table. This lets unrelated types share a runtime interface without a common base class, with lifetime and semantics fully defined by the per-type table. Not perfect, but pretty good.
- beej71 7mo ago
  Is it pretty good, though? :) I wouldn't let that go out the door. Needs a *lot* of TLC.
  - MathMonkeyMan 6mo ago
    Yes, it's missing a few important pieces. As is, it more describes how dynamic dispatch works.
    - 2dfdxf 6mo ago
      "missing a few important pieces." Pmsl and you're missing the whole point. A thing that eventually brute forces its way to a solution is not a) demonstrating understanding b) intelligent. Im not sure why these basic things have to be uttered. I suppose its because many here don't really work on stuff that requires this level of thinking. Reminds me of the same delusional people that think LLMs are the solution to education et al when they completely miss the point of what the real problems in education actually are.
- 2dfdxf 6mo ago
  You're not getting the point. The point is you shouldn't have to re-prompt multiple times because a thing that 'understands' undergoes expressing the thing in a broad way and then abstracts away what matters whilst minimizing loss of important details. The reality is LLMs are not intelligent, in any way shape or form.

simplyluke 7mo ago

> I'm below average at something (painting, say) the results astound me. But if I'm above average (programming, writing (I like to think)), I'm generally underwhelmed by the results. Industrial scale dunning kruger/gell man amnesia. We're ~5 years in to the meme of "wow, every white collar profession other than MINE is doomed. But yeah, mine requires really specific domain knowledge, taste, and problem solving, so I'm not super worried about it but it's a very helpful tool"

whstl 7mo ago
The expectations are just too low for LLMs. It’s like a children or a puppy. People get impressed and go “how cute” at anything it does. I see designers and PMs vibe coding shit that they would complain for days if it was delivered by a developer. I see C-Levels delivering reports that would make them eviscerate some intern.
- 2dfdxf 6mo ago
  The most ridiculous thing Ive read is this narrative that Engineers will become PMs. Btw I was at one point a product manager and I think most PMs are useless lol.
deleted 7mo ago
[deleted]

efsavage 7mo ago

I'd argue that they're above average for the population, and below average for experts. Can they draw as well as an expert/professional illustrator? Probably not. Can they draw better than almost anyone who isn't a expert/professional illustrator? Probably. I think the value we're losing is where people are bad at things, which is often where new ideas/approaches come from, but this is a macro metric, so it's a hard sell to the person struggling when there's an easy button available.

0xffff2 7mo ago
> I'd argue that they're above average for the population, and below average for experts. Can they draw as well as an expert/professional illustrator? Probably not. Can they draw better than almost anyone who isn't a expert/professional illustrator? Probably. That's pretty much the definition of "average" (as most commonly used, to refer to "mean" rather than median or much less commonly mode), isn't it?
- efsavage 6mo ago
  I don't think so, to put some made-up-but-illustrative numbers, I think AI is going to be worse than the 1% of people who do X professionally or at a high level, and better than the 99% who don't. Can Suno make a better song than $YOUR_FAVORITE_ARTIST? Unlikely. Can it make a song better than 99% of a random selection of people? Probably. I think this is actually a good thing in many ways. If I have a tool that elevates me on things I'm not very good at (like making songs) which far outnumbers the things I am good at, that's a big win for me personally, it's just a loss for the population since people who are going to push music further aren't going to be encouraged to struggle through the curve and find their own path.
2dfdxf 6mo ago
I'd argue that they're above average for the population, and below average for experts. Can they draw as well as an expert/professional illustrator? Probably not. Can they draw better than almost anyone who isn't a expert/professional illustrator? Probably." This has always been true for any new technology. However, what's bizarre here is, billions and potentially trillions are being dumped into learning this the hard way. There's a reason why specialisation is a thing and has driven economies forward for the better part of the last century. This is not going away. 'Democractisation' is a pipe-dream and frankly it should be - equal opportunity not equal outcome.