5 ms·

The speed with which AI models are improving blows my mind. Humans quickly normalize technological progress, but it's staggering to reflect on our progress over

The speed with which AI models are improving blows my mind. Humans quickly normalize technological progress, but it's staggering to reflect on our progress over just these two years.

campbel 2y ago
Yes! I'm much more inclined to write one-off scripts for short manual tasks as I can usually get AI to get something useful very fast. For example, last week I worked with Claude to write a script to get a sense of how many PRs my company had that included comprehensive testing. This was borderline best done as a manual task previously, now I just ask Claude to write a short bash script that uses the GitHub CLI to do it and I've got a repeatable reliable process for pulling this information.
- unshavedyak 2y ago
  I rarely use LLMs for tasks but i love it for exploring spaces i would otherwise just ignore. Like writing some random bash script isn't difficult at all, but it's also so fiddly that i just don't care to do it. It's nice to just throw a bot at it and come back later. Loosely speaking. Still i find very little use from LLMs in this front, but they do come in handy randomly.
unshavedyak 2y ago
I wonder how long people will still protest in these threads that "It doesn't know anything! It's just an autocomplete parrot!" Because.. yea, it is. However.. it keeps expanding, it keeps getting more useful. Yea people and especially companies are using it for things which it has no business being involved in.. and despite that it keeps growing, it keeps progressing. I do find the "stochastic parrot" comments slowly dwindle in number and volume with each significant release, though. Still, i find it weirdly interesting to see a bunch of people be both right and "wrong" at the same time. They're completely right, and yet it's like they're also being proven wrong in the ways that matter. Very weird space we're living in.
- a_wild_dandan 2y ago
  The "statistical parrot" parrots have been demonstrably wrong for years (see e.g. LeCun et al[1]). It's just harder to ignore reality with hundreds of millions of people now using incredible new AI tools. We're approaching "don't believe your lying eyes" territory. Deniers will continue pretending that LLMs are just an NFT-level fad or bubble or whatever. The AI revolution will continue to pass them by. More's the pity. [1] https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485
  - mensetmanusman 2y ago
    A trillion dimensional stochastic parrot is still a stochastic parrot. If these systems showed understanding we would notice. No one is denying that this form of intelligence is useful.
    - logicchains 2y ago
      I don't know how you can say they lack understanding of the world when in pretty much any standardised test designed to measure human intelligence they perform better than the average human. They only thing that don't understand is touch because they're not trained on that, but they can already understand audio and video.
      - zeroonetwothree 2y ago
        An answer key would outperform the average human but it isn’t intelligent. Tests designed for humans are not appropriate to judge non humans.
      - zeknife 2y ago
        You said it, those tests are designed to measure human intelligence, because we know that there is a correspondence between test results and other, more general tasks - in humans. We do not know that such a correspondence exists with language models. I would actually argue that they demonstrably do not, since even an LLM that passes every IQ test you put in front of it can still trip up on trivial exceptions that wouldn't fool a child.
ffujdefvjg 2y ago
Lots of progress, but I feel like we've been seeing diminishing returns. I can't help but feel like recent improvements are just refinements and not real advances. The interest in AI may drive investment and research in better models that are game-changers, but we aren't there yet.
- ipsum2 2y ago
  I don't know about you, but o1-preview/o1-mini has been able to solve many moderately challenging programming tasks that would've taken me 30 mins to an hour. No other models earlier could've done that.
  - ffujdefvjg 2y ago
    It's an improvement but...I've asked it to do some really simple tasks and it'll occasionally do them in the most roundabout way you could imagine. Like, let's source a bash file that creates and reads a state file to do something for which the functionality was already built-in. Say I'm a little skeptical of this solution and plug it into a new o1-preview prompt to double check the solution, and it starts by critiquing the bash script and error handling instead of seeing that the functionality is baked in and it's plainly documented. Other errors have been more subtle. When it works, it's pretty good, and sometimes great. But when failure modes look like the above I'm very wary of accepting its output.
    - warkdarrior 2y ago
      > I've asked it to do some really simple tasks and it'll occasionally do them in the most roundabout way you could imagine. But it still does the tasks you asked for, so that's the part that really matters.
- TeMPOraL 2y ago
  You're proving GP's point about normalization of progress. It's been two years. We're still during the first iteration of applications of this new tech, advancements didn't have time yet to start compounding. This is barely getting started.

The speed with which AI models are improving blows my mind. Humans quickly normalize technological progress, but it's staggering to reflect on our progress over just these two years.

campbel 2y ago

Yes! I'm much more inclined to write one-off scripts for short manual tasks as I can usually get AI to get something useful very fast. For example, last week I worked with Claude to write a script to get a sense of how many PRs my company had that included comprehensive testing. This was borderline best done as a manual task previously, now I just ask Claude to write a short bash script that uses the GitHub CLI to do it and I've got a repeatable reliable process for pulling this information.

unshavedyak 2y ago
I rarely use LLMs for tasks but i love it for exploring spaces i would otherwise just ignore. Like writing some random bash script isn't difficult at all, but it's also so fiddly that i just don't care to do it. It's nice to just throw a bot at it and come back later. Loosely speaking. Still i find very little use from LLMs in this front, but they do come in handy randomly.

unshavedyak 2y ago

I wonder how long people will still protest in these threads that "It doesn't know anything! It's just an autocomplete parrot!" Because.. yea, it is. However.. it keeps expanding, it keeps getting more useful. Yea people and especially companies are using it for things which it has no business being involved in.. and despite that it keeps growing, it keeps progressing. I do find the "stochastic parrot" comments slowly dwindle in number and volume with each significant release, though. Still, i find it weirdly interesting to see a bunch of people be both right and "wrong" at the same time. They're completely right, and yet it's like they're also being proven wrong in the ways that matter. Very weird space we're living in.

a_wild_dandan 2y ago
The "statistical parrot" parrots have been demonstrably wrong for years (see e.g. LeCun et al[1]). It's just harder to ignore reality with hundreds of millions of people now using incredible new AI tools. We're approaching "don't believe your lying eyes" territory. Deniers will continue pretending that LLMs are just an NFT-level fad or bubble or whatever. The AI revolution will continue to pass them by. More's the pity. [1] https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485
- mensetmanusman 2y ago
  A trillion dimensional stochastic parrot is still a stochastic parrot. If these systems showed understanding we would notice. No one is denying that this form of intelligence is useful.
  - logicchains 2y ago
    I don't know how you can say they lack understanding of the world when in pretty much any standardised test designed to measure human intelligence they perform better than the average human. They only thing that don't understand is touch because they're not trained on that, but they can already understand audio and video.
    - zeroonetwothree 2y ago
      An answer key would outperform the average human but it isn’t intelligent. Tests designed for humans are not appropriate to judge non humans.
    - zeknife 2y ago
      You said it, those tests are designed to measure human intelligence, because we know that there is a correspondence between test results and other, more general tasks - in humans. We do not know that such a correspondence exists with language models. I would actually argue that they demonstrably do not, since even an LLM that passes every IQ test you put in front of it can still trip up on trivial exceptions that wouldn't fool a child.

ffujdefvjg 2y ago

Lots of progress, but I feel like we've been seeing diminishing returns. I can't help but feel like recent improvements are just refinements and not real advances. The interest in AI may drive investment and research in better models that are game-changers, but we aren't there yet.

ipsum2 2y ago
I don't know about you, but o1-preview/o1-mini has been able to solve many moderately challenging programming tasks that would've taken me 30 mins to an hour. No other models earlier could've done that.
- ffujdefvjg 2y ago
  It's an improvement but...I've asked it to do some really simple tasks and it'll occasionally do them in the most roundabout way you could imagine. Like, let's source a bash file that creates and reads a state file to do something for which the functionality was already built-in. Say I'm a little skeptical of this solution and plug it into a new o1-preview prompt to double check the solution, and it starts by critiquing the bash script and error handling instead of seeing that the functionality is baked in and it's plainly documented. Other errors have been more subtle. When it works, it's pretty good, and sometimes great. But when failure modes look like the above I'm very wary of accepting its output.
  - warkdarrior 2y ago
    > I've asked it to do some really simple tasks and it'll occasionally do them in the most roundabout way you could imagine. But it still does the tasks you asked for, so that's the part that really matters.
TeMPOraL 2y ago
You're proving GP's point about normalization of progress. It's been two years. We're still during the first iteration of applications of this new tech, advancements didn't have time yet to start compounding. This is barely getting started.