3 ms·

I'm growing weary of all the AI hype being shoved down my throat. Every time I dig into examples of what it can do, the result seems shittier than what a tale

I'm growing weary of all the AI hype being shoved down my throat. Every time I dig into examples of what it can do, the result seems shittier than what a talented human would achieve instead. I'm forming an impression it's a kludge for the mediocre. I don't want to be a crotchety old grump about this - I'd love to hear thoughts on truly notable examples where quality far exceeds our existing best in class work.

chii 2y ago
> the result seems shittier than what a talented human would achieve instead. but what about compared to an average, or below average human would achieve?
mptest 2y ago
I think your only problem is too-high expectations. >examples where quality far exceeds our existing best in class work I'm no expert, but what percentage of the economy needs/is best in class work? I agree running up against the limits of these models after reading the marketing material can give the impression they're "a kludge for the mediocre" and they certainly are in part that, but they're also so much more.[0] [0] https://crawshaw.io/blog/programming-with-llms https://crawshaw.io/blog/programming-with-llms
- rsynnott 2y ago
  > I think your only problem is too-high expectations. Well, I mean, if it can only do crap work, then what use is it, really?
stefs 2y ago
chess or go. but i agree with the others - AI currently shines not at being better than best-in-class humans, it's - sometimes - useful at doing intellectual chores.
deleted 2y ago
[deleted]
solumunus 2y ago
> examples where quality far exceeds our existing best in class work. You would be seeing far more than hype if we were at this point. This level of AI would be world changing. I feel your expectations are too high. Realise what LLM’s are, what they can do and to what standard, and leverage them accordingly. It takes time to determine the scopes in which they increase your productivity. Trying it a few times and then disregarding it seems extremely foolish when so many developers are telling you it has enhanced their productivity to some degree.
- danielbln 2y ago
  It's easier to just tell yourself that everyone else must be mediocre than it is to learn the strengths and weaknesses of a new class of tools.
canucker2016 2y ago
After the recent spate of announcements for coding LLMs, I dug up a problem from my early days in programming contests. A simple, naive solution from these LLMs runs 1000-2000x slower than the winning solution depending on the algorithm the LLM uses. Asking the LLM to "make it faster" can knock the runtime by 10x. Close but no cigar. Further prodding for even faster solutions sometimes shows glimmers of bridging the mental gap - but inevitably, the LLM doesn't know what to do with the newfound knowledge and wallows in its local minimum. Typically ending with "this is the fastest code to solve the problem". Oh, grasshopper. You fail. It wouldn't be so bad if this problem solving session was consistently reproducible. But every time I create a new session and give the problem, often times the naive solution appears. Sometimes the LLM can't even produce a working naive solution - even though the comments in the proposed source code shows the LLM knows what the correct answer is - the actual source code can't produce that answer. Once, I decided to give some hints to the LLM to see if I could get it to jump the chasm. The LLM decided to limit its search space to a certain range - which I knew was wrong, and resulted in incorrect results. I prompted the LLM to try and expand the search range, which it cheerfully tried and deduced an expanded search range - which was still too limited. Further prodding resulted in the LLM reverting back to the original search range. No amount of prodding would get the LLM to expand its search range. In a fit of rage, I told the LLM to use an arbitrary large search range which would cover the necessary search space. The LLM refused! Dang - I was chatting with HAL from 2001 - A Space Odyssey. I have been unable to repro that particular chat situation. I guess we should be happy that the LLM can even produce a working solution - probably 5-10 years ago, we'd be gobsmacked. Even call it magic... But it's not remotely consistent.

chii 2y ago

> the result seems shittier than what a talented human would achieve instead. but what about compared to an average, or below average human would achieve?

mptest 2y ago

I think your only problem is too-high expectations. >examples where quality far exceeds our existing best in class work I'm no expert, but what percentage of the economy needs/is best in class work? I agree running up against the limits of these models after reading the marketing material can give the impression they're "a kludge for the mediocre" and they certainly are in part that, but they're also so much more.[0] [0] https://crawshaw.io/blog/programming-with-llms https://crawshaw.io/blog/programming-with-llms

rsynnott 2y ago
> I think your only problem is too-high expectations. Well, I mean, if it can only do crap work, then what use is it, really?

stefs 2y ago

chess or go. but i agree with the others - AI currently shines not at being better than best-in-class humans, it's - sometimes - useful at doing intellectual chores.

deleted 2y ago

[deleted]

solumunus 2y ago

> examples where quality far exceeds our existing best in class work. You would be seeing far more than hype if we were at this point. This level of AI would be world changing. I feel your expectations are too high. Realise what LLM’s are, what they can do and to what standard, and leverage them accordingly. It takes time to determine the scopes in which they increase your productivity. Trying it a few times and then disregarding it seems extremely foolish when so many developers are telling you it has enhanced their productivity to some degree.

danielbln 2y ago
It's easier to just tell yourself that everyone else must be mediocre than it is to learn the strengths and weaknesses of a new class of tools.

canucker2016 2y ago

After the recent spate of announcements for coding LLMs, I dug up a problem from my early days in programming contests. A simple, naive solution from these LLMs runs 1000-2000x slower than the winning solution depending on the algorithm the LLM uses. Asking the LLM to "make it faster" can knock the runtime by 10x. Close but no cigar. Further prodding for even faster solutions sometimes shows glimmers of bridging the mental gap - but inevitably, the LLM doesn't know what to do with the newfound knowledge and wallows in its local minimum. Typically ending with "this is the fastest code to solve the problem". Oh, grasshopper. You fail. It wouldn't be so bad if this problem solving session was consistently reproducible. But every time I create a new session and give the problem, often times the naive solution appears. Sometimes the LLM can't even produce a working naive solution - even though the comments in the proposed source code shows the LLM knows what the correct answer is - the actual source code can't produce that answer. Once, I decided to give some hints to the LLM to see if I could get it to jump the chasm. The LLM decided to limit its search space to a certain range - which I knew was wrong, and resulted in incorrect results. I prompted the LLM to try and expand the search range, which it cheerfully tried and deduced an expanded search range - which was still too limited. Further prodding resulted in the LLM reverting back to the original search range. No amount of prodding would get the LLM to expand its search range. In a fit of rage, I told the LLM to use an arbitrary large search range which would cover the necessary search space. The LLM refused! Dang - I was chatting with HAL from 2001 - A Space Odyssey. I have been unable to repro that particular chat situation. I guess we should be happy that the LLM can even produce a working solution - probably 5-10 years ago, we'd be gobsmacked. Even call it magic... But it's not remotely consistent.