6 ms·

The over-hype is causing it to look under-hyped. 48% of the code being freaking correct from 0% about 5 years ago is huge. With everyone and all the news talki

The over-hype is causing it to look under-hyped. 48% of the code being freaking correct from 0% about 5 years ago is huge. With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes. This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side. It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity. People start thinking it's not cool anymore. That's the bias you are suffering from.

tofukant 3y ago
100-62=38%
- albedoa 3y ago
  [flagged]
  - corethree 3y ago
    Oh shut up. Don't be a dick.
- corethree 3y ago
  dumb. I admit the mistake.
deleted 3y ago
[deleted]
Blahah 3y ago
100% - 62% = 38%
- corethree 3y ago
  my mistake.
Panzer04 3y ago
Except the eternal problem with all "AI" so far is edge cases. A product that is 90% correct may as well be useless, honestly.
- corethree 3y ago
  Well then we are part of the way their still right? It works but only edge cases are a problem. Next step is to fix the edge cases.
- jki275 3y ago
  I wouldn't argue useless, I'd argue it's a product that still needs an expert user to validate.
wpietri 3y ago
> Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side. What this study looked at was feeding StackOverflow questions to LLMs and then looking at the quality of the code. If you think a programmer's job is just turning an english-language description of a function into isolated code that never gets modified, I don't know what to tell you. In my opinion, anybody like that should not have a job today, never mind the future. A proper professional programming job involved AGI-level understanding of human users and their needs as embedded in a social context. Plus the ability to create novel solutions. Plus the ability to use code as a collaborative medium to make a code base that is sustainable over the long term by their colleagues. LLMs are not even 1% of the way to replacing professional developers.
- corethree 3y ago
  Not true. They are well past 1%. That's not looking at reality. You can lower it below 38% but 1% is pure fantasy.
  - wpietri 3y ago
    First off, you did not address most of my point. To answer your complaint, I am absolutely looking at reality. Please point us to an actual real-world professional programming job, matching the criteria I list above, for which a current LLM can economically replace a human for over a 5-year period. My believe is that there are exactly zero such jobs. If your claim is that we're at least at 1%, then you're claiming that there are at least 269k fully automatable programmer jobs [1]. It shouldn't be hard for you to find at least one to start. [1] Wikipedia says there are an estimated 26.9m professional programmers: https://en.wikipedia.org/wiki/Software_engineering_demographics https://en.wikipedia.org/wiki/Software_engineering_demograph...
    - corethree 3y ago
      >My believe is that there are exactly zero such jobs. If your claim is that we're at least at 1%, then you're claiming that there are at least 269k fully automatable programmer jobs [1]. It shouldn't be hard for you to find at least one to start. If I built an entire car but I am missing the key. I cannot drive the car but the car is 99% complete. It's just missing the key. So what happens is, it can replace 0% of human locomotion but it's 99% complete? get it? Just because the tool can't be used doesn't mean it's zero percent of the way to being drive able. Same with your job. The LLM can't replace any job yet, but it doesn't mean it's 1% of the way there. But you see what I just explained to you is obvious. You already know this just like I and every one else on the face of the earth already knows this fact. You're taking the discussion into a play on words. What does 1% apply to? Rather then use common sense and derive what I mean you prefer to redirect the conversation into a very specific definition of 1% that serves your own purpose. We can play this game all day. We can discuss whether your application of 1% is more fitting than my application of 1%. What a waste of everyone's time. I think it's better if rather then playing these games to "win" discussions, use your common sense to move the discussion past these games. Otherwise we're going to be talking about obvious things all day and getting all worked up about personal definitions. Clearly the LLM is more than just 1%.
raincole 3y ago
38%. And 38% is not "a third way to 100%". I'm sure you know what diminishing return is.
- corethree 3y ago
  my mistake. 40% of the way there, point still stands.
  - raincole 3y ago
    > point still stands No? I assume you actually don't know the concept of diminishing return, and it's totally fine. Every other comment here is explaining it to you already. I won't bother to repeat. Please read the sibling comments carefully.
    - corethree 3y ago
      "Hey do you know what exponential growth is?" You think I just say that sentence and suddenly all your arguments are suddenly flushed down the toilet because I stated some random concept out of nowhere? Come on. You need evidence. You brought up this question of diminishing returns out of nowhere and offer no evidence for it. So why would my point not stand? I read the sibling comments. One person brought it up but offered no proof that this is what's happening. Simply did what you did in a manner that was less rude, just stated the concept of diminishing returns applies to LLMs without offering a shred of evidence that indicates this is what is happening. There is a difference between introducing the concept of diminishing returns out of nowhere, and providing evidence that diminishing returns is WHAT is actually HAPPENING with LLMS. We're about a year out from the introduction of chatgpt, that's not enough time to know if all the gains in the past decade of AI has suddenly hit wall of diminishing returns.
jsunderland323 3y ago
> 48% if half way to 100% … It's halfway their to taking your job and your underwhelmed. You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. > It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity. I think applying the sociology of hipster band fans to LLMs is a mistake and I’m not sure how the vogueness of the technology correlates with the correctness of the actual models. Sometimes an early technology seems docile or useless at first but eventually reaches ubiquity and in retrospect the utility is obvious. But sometimes (more often than not) standard adoption curves don’t make sense to apply to a new technology because it isn’t useful enough to go through an adoption cycle. I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice. Furthermore given the massive amount of premature hype due to the AI zeitgeist, you can safely assume enough user hours have been spent messing with LLMs to get a verdict on their utility. Unlike the feeble technology that takes a longtime to reach a scale to know its utility, we don’t have to wait 10 years for 100m people to try LLMs, it happened in a week. My only point being, I think we should be apprehensive about trying to draw analogies of other adoption cycles that structurally are very very different to this one. Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree. Just this morning I had such a good use for an LLM that I was seriously considering building (and admittedly still am pondering) but the second I started to think through the LLM faultiness, I had to consider the complexity of the safe guards and weigh if it was really better to use GPT or just write a nasty regex script and constrain the problem. I’m leaning heavily towards the latter but I’d much prefer a silver bullet if it really killed vampires. Until then (if that day ever comes), it’s lead bullets for me.
- corethree 3y ago
  >You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. Nah even 100% reliability isn't attainable by a human. 100% obviously doesn't imply solving the halting problem. 100% as in 100% as reliable as a human. Even surpassing a human. Sort of in the same way as a computer now beats humans at chess. >I think applying the sociology of hipster band fans to LLMs is a mistake Why would it be a mistake? Human psychology is similar across all spectrums. What happens in one area is likely possible in another area. If it happens among hipster bands it can happen among hipster technology fads. >I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice. This is a valid speculation. But it's speculation. Basically your saying that LLMs are fundamentally broken and stuck at 38% forever because of fundamental and permanent flaws. The jury on that one is still out. And your point is highly, highly speculative. We see quantitative improvements on LLMs constantly AND this is a nascent technology we don't completely understand yet. The most probably and logical conclusion is to follow the technological trendline. That trendline is pointing up. To speculate on fundamental flaws of the LLM when people don't even fully understand what's going on with the LLM is illogical because you can't derive conclusions from something you don't understand. We can only generalize the trendline and constant improvements we've seen in AI for the past decade. Again that trendline is pointing to further break throughs in the future. >Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree. No this is not obvious to me. I disagree. I think some people are like you but other people, for example Geoffrey Hinton are in the apocalyptic camp. Personally I'm in the middle, I think it could go either way. It will definitely harm a segment of our society by taking over work, but whether the benefits of AGI outweighs the harm remains to be seen.
jki275 3y ago
Even if it were possible to get to 100% (it isn't), I'd still have a job. It's a great assistant. I use it. It's not taking anybody's job.
- corethree 3y ago
  It's taking your assistants job.
  - jki275 3y ago
    That's one contention, but I don't agree with that either.
jhugo 3y ago
You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. Many programmers also do work that doesn’t just involve stitching APIs together; GPT consistently fails badly when forced to reason, making it useless for many tasks. For example, ask it to implement a common algorithm like SHA - it will do a decent job. Then ask it to do it with an arbitrary limitation which no implementation it was trained on would have (e.g. use no integer type wider than 8 bits). In my experience, it cannot achieve this task with a correct result, even with significant hand-holding.
- corethree 3y ago
  >You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. I meant 38% aka 40% made a mistake on my math. It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. I understand where you're coming from though. Many products made by entrepreneurs follow that model where the remaining percentage is always harder than the beginning green fielded tech that was built. But can we honestly say that this is what AGI will be like? The LLM was an unexpected jump forward by an unprecedented amount. It could be we fill the gap to 100% with another such jump.
  - __loam 3y ago
    > It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. And it's wrong, as anyone with experience in ML will tell you. 60% is easy. 70-85% is not too bad. 95% is hard. 100% is effectively impossible. This is the core problem of ML systems. They're good enough for 90% of cases but that last 10% can be incredibly important, to the point that the systems become unusable.
    - corethree 3y ago
__loam 3y ago
I think you're misunderstanding the problem here. It could be 97% correct and it would still be unusable. Getting things to 100% is so hard it might as well be impossible, and the fact that you always need a person to check all the output limits the utility.

tofukant 3y ago

100-62=38%

albedoa 3y ago
[flagged]
- corethree 3y ago
  Oh shut up. Don't be a dick.
corethree 3y ago
dumb. I admit the mistake.

deleted 3y ago

[deleted]

Blahah 3y ago

100% - 62% = 38%

corethree 3y ago
my mistake.

Panzer04 3y ago

Except the eternal problem with all "AI" so far is edge cases. A product that is 90% correct may as well be useless, honestly.

corethree 3y ago
Well then we are part of the way their still right? It works but only edge cases are a problem. Next step is to fix the edge cases.
jki275 3y ago
I wouldn't argue useless, I'd argue it's a product that still needs an expert user to validate.

wpietri 3y ago

> Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side. What this study looked at was feeding StackOverflow questions to LLMs and then looking at the quality of the code. If you think a programmer's job is just turning an english-language description of a function into isolated code that never gets modified, I don't know what to tell you. In my opinion, anybody like that should not have a job today, never mind the future. A proper professional programming job involved AGI-level understanding of human users and their needs as embedded in a social context. Plus the ability to create novel solutions. Plus the ability to use code as a collaborative medium to make a code base that is sustainable over the long term by their colleagues. LLMs are not even 1% of the way to replacing professional developers.

corethree 3y ago
Not true. They are well past 1%. That's not looking at reality. You can lower it below 38% but 1% is pure fantasy.
- wpietri 3y ago
  First off, you did not address most of my point. To answer your complaint, I am absolutely looking at reality. Please point us to an actual real-world professional programming job, matching the criteria I list above, for which a current LLM can economically replace a human for over a 5-year period. My believe is that there are exactly zero such jobs. If your claim is that we're at least at 1%, then you're claiming that there are at least 269k fully automatable programmer jobs [1]. It shouldn't be hard for you to find at least one to start. [1] Wikipedia says there are an estimated 26.9m professional programmers: https://en.wikipedia.org/wiki/Software_engineering_demographics https://en.wikipedia.org/wiki/Software_engineering_demograph...
  - corethree 3y ago
    >My believe is that there are exactly zero such jobs. If your claim is that we're at least at 1%, then you're claiming that there are at least 269k fully automatable programmer jobs [1]. It shouldn't be hard for you to find at least one to start. If I built an entire car but I am missing the key. I cannot drive the car but the car is 99% complete. It's just missing the key. So what happens is, it can replace 0% of human locomotion but it's 99% complete? get it? Just because the tool can't be used doesn't mean it's zero percent of the way to being drive able. Same with your job. The LLM can't replace any job yet, but it doesn't mean it's 1% of the way there. But you see what I just explained to you is obvious. You already know this just like I and every one else on the face of the earth already knows this fact. You're taking the discussion into a play on words. What does 1% apply to? Rather then use common sense and derive what I mean you prefer to redirect the conversation into a very specific definition of 1% that serves your own purpose. We can play this game all day. We can discuss whether your application of 1% is more fitting than my application of 1%. What a waste of everyone's time. I think it's better if rather then playing these games to "win" discussions, use your common sense to move the discussion past these games. Otherwise we're going to be talking about obvious things all day and getting all worked up about personal definitions. Clearly the LLM is more than just 1%.

raincole 3y ago

38%. And 38% is not "a third way to 100%". I'm sure you know what diminishing return is.

corethree 3y ago
my mistake. 40% of the way there, point still stands.
- raincole 3y ago
  > point still stands No? I assume you actually don't know the concept of diminishing return, and it's totally fine. Every other comment here is explaining it to you already. I won't bother to repeat. Please read the sibling comments carefully.
  - corethree 3y ago
    "Hey do you know what exponential growth is?" You think I just say that sentence and suddenly all your arguments are suddenly flushed down the toilet because I stated some random concept out of nowhere? Come on. You need evidence. You brought up this question of diminishing returns out of nowhere and offer no evidence for it. So why would my point not stand? I read the sibling comments. One person brought it up but offered no proof that this is what's happening. Simply did what you did in a manner that was less rude, just stated the concept of diminishing returns applies to LLMs without offering a shred of evidence that indicates this is what is happening. There is a difference between introducing the concept of diminishing returns out of nowhere, and providing evidence that diminishing returns is WHAT is actually HAPPENING with LLMS. We're about a year out from the introduction of chatgpt, that's not enough time to know if all the gains in the past decade of AI has suddenly hit wall of diminishing returns.

jsunderland323 3y ago

> 48% if half way to 100% … It's halfway their to taking your job and your underwhelmed. You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. > It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity. I think applying the sociology of hipster band fans to LLMs is a mistake and I’m not sure how the vogueness of the technology correlates with the correctness of the actual models. Sometimes an early technology seems docile or useless at first but eventually reaches ubiquity and in retrospect the utility is obvious. But sometimes (more often than not) standard adoption curves don’t make sense to apply to a new technology because it isn’t useful enough to go through an adoption cycle. I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice. Furthermore given the massive amount of premature hype due to the AI zeitgeist, you can safely assume enough user hours have been spent messing with LLMs to get a verdict on their utility. Unlike the feeble technology that takes a longtime to reach a scale to know its utility, we don’t have to wait 10 years for 100m people to try LLMs, it happened in a week. My only point being, I think we should be apprehensive about trying to draw analogies of other adoption cycles that structurally are very very different to this one. Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree. Just this morning I had such a good use for an LLM that I was seriously considering building (and admittedly still am pondering) but the second I started to think through the LLM faultiness, I had to consider the complexity of the safe guards and weigh if it was really better to use GPT or just write a nasty regex script and constrain the problem. I’m leaning heavily towards the latter but I’d much prefer a silver bullet if it really killed vampires. Until then (if that day ever comes), it’s lead bullets for me.

corethree 3y ago
>You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. Nah even 100% reliability isn't attainable by a human. 100% obviously doesn't imply solving the halting problem. 100% as in 100% as reliable as a human. Even surpassing a human. Sort of in the same way as a computer now beats humans at chess. >I think applying the sociology of hipster band fans to LLMs is a mistake Why would it be a mistake? Human psychology is similar across all spectrums. What happens in one area is likely possible in another area. If it happens among hipster bands it can happen among hipster technology fads. >I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice. This is a valid speculation. But it's speculation. Basically your saying that LLMs are fundamentally broken and stuck at 38% forever because of fundamental and permanent flaws. The jury on that one is still out. And your point is highly, highly speculative. We see quantitative improvements on LLMs constantly AND this is a nascent technology we don't completely understand yet. The most probably and logical conclusion is to follow the technological trendline. That trendline is pointing up. To speculate on fundamental flaws of the LLM when people don't even fully understand what's going on with the LLM is illogical because you can't derive conclusions from something you don't understand. We can only generalize the trendline and constant improvements we've seen in AI for the past decade. Again that trendline is pointing to further break throughs in the future. >Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree. No this is not obvious to me. I disagree. I think some people are like you but other people, for example Geoffrey Hinton are in the apocalyptic camp. Personally I'm in the middle, I think it could go either way. It will definitely harm a segment of our society by taking over work, but whether the benefits of AGI outweighs the harm remains to be seen.

jki275 3y ago

Even if it were possible to get to 100% (it isn't), I'd still have a job. It's a great assistant. I use it. It's not taking anybody's job.

corethree 3y ago
It's taking your assistants job.
- jki275 3y ago
  That's one contention, but I don't agree with that either.

jhugo 3y ago

You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. Many programmers also do work that doesn’t just involve stitching APIs together; GPT consistently fails badly when forced to reason, making it useless for many tasks. For example, ask it to implement a common algorithm like SHA - it will do a decent job. Then ask it to do it with an arbitrary limitation which no implementation it was trained on would have (e.g. use no integer type wider than 8 bits). In my experience, it cannot achieve this task with a correct result, even with significant hand-holding.

corethree 3y ago
>You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. I meant 38% aka 40% made a mistake on my math. It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. I understand where you're coming from though. Many products made by entrepreneurs follow that model where the remaining percentage is always harder than the beginning green fielded tech that was built. But can we honestly say that this is what AGI will be like? The LLM was an unexpected jump forward by an unprecedented amount. It could be we fill the gap to 100% with another such jump.
- __loam 3y ago
  > It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. And it's wrong, as anyone with experience in ML will tell you. 60% is easy. 70-85% is not too bad. 95% is hard. 100% is effectively impossible. This is the core problem of ML systems. They're good enough for 90% of cases but that last 10% can be incredibly important, to the point that the systems become unusable.
  - corethree 3y ago

__loam 3y ago

I think you're misunderstanding the problem here. It could be 97% correct and it would still be unusable. Getting things to 100% is so hard it might as well be impossible, and the fact that you always need a person to check all the output limits the utility.