3 ms·

You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to

by jhugo 3y ago

You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. Many programmers also do work that doesn’t just involve stitching APIs together; GPT consistently fails badly when forced to reason, making it useless for many tasks. For example, ask it to implement a common algorithm like SHA - it will do a decent job. Then ask it to do it with an arbitrary limitation which no implementation it was trained on would have (e.g. use no integer type wider than 8 bits). In my experience, it cannot achieve this task with a correct result, even with significant hand-holding.

corethree 3y ago
>You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. I meant 38% aka 40% made a mistake on my math. It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. I understand where you're coming from though. Many products made by entrepreneurs follow that model where the remaining percentage is always harder than the beginning green fielded tech that was built. But can we honestly say that this is what AGI will be like? The LLM was an unexpected jump forward by an unprecedented amount. It could be we fill the gap to 100% with another such jump.
- __loam 3y ago
  > It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. And it's wrong, as anyone with experience in ML will tell you. 60% is easy. 70-85% is not too bad. 95% is hard. 100% is effectively impossible. This is the core problem of ML systems. They're good enough for 90% of cases but that last 10% can be incredibly important, to the point that the systems become unusable.
  - corethree 3y ago
    The topic at hand is the pace of AI technology. This one is hard to quantify as it's been mostly linear in terms of technological progress. Example: From gans to LLMs. That progress has not been diminishing. You seem to think the topic is about the current state of training algorithms and how it's hard to bring the model to 100 percent. That's a different topic with a subtle distinction.

by jhugo 3y ago

corethree 3y ago

>You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. I meant 38% aka 40% made a mistake on my math. It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. I understand where you're coming from though. Many products made by entrepreneurs follow that model where the remaining percentage is always harder than the beginning green fielded tech that was built. But can we honestly say that this is what AGI will be like? The LLM was an unexpected jump forward by an unprecedented amount. It could be we fill the gap to 100% with another such jump.

__loam 3y ago
> It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption. And it's wrong, as anyone with experience in ML will tell you. 60% is easy. 70-85% is not too bad. 95% is hard. 100% is effectively impossible. This is the core problem of ML systems. They're good enough for 90% of cases but that last 10% can be incredibly important, to the point that the systems become unusable.
- corethree 3y ago
  The topic at hand is the pace of AI technology. This one is hard to quantify as it's been mostly linear in terms of technological progress. Example: From gans to LLMs. That progress has not been diminishing. You seem to think the topic is about the current state of training algorithms and how it's hard to bring the model to 100 percent. That's a different topic with a subtle distinction.