4 ms·

> Because eventually we’ll iron out all the bugs so the AIs will get more reliable over time Honestly this feels like a true statement to me. It's obviously a

> Because eventually we’ll iron out all the bugs so the AIs will get more reliable over time Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.

piyh 1y ago
Emergent misalignment and power seeking isn't a bug we can squash with a PR and a unit test
criddell 1y ago
Right away my mind went to "well, are people more reliable than they used to be?" and I'm not sure they are. Of course LLMs aren't people, but an AGI might behave like a person.
- adastra22 1y ago
  Older people are generally more reliable than younger people.
  - saulpw 1y ago
    I'm not sure that's generally true. However, older people have a track record, and a reliable older person is likely to be more reliable than a younger person without such a track record.
    - adastra22 1y ago
      Reliable has different meanings. I think in this case the meaning is closer to "deterministic" and "follows instructions." An older worker will more reliably behave the same way twice, and more reliably follow the same set of instructions they've been following throughout their career.
  - saghm 1y ago
    On the other hand, within 120 years (and sometimes decades earlier), every human so far also has reached reach a point where they can't do anything anymore, reliably or otherwise. I don't think humans provide much of a precedent of sustained growth in reliable over anything other than a fixed period, and it's not clear to me that we have any idea whether or not we'll reach a ceiling in how reliable we can make LLMs before they outperform humans in tasks like programming.
- Yoric 1y ago
  By the time a junior dev graduates to senior, I expect that they'll be more reliable. In fact, at the end of each project, I expect the junior dev to have grown more reliable. LLMs don't learn from a project. At best, you learn how to better use the LLM. They do have other benefits, of course, i.e. once you have trained one generation of Claude, you have as many instances as you need, something that isn't true with human beings. Whether that makes up for the lack of quality is an open question, which presumably depends on the projects.
  - tkgally 1y ago
    > LLMs don't learn from a project. How long do you think that will remain true? I've bootstrapped some workflows with Claude Code where it writes a markdown file at the end of each session for its own reference in later sessions. It worked pretty well. I assume other people are developing similar memory systems that will be more useful and robust than anything I could hack together.
    - intended 1y ago
      Permanently. There’s an interplay between two different ideas of reliability here. LLMs can only provide output which is somehow within training boundaries. We can get better at expanding the area within these boundaries. It will still not be reliable like code is.
    - Yoric 1y ago
      For LLMs? Mostly permanently. This is a limitation of the architecture. Yes, there are workarounds, including ChatGPT's "memory" or your technique (which I believe are mostly equivalent), but they are limited, slow and expensive. Many of the inventors of LLMs have moved on to (what they believe are) better models that would handle such learnings much better. I guess we'll see in 10-20 years if they have succeeded.
CobrastanJorji 1y ago
They have certainly gotten better, but it seems to me like the growth will be kind of logarithmic. I'd expect them to keep getting better quickly for a few more years and then kinda slow and eventually flatline as we reach the maximum for this sort of pattern matching kind of ML. And I expect that flat line will be well below the threshold needed for, say, a small software company to not require a programmer.
- Terr_ 1y ago
  > kind of logarithmic https://en.wikipedia.org/wiki/Sigmoid_function https://en.wikipedia.org/wiki/Sigmoid_function
  - CobrastanJorji 1y ago
    Ironically, yes. :)
- Legend2440 1y ago
  Ahaha, of course nothing will ever be able to do my job!
  - troupo 1y ago
    Since "AGI has been achieved internally" tweet I've only seen incremental improvements that are guaranteed to never be able to do my job. Or most people's jobs.
  - sayamqazi 1y ago
    It will take at least a few more decades at least from the looks of it. I would be 6 feet under by then so yes "Nothing will ever be able to do my job".
scuff3d 1y ago
Ironically I came to the comments to point out that all over hackernews you see this sentiment repeated, and that's by a group I would consider to be far more technically competent then your average person. And very helpfully there is one just a few comments down from the top.
- smokel 1y ago
  Technical competence and an interest in sociological development do not always coincide. Technology often seeks simplicity, whereas sociology examines inherently complex human behavior.
throw-10-13 1y ago
I read this as "10x better at generating code that looks correct but hides nasty bugs behind a facade of sane looking slop".
beyarkay 1y ago
I'll totally grant that thing will get better over time. But a point that I was (mostly failing) to make, is that software is discrete, NNs are continuous. No matter how buggy your program is, it's got a countable number of lines, so you can have some notion of "all the bugs" (ignoring distributed systems). But NNs are fundamentally continuous, I don't think it even makes sense to "count" bugs. You can have a list of prompts to which the model gives unwanted output, but it's a completely different ball game compared to regular software.

piyh 1y ago

Emergent misalignment and power seeking isn't a bug we can squash with a PR and a unit test

criddell 1y ago

Right away my mind went to "well, are people more reliable than they used to be?" and I'm not sure they are. Of course LLMs aren't people, but an AGI might behave like a person.

adastra22 1y ago
Older people are generally more reliable than younger people.
- saulpw 1y ago
  I'm not sure that's generally true. However, older people have a track record, and a reliable older person is likely to be more reliable than a younger person without such a track record.
  - adastra22 1y ago
    Reliable has different meanings. I think in this case the meaning is closer to "deterministic" and "follows instructions." An older worker will more reliably behave the same way twice, and more reliably follow the same set of instructions they've been following throughout their career.
- saghm 1y ago
  On the other hand, within 120 years (and sometimes decades earlier), every human so far also has reached reach a point where they can't do anything anymore, reliably or otherwise. I don't think humans provide much of a precedent of sustained growth in reliable over anything other than a fixed period, and it's not clear to me that we have any idea whether or not we'll reach a ceiling in how reliable we can make LLMs before they outperform humans in tasks like programming.
Yoric 1y ago
By the time a junior dev graduates to senior, I expect that they'll be more reliable. In fact, at the end of each project, I expect the junior dev to have grown more reliable. LLMs don't learn from a project. At best, you learn how to better use the LLM. They do have other benefits, of course, i.e. once you have trained one generation of Claude, you have as many instances as you need, something that isn't true with human beings. Whether that makes up for the lack of quality is an open question, which presumably depends on the projects.
- tkgally 1y ago
  > LLMs don't learn from a project. How long do you think that will remain true? I've bootstrapped some workflows with Claude Code where it writes a markdown file at the end of each session for its own reference in later sessions. It worked pretty well. I assume other people are developing similar memory systems that will be more useful and robust than anything I could hack together.
  - intended 1y ago
    Permanently. There’s an interplay between two different ideas of reliability here. LLMs can only provide output which is somehow within training boundaries. We can get better at expanding the area within these boundaries. It will still not be reliable like code is.
  - Yoric 1y ago
    For LLMs? Mostly permanently. This is a limitation of the architecture. Yes, there are workarounds, including ChatGPT's "memory" or your technique (which I believe are mostly equivalent), but they are limited, slow and expensive. Many of the inventors of LLMs have moved on to (what they believe are) better models that would handle such learnings much better. I guess we'll see in 10-20 years if they have succeeded.

CobrastanJorji 1y ago

They have certainly gotten better, but it seems to me like the growth will be kind of logarithmic. I'd expect them to keep getting better quickly for a few more years and then kinda slow and eventually flatline as we reach the maximum for this sort of pattern matching kind of ML. And I expect that flat line will be well below the threshold needed for, say, a small software company to not require a programmer.

Terr_ 1y ago
> kind of logarithmic https://en.wikipedia.org/wiki/Sigmoid_function https://en.wikipedia.org/wiki/Sigmoid_function
- CobrastanJorji 1y ago
  Ironically, yes. :)
Legend2440 1y ago
Ahaha, of course nothing will ever be able to do my job!
- troupo 1y ago
  Since "AGI has been achieved internally" tweet I've only seen incremental improvements that are guaranteed to never be able to do my job. Or most people's jobs.
- sayamqazi 1y ago
  It will take at least a few more decades at least from the looks of it. I would be 6 feet under by then so yes "Nothing will ever be able to do my job".

scuff3d 1y ago

Ironically I came to the comments to point out that all over hackernews you see this sentiment repeated, and that's by a group I would consider to be far more technically competent then your average person. And very helpfully there is one just a few comments down from the top.

smokel 1y ago
Technical competence and an interest in sociological development do not always coincide. Technology often seeks simplicity, whereas sociology examines inherently complex human behavior.

throw-10-13 1y ago

I read this as "10x better at generating code that looks correct but hides nasty bugs behind a facade of sane looking slop".

beyarkay 1y ago

I'll totally grant that thing will get better over time. But a point that I was (mostly failing) to make, is that software is discrete, NNs are continuous. No matter how buggy your program is, it's got a countable number of lines, so you can have some notion of "all the bugs" (ignoring distributed systems). But NNs are fundamentally continuous, I don't think it even makes sense to "count" bugs. You can have a list of prompts to which the model gives unwanted output, but it's a completely different ball game compared to regular software.