4 ms·

There's not much of a world in which we don't check up and verify what humans are doing to some degree periodically. Non-deterministic behavior will never be tr

There's not much of a world in which we don't check up and verify what humans are doing to some degree periodically. Non-deterministic behavior will never be trusted by default, as it's simply not trustable. As machines become more non-deterministic, we're going to start feeling about them in similar ways we already feel about other such processes.

NitpickLawyer 1y ago
> Non-deterministic behavior will never be trusted by default, as it's simply not trustable. Never is a long time... If you have a task that is easily benchmarkable (i.e. matrix multiplication or algorithm speedup) you can totally "trust" that a system can non-deterministically work the problem until the results are "better" (speed, memory, etc).
- Sharlin 1y ago
  Proving the correctness of the “improvements” is another thing entirely, though.
  - NitpickLawyer 1y ago
    I agree. At first the problems that you try to solve need to be verifiable. But there's progress on many fronts on this. There's been increased interest in provers (natural language to lean for example). There's also been progress in LLM-as-a-judge on open-ish problems. And it seems that RL can help with extracting step rewards from sparse rewards domains.
jerf 1y ago
You will always get much, much, MUCH better performance from something that looks like assembler code than from having an LLM do everything. So I think the model of "AIs build something that looks recognizably like code" is going to continue indefinitely, and that code is generally going to be more deterministic than an AI will be. I'm not saying nothing will change. AIs may be constantly writing their own code for themselves internally in a much more fluid mixed environment, AIs may be writing into AI-specific languages built for their own quirks and preferences that make it harder for humans to follow than when AIs work in relatively human stacks, etc. I'm just saying, the concept of "code" that we could review is definitely going to stick around indefinitely, because the performance gains and reduction in resource usage are always going to be enormous. Even AIs that want to review AI work will want to review the generated and executing code, not the other AIs themselves. AIs will always be nondeterministic by their nature (because even if you run them in some deterministic mode, you will not be able to predict their exact results anyhow, which is in practice non-determinism), but non-AI code could conceivably actually get better and more deterministic, depending on how AI software engineering ethos develop.

NitpickLawyer 1y ago

> Non-deterministic behavior will never be trusted by default, as it's simply not trustable. Never is a long time... If you have a task that is easily benchmarkable (i.e. matrix multiplication or algorithm speedup) you can totally "trust" that a system can non-deterministically work the problem until the results are "better" (speed, memory, etc).

Sharlin 1y ago
Proving the correctness of the “improvements” is another thing entirely, though.
- NitpickLawyer 1y ago
  I agree. At first the problems that you try to solve need to be verifiable. But there's progress on many fronts on this. There's been increased interest in provers (natural language to lean for example). There's also been progress in LLM-as-a-judge on open-ish problems. And it seems that RL can help with extracting step rewards from sparse rewards domains.

jerf 1y ago

You will always get much, much, MUCH better performance from something that looks like assembler code than from having an LLM do everything. So I think the model of "AIs build something that looks recognizably like code" is going to continue indefinitely, and that code is generally going to be more deterministic than an AI will be. I'm not saying nothing will change. AIs may be constantly writing their own code for themselves internally in a much more fluid mixed environment, AIs may be writing into AI-specific languages built for their own quirks and preferences that make it harder for humans to follow than when AIs work in relatively human stacks, etc. I'm just saying, the concept of "code" that we could review is definitely going to stick around indefinitely, because the performance gains and reduction in resource usage are always going to be enormous. Even AIs that want to review AI work will want to review the generated and executing code, not the other AIs themselves. AIs will always be nondeterministic by their nature (because even if you run them in some deterministic mode, you will not be able to predict their exact results anyhow, which is in practice non-determinism), but non-AI code could conceivably actually get better and more deterministic, depending on how AI software engineering ethos develop.