3 ms·

The thing I've never understood is how Leike's superalignment scheme could work. There's a high-level description of their research aims here: https://openai.co

by AlanYx 2y ago

The thing I've never understood is how Leike's superalignment scheme could work. There's a high-level description of their research aims here: https://openai.com/index/introducing-superalignment/ https://openai.com/index/introducing-superalignment/ Basically they want to train a model to evaluate the safety of other models, because "humans won’t be able to reliably supervise AI systems much smarter than us". But we know that memory-augmented LLMs (not conventional LLMs, but a minor extension that will likely become commonplace, or some variation thereof) are Turing complete, so being able to inspect a model and guarantee some property is equivalent to the halting problem. It's probably hard to justify a research team whose goal is solving the halting problem, which is provably undecidable in the general case.

sitkack 2y ago
What if you solve 100% of the problem over a smaller domain? It is important to disambiguate the scientific problem from the engineering problem. We can prove that lots of programs halt, not that all programs can halt. It isn't a reason to not do the research. Sounds like an armchair way to not try.
- AlanYx 2y ago
  Isn't it the case that you can only prove that a program halts if it can be reduced/transformed to a computational model (e.g., a model that is effectively calculable) that is less powerful than a Turing machine? Had they framed their research as "how can we design models that are limited enough that we can guarantee their safety" (rather than how can we design a powerful extrinsic inspector to supervise), that would have made total sense to me. But if you can do that, many of the motivations for superalignment don't exist. Put another way, implicit in the superalignment game is the idea that models that require superalignment are going to be at least as powerful as what's on the horizon, not some reduceable subset thereof.
  - 6gvONxR4sf7o 2y ago
    > Isn't it the case that you can only prove that a program halts if it can be reduced/transformed to a computational model (e.g., a model that is effectively calculable) that is less powerful than a Turing machine? Models less power than a turing machine are still incredibly powerful. Some useful general purpose languages only accept programs that terminate. Hell, every useful program I ever wrote could be (in principle) proved to halt (excepting a `while True: do useful terminating thing again` main loop, of course).
  - ndriscoll 2y ago
    In practice, most real world programs can be written in a way that only uses bounded recursion, so the halting problem doesn't really need to affect real-world programs. Or you can trivially ensure a program halts: add a timeout (and real-world programs like web servers do this all the time). I suppose the analogy for models is that if it can't decide whether output of another model is "safe", then it is not safe, so terminate. In practice, this could possibly be useful in the way a web server that times out after 1 second is still useful when its response times are normally measured in milliseconds. (Assuming of course that safety means anything in the first place)
deleted 2y ago
[deleted]

by AlanYx 2y ago

sitkack 2y ago

What if you solve 100% of the problem over a smaller domain? It is important to disambiguate the scientific problem from the engineering problem. We can prove that lots of programs halt, not that all programs can halt. It isn't a reason to not do the research. Sounds like an armchair way to not try.

AlanYx 2y ago
Isn't it the case that you can only prove that a program halts if it can be reduced/transformed to a computational model (e.g., a model that is effectively calculable) that is less powerful than a Turing machine? Had they framed their research as "how can we design models that are limited enough that we can guarantee their safety" (rather than how can we design a powerful extrinsic inspector to supervise), that would have made total sense to me. But if you can do that, many of the motivations for superalignment don't exist. Put another way, implicit in the superalignment game is the idea that models that require superalignment are going to be at least as powerful as what's on the horizon, not some reduceable subset thereof.
- 6gvONxR4sf7o 2y ago
  > Isn't it the case that you can only prove that a program halts if it can be reduced/transformed to a computational model (e.g., a model that is effectively calculable) that is less powerful than a Turing machine? Models less power than a turing machine are still incredibly powerful. Some useful general purpose languages only accept programs that terminate. Hell, every useful program I ever wrote could be (in principle) proved to halt (excepting a `while True: do useful terminating thing again` main loop, of course).
- ndriscoll 2y ago
  In practice, most real world programs can be written in a way that only uses bounded recursion, so the halting problem doesn't really need to affect real-world programs. Or you can trivially ensure a program halts: add a timeout (and real-world programs like web servers do this all the time). I suppose the analogy for models is that if it can't decide whether output of another model is "safe", then it is not safe, so terminate. In practice, this could possibly be useful in the way a web server that times out after 1 second is still useful when its response times are normally measured in milliseconds. (Assuming of course that safety means anything in the first place)

deleted 2y ago

[deleted]