4 ms·

No, then you start worrying when the technology is there. There is no point in worrying about it now when it is fiction. If AGI/ASI does not arise in some kind

by golol 3y ago

No, then you start worrying when the technology is there. There is no point in worrying about it now when it is fiction. If AGI/ASI does not arise in some kind of uncontrollable "foom", you can always examine what you are dealing with right now and act accordingly. If AGI is achievable in a 5 years takeoff scenario then we have plenty of time and steps along the way fo study what we have and examine its danger. Right now we do not see AGI on the horizon and there is no significant danger from our models, so no reason to worry yet. If it happens, then long before it is "too late to stop it" we will see that GPT-6 or whatever is quite good and potentially dangepurs at certain tasks. Then you can go ahead and regulate. There is no point in doing it now. On a sidenote, I don't really believe we can figure out anything in alignment anyways without having access to the models of corresponding capability, so waiting 5 years for alignment to catch up will so nothing.

theptip 3y ago
If AGI is achievable in 5 years, then we do need to worry now, because we currently have no idea how to solve problems like prompt hijacking or how to durably convey our values to these systems. > There is no point in worrying about it now when it is fiction. I find this sort of epistemic nihilism completely baffling. Just because we have wide error bars, doesn’t mean we have to give up on trying to steer the future. If your 95% confidence window for “path A” includes ruin, and “path B” has lower EV but no ruin, why would you pick A? The problem with alignment right now is we can’t even align GPT-3.5, let alone GPT-4. I believe this is a tech tree. New things building on old things at each generation. If we can’t keep up with the frontier, how will we align GPT-Omega? Concretely, things like Mechanistic Interpretability seem required. If we could pause progress for 5-10 years I feel optimistic that we could fully understand GPT-4 and how it actually renders concepts and decisions. (I don’t think we can pause progress, this is my case for why we should be at least a bit worried, and invest much more in interpretability research.)
- golol 3y ago
  It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please show me a single result of alignment research that has real world applicability and is not just a variant of "train your model on aligned data so it is aligned". And the latter form of alignment success, RLHF and instruction finetuning, came after the base model capability was unlocked. If you are doing any sort of challenging intellectual endeavour you need one of two things in order not to be inevitably wrong: you either use rigor as in math or the scientific method, or you do engineering with real things that you can test and play around with. If neither of these two are giben then your field is likely to produce decades worth of junk results and predictions such as it happens in economics, sociology, psychology, and especially philosophy. You could fund a billion dollar 10 year program with economists in order to prevent the next big market crisis and they would not produce any valuable work. I see AGI alignment research as being kind of in this position: Rigorous mathematical results are impossible or impractical (theoretical results about perfect actors maximizing utility functions or sth are unlikely to apply on the real world), and if you do not have AI models of that capability level available, you can also not engineer, test, and play around with your ideas. Hence I lots and lots of papers and theories will be produced and eventually the real world will show what was correct, but no valuable predictions will have been obtained.
  - theptip 3y ago
    > an attempt to regulate and control a technology that doesn't exist yet. I’m not even proposing regulation. Just trying to understand the technology. You say it doesn’t exist yet, but many AI researchers (including Sutskever and Hinton, pioneers that know as much about the state of the art as anyone alive) think AGI could be achieved by scaling up the current architectures. If there is a non-zero probability of AGI built atop transformers, we must invest more in understanding how transformers actually work. Even if it’s not that likely, it’s worth spending a lot more than we do now as a hedge. And I believe that if transformers do not end up being anything to do with AGI, it is likely that we will still carry over some expertise from catching up and understanding them. For papers that are actually building our understanding here, see https://transformer-circuits.pub/ https://transformer-circuits.pub/. For example I think “Towards Monosemanticity”[1] was fairly widely considered to be a big step forward in the field. There have also been meaningful results around steering models, which again relies upon being able to decode what features are actually encoding specific concepts. I’m here for the idea that most papers are bunk, but isn’t that science in general? Really hard to pick good projects and researchers to invest in ahead of time, and so you kind of have to spray and pray at some level. Still worth it on net. [1]: https://transformer-circuits.pub/2023/monosemantic-features/index.html https://transformer-circuits.pub/2023/monosemantic-features/...
  - calf 3y ago
    What about European regulations about genetic experimentation and GMO technology? The experiments in the broadest sense literally do not exist. They are not supposed to. I mean, the GMO regulation papers are always referring to "nascent" technology. So existence is itself a dichotomous term for vague future capabilities. So why can't people argue that alignment research is just a subset of the precautionary principle, which has already been applied to other such domains, at least in Europe? Precaution implies an absence of knowledge, but still having the agency for society and institutions to do something about it.
pixl97 3y ago
"Don't worry about country X's nuclear research program, they'll never make a bomb" I do hope you're not in charge of any risk management.

by golol 3y ago

theptip 3y ago

If AGI is achievable in 5 years, then we do need to worry now, because we currently have no idea how to solve problems like prompt hijacking or how to durably convey our values to these systems. > There is no point in worrying about it now when it is fiction. I find this sort of epistemic nihilism completely baffling. Just because we have wide error bars, doesn’t mean we have to give up on trying to steer the future. If your 95% confidence window for “path A” includes ruin, and “path B” has lower EV but no ruin, why would you pick A? The problem with alignment right now is we can’t even align GPT-3.5, let alone GPT-4. I believe this is a tech tree. New things building on old things at each generation. If we can’t keep up with the frontier, how will we align GPT-Omega? Concretely, things like Mechanistic Interpretability seem required. If we could pause progress for 5-10 years I feel optimistic that we could fully understand GPT-4 and how it actually renders concepts and decisions. (I don’t think we can pause progress, this is my case for why we should be at least a bit worried, and invest much more in interpretability research.)

golol 3y ago
It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please show me a single result of alignment research that has real world applicability and is not just a variant of "train your model on aligned data so it is aligned". And the latter form of alignment success, RLHF and instruction finetuning, came after the base model capability was unlocked. If you are doing any sort of challenging intellectual endeavour you need one of two things in order not to be inevitably wrong: you either use rigor as in math or the scientific method, or you do engineering with real things that you can test and play around with. If neither of these two are giben then your field is likely to produce decades worth of junk results and predictions such as it happens in economics, sociology, psychology, and especially philosophy. You could fund a billion dollar 10 year program with economists in order to prevent the next big market crisis and they would not produce any valuable work. I see AGI alignment research as being kind of in this position: Rigorous mathematical results are impossible or impractical (theoretical results about perfect actors maximizing utility functions or sth are unlikely to apply on the real world), and if you do not have AI models of that capability level available, you can also not engineer, test, and play around with your ideas. Hence I lots and lots of papers and theories will be produced and eventually the real world will show what was correct, but no valuable predictions will have been obtained.
- theptip 3y ago
  > an attempt to regulate and control a technology that doesn't exist yet. I’m not even proposing regulation. Just trying to understand the technology. You say it doesn’t exist yet, but many AI researchers (including Sutskever and Hinton, pioneers that know as much about the state of the art as anyone alive) think AGI could be achieved by scaling up the current architectures. If there is a non-zero probability of AGI built atop transformers, we must invest more in understanding how transformers actually work. Even if it’s not that likely, it’s worth spending a lot more than we do now as a hedge. And I believe that if transformers do not end up being anything to do with AGI, it is likely that we will still carry over some expertise from catching up and understanding them. For papers that are actually building our understanding here, see https://transformer-circuits.pub/ https://transformer-circuits.pub/. For example I think “Towards Monosemanticity”[1] was fairly widely considered to be a big step forward in the field. There have also been meaningful results around steering models, which again relies upon being able to decode what features are actually encoding specific concepts. I’m here for the idea that most papers are bunk, but isn’t that science in general? Really hard to pick good projects and researchers to invest in ahead of time, and so you kind of have to spray and pray at some level. Still worth it on net. [1]: https://transformer-circuits.pub/2023/monosemantic-features/index.html https://transformer-circuits.pub/2023/monosemantic-features/...
- calf 3y ago
  What about European regulations about genetic experimentation and GMO technology? The experiments in the broadest sense literally do not exist. They are not supposed to. I mean, the GMO regulation papers are always referring to "nascent" technology. So existence is itself a dichotomous term for vague future capabilities. So why can't people argue that alignment research is just a subset of the precautionary principle, which has already been applied to other such domains, at least in Europe? Precaution implies an absence of knowledge, but still having the agency for society and institutions to do something about it.

pixl97 3y ago

"Don't worry about country X's nuclear research program, they'll never make a bomb" I do hope you're not in charge of any risk management.