3 ms·

Usually “mundane” vs “existential” harms/risks. > ASI/AGI safety is only a concern right now if you believe in foom False. There are plenty of fast takeoff sc

Usually “mundane” vs “existential” harms/risks. > ASI/AGI safety is only a concern right now if you believe in foom False. There are plenty of fast takeoff scenarios that look real bad that aren’t “foom”. For example if you think AGI might be achievable within 5 year right now, and alignment is 10 years away, then you are very worried and want to slow things down a little.

swatcoder 3y ago
If you're suggesting the emergence of an AI that can exponentially self-improve, coopt the resources to overwhelem human confrol, and magically build out the physical infrastructure sufficient to drive its globe-spanning humanity-squelching computational needs... that's a foom scenario. If something that some people call AGI emerges in the next five years, (a) it's designation is going to be under tremendous dispute, (b) the infrastructure to enable a nightmare scenario won't be anywhere near reality. Escaping that practical reality requires a complete ahistorical, ascientific faith in "yeah, but what if it's that capable?!?! The point is we can't know!!" that more strongly recalls history's secretive, sometimes dangerous, perennially incorrect apocolypse cults and alchemical societies than it does the diligent and reasoned scientists we might usually think of handing hundreds of billions of dollars.
- theptip 3y ago
  Foom specifically refers to a very fast takeoff to ASI, usually hours/days or at the outside weeks. It’s a term of art, well-defined. https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yudkowsky-on-doom-from-foom-2 https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yud... It does not refer to a fast takeoff where an AGI self-improves to ASI over multiple years. It’s also not referring to the AGI threshold. If foom happens it’s probably unambiguous; there is now an entity that is far more intelligent than humans. I think the foom scenarios are fairly unrealistic, for thermodynamic reasons. But I think it’s perfectly plausible that an ASI could persuade the company that built it to keep it secret while it acquired more resources and wealth over the course of years.
golol 3y ago
No, then you start worrying when the technology is there. There is no point in worrying about it now when it is fiction. If AGI/ASI does not arise in some kind of uncontrollable "foom", you can always examine what you are dealing with right now and act accordingly. If AGI is achievable in a 5 years takeoff scenario then we have plenty of time and steps along the way fo study what we have and examine its danger. Right now we do not see AGI on the horizon and there is no significant danger from our models, so no reason to worry yet. If it happens, then long before it is "too late to stop it" we will see that GPT-6 or whatever is quite good and potentially dangepurs at certain tasks. Then you can go ahead and regulate. There is no point in doing it now. On a sidenote, I don't really believe we can figure out anything in alignment anyways without having access to the models of corresponding capability, so waiting 5 years for alignment to catch up will so nothing.
- theptip 3y ago
  If AGI is achievable in 5 years, then we do need to worry now, because we currently have no idea how to solve problems like prompt hijacking or how to durably convey our values to these systems. > There is no point in worrying about it now when it is fiction. I find this sort of epistemic nihilism completely baffling. Just because we have wide error bars, doesn’t mean we have to give up on trying to steer the future. If your 95% confidence window for “path A” includes ruin, and “path B” has lower EV but no ruin, why would you pick A? The problem with alignment right now is we can’t even align GPT-3.5, let alone GPT-4. I believe this is a tech tree. New things building on old things at each generation. If we can’t keep up with the frontier, how will we align GPT-Omega? Concretely, things like Mechanistic Interpretability seem required. If we could pause progress for 5-10 years I feel optimistic that we could fully understand GPT-4 and how it actually renders concepts and decisions. (I don’t think we can pause progress, this is my case for why we should be at least a bit worried, and invest much more in interpretability research.)
  - golol 3y ago
    It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please show me a single result of alignment research that has real world applicability and is not just a variant of "train your model on aligned data so it is aligned". And the latter form of alignment success, RLHF and instruction finetuning, came after the base model capability was unlocked. If you are doing any sort of challenging intellectual endeavour you need one of two things in order not to be inevitably wrong: you either use rigor as in math or the scientific method, or you do engineering with real things that you can test and play around with. If neither of these two are giben then your field is likely to produce decades worth of junk results and predictions such as it happens in economics, sociology, psychology, and especially philosophy. You could fund a billion dollar 10 year program with economists in order to prevent the next big market crisis and they would not produce any valuable work. I see AGI alignment research as being kind of in this position: Rigorous mathematical results are impossible or impractical (theoretical results about perfect actors maximizing utility functions or sth are unlikely to apply on the real world), and if you do not have AI models of that capability level available, you can also not engineer, test, and play around with your ideas. Hence I lots and lots of papers and theories will be produced and eventually the real world will show what was correct, but no valuable predictions will have been obtained.

swatcoder 3y ago

If you're suggesting the emergence of an AI that can exponentially self-improve, coopt the resources to overwhelem human confrol, and magically build out the physical infrastructure sufficient to drive its globe-spanning humanity-squelching computational needs... that's a foom scenario. If something that some people call AGI emerges in the next five years, (a) it's designation is going to be under tremendous dispute, (b) the infrastructure to enable a nightmare scenario won't be anywhere near reality. Escaping that practical reality requires a complete ahistorical, ascientific faith in "yeah, but what if it's that capable?!?! The point is we can't know!!" that more strongly recalls history's secretive, sometimes dangerous, perennially incorrect apocolypse cults and alchemical societies than it does the diligent and reasoned scientists we might usually think of handing hundreds of billions of dollars.

theptip 3y ago
Foom specifically refers to a very fast takeoff to ASI, usually hours/days or at the outside weeks. It’s a term of art, well-defined. https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yudkowsky-on-doom-from-foom-2 https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yud... It does not refer to a fast takeoff where an AGI self-improves to ASI over multiple years. It’s also not referring to the AGI threshold. If foom happens it’s probably unambiguous; there is now an entity that is far more intelligent than humans. I think the foom scenarios are fairly unrealistic, for thermodynamic reasons. But I think it’s perfectly plausible that an ASI could persuade the company that built it to keep it secret while it acquired more resources and wealth over the course of years.

golol 3y ago

No, then you start worrying when the technology is there. There is no point in worrying about it now when it is fiction. If AGI/ASI does not arise in some kind of uncontrollable "foom", you can always examine what you are dealing with right now and act accordingly. If AGI is achievable in a 5 years takeoff scenario then we have plenty of time and steps along the way fo study what we have and examine its danger. Right now we do not see AGI on the horizon and there is no significant danger from our models, so no reason to worry yet. If it happens, then long before it is "too late to stop it" we will see that GPT-6 or whatever is quite good and potentially dangepurs at certain tasks. Then you can go ahead and regulate. There is no point in doing it now. On a sidenote, I don't really believe we can figure out anything in alignment anyways without having access to the models of corresponding capability, so waiting 5 years for alignment to catch up will so nothing.

theptip 3y ago
If AGI is achievable in 5 years, then we do need to worry now, because we currently have no idea how to solve problems like prompt hijacking or how to durably convey our values to these systems. > There is no point in worrying about it now when it is fiction. I find this sort of epistemic nihilism completely baffling. Just because we have wide error bars, doesn’t mean we have to give up on trying to steer the future. If your 95% confidence window for “path A” includes ruin, and “path B” has lower EV but no ruin, why would you pick A? The problem with alignment right now is we can’t even align GPT-3.5, let alone GPT-4. I believe this is a tech tree. New things building on old things at each generation. If we can’t keep up with the frontier, how will we align GPT-Omega? Concretely, things like Mechanistic Interpretability seem required. If we could pause progress for 5-10 years I feel optimistic that we could fully understand GPT-4 and how it actually renders concepts and decisions. (I don’t think we can pause progress, this is my case for why we should be at least a bit worried, and invest much more in interpretability research.)
- golol 3y ago
  It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please show me a single result of alignment research that has real world applicability and is not just a variant of "train your model on aligned data so it is aligned". And the latter form of alignment success, RLHF and instruction finetuning, came after the base model capability was unlocked. If you are doing any sort of challenging intellectual endeavour you need one of two things in order not to be inevitably wrong: you either use rigor as in math or the scientific method, or you do engineering with real things that you can test and play around with. If neither of these two are giben then your field is likely to produce decades worth of junk results and predictions such as it happens in economics, sociology, psychology, and especially philosophy. You could fund a billion dollar 10 year program with economists in order to prevent the next big market crisis and they would not produce any valuable work. I see AGI alignment research as being kind of in this position: Rigorous mathematical results are impossible or impractical (theoretical results about perfect actors maximizing utility functions or sth are unlikely to apply on the real world), and if you do not have AI models of that capability level available, you can also not engineer, test, and play around with your ideas. Hence I lots and lots of papers and theories will be produced and eventually the real world will show what was correct, but no valuable predictions will have been obtained.