4 ms·
It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please sho
by golol 3y ago
It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please show me a single result of alignment research that has real world applicability and is not just a variant of "train your model on aligned data so it is aligned". And the latter form of alignment success, RLHF and instruction finetuning, came after the base model capability was unlocked.
If you are doing any sort of challenging intellectual endeavour you need one of two things in order not to be inevitably wrong: you either use rigor as in math or the scientific method, or you do engineering with real things that you can test and play around with.
If neither of these two are giben then your field is likely to produce decades worth of junk results and predictions such as it happens in economics, sociology, psychology, and especially philosophy.
You could fund a billion dollar 10 year program with economists in order to prevent the next big market crisis and they would not produce any valuable work.
I see AGI alignment research as being kind of in this position: Rigorous mathematical results are impossible or impractical (theoretical results about perfect actors maximizing utility functions or sth are unlikely to apply on the real world), and if you do not have AI models of that capability level available, you can also not engineer, test, and play around with your ideas. Hence I lots and lots of papers and theories will be produced and eventually the real world will show what was correct, but no valuable predictions will have been obtained.
- theptip 3y ago> an attempt to regulate and control a technology that doesn't exist yet. I’m not even proposing regulation. Just trying to understand the technology. You say it doesn’t exist yet, but many AI researchers (including Sutskever and Hinton, pioneers that know as much about the state of the art as anyone alive) think AGI could be achieved by scaling up the current architectures. If there is a non-zero probability of AGI built atop transformers, we must invest more in understanding how transformers actually work. Even if it’s not that likely, it’s worth spending a lot more than we do now as a hedge. And I believe that if transformers do not end up being anything to do with AGI, it is likely that we will still carry over some expertise from catching up and understanding them. For papers that are actually building our understanding here, see https://transformer-circuits.pub/ https://transformer-circuits.pub/. For example I think “Towards Monosemanticity”[1] was fairly widely considered to be a big step forward in the field. There have also been meaningful results around steering models, which again relies upon being able to decode what features are actually encoding specific concepts. I’m here for the idea that most papers are bunk, but isn’t that science in general? Really hard to pick good projects and researchers to invest in ahead of time, and so you kind of have to spray and pray at some level. Still worth it on net. [1]: https://transformer-circuits.pub/2023/monosemantic-features/index.html https://transformer-circuits.pub/2023/monosemantic-features/...
- calf 3y agoWhat about European regulations about genetic experimentation and GMO technology? The experiments in the broadest sense literally do not exist. They are not supposed to. I mean, the GMO regulation papers are always referring to "nascent" technology. So existence is itself a dichotomous term for vague future capabilities. So why can't people argue that alignment research is just a subset of the precautionary principle, which has already been applied to other such domains, at least in Europe? Precaution implies an absence of knowledge, but still having the agency for society and institutions to do something about it.