3 ms·
I don't know anything about the movement, so can't comment on the earlier points. >If AGI has even a 1% chance of happening this century and those donations re
by simonh 4y ago
I don't know anything about the movement, so can't comment on the earlier points.
>If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent.
Ive seen this argument many times on many different issues and I think it's almost always bunk. The classic example in Fusion research, I had friends when I was a student in the 80s arguing that if we only doubled fusion research spending we'd have unlimited cheap electricity by the year 2000, or others said that AI was the most important thing we could be researching at that point.
The problem is if you're too early on in the development process of a technology, you just flat out don't know enough about it to be able to make informed decisions. That early phase is about exploring avenues of research and figuring out what works and what doesn't, and that takes an irreducible amount of human time, diligence and hard work. Yes you need to invest in that, but above a certain threshold anything more you invest is almost entirely going to be wasted.
With general, or strong AI we only just barely have the beginnings of an idea of how to even do preliminary research on it. We're groping around in the dark for a starting point. It's far, far too early to start worrying about controls and management strategies for the end point of that research. I think there's very little chance much of that is likely to turn out to be relevant, and there's nothing to stop us focusing on that area once we do know what is going to be relevant.
I'm not saying ignore the issue, or don't think about it at all, but the idea that it's some urgent issue that we absolutely have to invest heavily in right now IMHO doesn't stand.
- edanm 4y agoYou might totally be right. The problem is, humanity is pretty bad at estimating when a new technology is about to be ready. Even the experts are pretty bad. See e.g. multiple people saying human flight is many years away, including some who were saying it unknowingly after it had been achieved! So I think the question is: when should we worry about ai safety? How early is too early? Plenty of experts think now is a good time to be spendning at least some resources on this. And we're not talking nuclear fusion level resources here; I don't know the numbers, but I imagine the amount of resources are actually fairly tiny compared to almost all other scientific spending. (And this was certainly true 10 years ago.) I personally think that, given the chance of AI systems being an existential risk, we should be putting way more resources into it than into other things. The same goes true for preventing pandemics or other biological x-risks, etc.
- bootsmann 4y ago> Plenty of experts think now is a good time to be spendning at least some resources on this. And we're not talking nuclear fusion level resources here; I don't know the numbers, but I imagine the amount of resources are actually fairly tiny compared to almost all other scientific spending. (And this was certainly true 10 years ago.) The risk theses experts are talking about isn't existential. Its stuff like GPT3 being truncated to generate propaganda for dictatorships, deepfakes, algorithm-reinforced biases in all kinds of models (because the data used to train them was generated by humans). This is concerning to some degree and deserves funding and research (though all too often the results boil down to unusable advice and guidelines). This is entirely orthogonal to any existential risk a supposed AGI might create. One is a testable scientific hypothesis and the other is a futurist horror story which is untestable by definition.
- naasking 4y ago> This is entirely orthogonal to any existential risk a supposed AGI might create. One is a testable scientific hypothesis and the other is a futurist horror story which is untestable by definition. There is no difference in their testability, it's just that the existential risk test means no more humans to care about the results. The danger of a biolab creating a pathogen that wipes out all humanity is real but remote, and is a simple extension of the argument that the lab could create pathogens that can "only" kill or harm millions. That's why biolab safety is carefully studied, and the same argument applies to AI safety.
- naasking 4y ago> With general, or strong AI we only just barely have the beginnings of an idea of how to even do preliminary research on it. Maybe. Or maybe we're just on small change away from AGI. We could be, because we don't yet really know what's required and we may not ever know until after we've done it. That prospect should be terrifying. > It's far, far too early to start worrying about controls and management strategies for the end point of that research. No, because if you create AGI before we have some containment or safety protocols, then it may already be too late. Consider whether we should have researched biolab safety before we started experimenting on modifying pathogens in various ways. Doesn't that just seem blindingly brain-dead obvious? For some reason this clear and obvious conclusion just goes out the window when it comes to AI.