3 ms·
at this point there's a respectable literature on the potential dangers of unconstrained, powerful optimizing agents, of which I linked a couple of examples. T
by AndrewKemendo 12y ago
at this point there's a respectable literature on the potential dangers of unconstrained, powerful optimizing agents, of which I linked a couple of examples.
There isn't really though. Superintelligence, while an interesting book, doesn't do much more than pontificate on sci-fi futures based largely on writing from Yudkowski. Bostrom gives no clear pathway to AGI. Neither does Yudkowski in his own writings or Barrat (Our final invention). By the way Super intelligence is kind of an offshoot book from Global Catastrophic Risks.
All of them take these interesting approaches, WBE, BCI, AGI etc... and just assume they will achieve the goal without looking realistically about where the field is or how it would get there.
So what they do is they say: We are here right now. In some possible fantasy world we could be there. The problem is they can't draw the dots between them.
For example, find me someone who can tell me what kind of hardware we need for an AGI. Can't do it, because no one has any idea. What about interface, what is the interface for an AGI?
Even better (this was my thesis project): What group would have a strong enough requirement that an AGI would be required to build? Industry, Academia, Military? Ok great, can they get funding or resources for it?
etc...
Note, I am not saying that there is no potential that AGI could be a problem. The point here is that nobody has firm footing to say that it will definitely be a huge risk and that we need to regulate it's development.
There are actually people calling for legislation to prevent/slow AGI development - see Altman's blog post and many of the writings on MIRI. So that is what I am rallying against.
We need more development into AGI not less.
- maaku 12y agoI'm more in your camp than MIRI's but... Could you argue against the position directly? You kinda took a swerve in the middle there. Bostrom, Yudkowsky, and Barrat's positions are basically "superintelligence is possible, it will eventually happen in some way, and when it does ..." It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence. So you would like to see unencumbered research in this area. They argue that the long term outcome of this research is existentially risky and therefore should be tightly regulated, and the sooner the better. Do you have any points regarding this particular difference of opinion?
- csallen 12y agoI'm not worried about the immediate future at all. We definitely haven't reached the point where hastily-implemented regulations will do more good than harm. That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again.
- darkmighty 12y agoMaybe it's precisely because the it can't be argued against that it's not much use? There's no way we could tell it will ever happen in the moment, no way to tell how intelligent we can make computers, and so on. You can't reliably argue for or against. So what's the point? I mean, you may want to form a group and discuss those scenarios, but it's not excusable to spread fear mongering in the media with no substance like that. It's unscientific in a way. We don't really need to start writing traffic rules for flying cars.
- AndrewKemendo 12y agoWe don't really need to start writing traffic rules for flying cars. This is a great metaphor and one I wish I would have thought up sooner.
- maaku 12y agoIt's a terrible metaphor, or at least it argues against your own position. If flying cars were in fact something on the horizon, we would need discussions about air spaces, commute lanes, and traffic rules. Otherwise lethal mid-air collisions would be much more probable, resulting in significant bystander and property damage as these wreckage fall out of the sky. And, no surprise, we're seeing exactly this discussion going on right now about drones.
- csallen 12y agoI disagree that it can't be argued against. It's a set of logical steps, any one of which can be critiqued: 1 - There's no real dispute that computers will eventually be as "powerful" as brains. In fact, it's likely that they will one day be far more powerful. 2 - Assuming the brain evolved naturally, there's no reason to assume humans won't eventually duplicate the software once we have the hardware. It's really only a matter of when. 3 - An AGI with equal/more power than the human brain will, eventually, be able change its own code and improve upon itself. 4 - Given the characteristics of intelligence as it has been observed thus far in nature, even small increases in IQ lead to massive improvements in capability that are difficult for lesser intelligences to comprehend. A sufficiently advanced AGI would not only be highly capable, but would quite possibly be too smart for us to predict how it might behave. This is a dangerous combination. Further complicating things, we might hit step 2 on accident, without realizing that we hit it. Or some group might accomplish step 2 in secret. What I'd like to see someone do is argue that the chance of these things happening so small and/or the consequences so minuscule that it's not worth worrying about or planning for.
- im3w1l 12y ago>For example, find me someone who can tell me what kind of hardware we need for an AGI. My guess: Something with tons of flops (or even fixed point ops, if that would be easier). Everything else secondary.
- bglazer 12y ago> For example, find me someone who can tell me what kind of hardware we need for an AGI. Can't do it, because no one has any idea. Wouldn't the human brain qualify?
- AndrewKemendo 12y agoNo, it's not Artificial.
- jdietrich 12y agoI cannot disagree more strongly. Look at the technologies underpinning the internet. We badly screwed up a lot of things that are very difficult to fix in retrospect, because we didn't invest enough time and effort into foreseeing the consequences. Nobody worried about the fact that SMTP allows people to send millions of unsolicited messages at practically zero cost until it was far too late to fix the protocol. It is entirely plausible that an RFC describing a proof-of-work scheme could have fixed the problem from the outset, but now we're stuck with Bayesian filters and black/whitelists that sort of work acceptably most of the time. The issues we're seeing regarding surveillance and censorship could have been hugely ameliorated if our protocols were designed with greater foresight, if people like Cerf and Berners-Lee were more aware of those risks. We look back in horror at the naive techno-utopianism of the past and the harms that resulted - tetraethyl lead, asbestos, nuclear fission, CFCs. They were all heralded as miracles of the modern age, but caused harms that we're still dealing with today. The technology industry needs to be far more circumspect about the hazards of what we do. We need less hype about changing the world, and more sleepless nights about the people we might be harming. We need to be having big discussions right now about every aspect of technology, before we open Pandora's box. For example, the World Anti-Doping Agency already have rules prohibiting genetic doping in sport, because they know that it's a potential risk. They would rather figure out their policies now in the cold light of day, rather than in a panic when the Russian team arrive at the 2032 Olympics with a team of superhuman mutants. AGI is a starting point for broader discussions about how society deals with increasingly powerful and ubiquitous computing technologies. Maybe it will happen, maybe it won't, but I don't think that's particularly important; What we can all agree on is that machine intelligence will only become more disruptive to society. We need to plan now for what we would do if certain technologies come good. Driverless cars might never happen or they might be five years away, but we need to figure out what to do with the millions of Americans who drive for a living before we put them all out of work. What would be the social and ethical implications of brain-computer interfaces, of algorithmic decision-making in medicine or the criminal justice system, of ubiquitous facial recognition and tracking? How can we plan to mitigate those harms? If we don't give serious thought to those kinds of questions, then we're sleepwalking into the future.