3 ms·

Ugh, there's a lot of argument about that in the cfar alumni community. Some folks take it for granted (why??? Take things for granted about such an uncertain s

by lahwran 11y ago

Ugh, there's a lot of argument about that in the cfar alumni community. Some folks take it for granted (why??? Take things for granted about such an uncertain subject???) that we're just doomed unless you Give Miri Money(tm). Those folks tend to be pretty good about actually carrying out what is reasonable behaviour in most other ways if only they were right about that one thing - if it really were such a big deal, you'd want to not ignore it. Meanwhile, another part of the alumni community actually understands the theory behind ai and machine learning, and those folks end up in arguments frequently with the first category about the topic. The reason you hear about it is the first category is a pretty panicked and hopeless group - for the people who actually believe yudkowsky's "recursive algorithmic improvement" to be able to give large improvements, they generally think that humanity "loses by default" if they do nothing. So they tend to be very, very into recruiting. Thankfully they're not so nuts about it that they'll never change their minds, the problem is it takes a lot of explaining to get the theoretical basis for why the recursive self improvement thing isn't actually as scary as they think it is. No, it's not going to take an hour as soon as an ai is built, learning is hard, and humans are freakishly good at it. Computers will beat us at data efficiency eventually but it's gonna take a while, and current machine learning is better at being data inefficient but getting good results from the large amounts of data. And the best you can do isn't good enough to make miri's monster - unbiased, maximally data-efficient Bayesian inference doesn't actually fit in the universe in either a time or memory sense if you try to build a full ai out of just that one thing. And approximating it is, you guessed it, less data efficient.

edanm 11y ago
You seem to assume that the belief that humanity "loses by default" is wrong. I'm curious to hear why, since you seem to be know about this topic. (And because I disagree with your view, so hearing might educate me). "Some folks take it for granted (why??? Take things for granted about such an uncertain subject???) that we're just doomed unless you Give Miri Money(tm)." Are there really people who would put it that way? I know plenty of people (including myself) who think it's a good idea to donate to MIRI, but I certainly wouldn't put it in the terms you did.
- lahwran 11y ago
  So as I was writing a reply, I found that my opinion doesn't actually differ in "do we need to do a bunch of stuff"; it's more that I don't think miri is taking a useful approach. After talking to a friend who is also in ml about this really, I think the key point I should make is that it's more a matter of engineering control tools that ensure we can spy on its thoughts. If we can build an agi, we can also build safeguards that it doesn't realize are there until too late. Anyway, here is the original comment I was going to write. You can read it and extract your thoughts; I'm fairly confident I got the theory right, but I've reduced my confidence in the point/counterpoint. Regardless, I definitely think miri's position is unreasonably extreme, and this is a pretty ok explanation of why. https://gist.github.com/lahwran/99d84c3f8461ece9153f https://gist.github.com/lahwran/99d84c3f8461ece9153f
  - edanm 11y ago
    Thanks for the detailed reply! So firstly, I'm glad we agree on the core idea - some effort/funding should be put into this problem. You don't think MIRI is the best vehicle for such funding - that's understandable, although I'm semi-biased in their favor as they probably brought the most attention to this issue, and have been working on it for a long time. I assume that they have good reasons for the approaches they're taking right now. But again, not believing in MIRI or in a foom event is totally compatible with still believing we should be doing something about this problem, so you and I are pretty closely in sync. Having said that, I read through your notes, and I do have a major object. (You seem to understand more of the technical details than me, so take with a grain of salt). I think your argument boils down to one main thing - that humans are pretty much "efficient" with respect to data usage already, because of evolution. If we take away that belief, then your reasons for thinking we can't build an AI that's superintelligent goes away - we would be able to squeeze more efficiency out of it. So here's the problem - I don't understand why you believe that humans are data efficient. I understand the idea behind it, but it seemed to me like the only explanation for data efficiency is that evolution would've taken care of it. You even mention that the default argument agains this point is that eveolution might get stuck in local optimums, but it's very hard to do better. But there are 2 counterarguments to this: 1. I don't think it's so much difficult to do better than evolution, since we do it regularly. We defeat diseases. We build machines that are faster/stronger/etc than anything evolution has "come up with". Etc. 2. I don't think evolution was "optimizing" for intelligence anyway. Our intelligence, however limited, is enough to become the most powerful species on the planet. Biology faces very hard trade-offs in squeezing more intelligence into us, but this isn't true of digital equipment, which doesn't have to be based on hardware that, e.g. has to be born, therefore has to squeeze through the birth canal, etc. These are limits that evolution was "working under", but that we don't have. Note: All anthropomorphizing of evolution done figuratively.
someguydave 11y ago
Okay so: 1.) preaches about new destroyer god 2.) saviors who possess the secret knowledge of salvation 3.) give them money 4.) think correctly, not incorrectly Yep, it's a cult.
- lahwran 11y ago
  Eh... Sort of? The two organisations together would form a cult, but the community split I mention makes it a bit confusing. Overall, I agree that miri's level-of-cult is too damn high.
  - someguydave 11y ago
    Interestingly, it's an ancient form of cult called Gnosticism. Gnostics teach that the material world is evil ("gives rise to the AI"), and only through hidden knowledge ("correct thinking") can one find the true path to spiritual salvation ("become an immortal human") https://en.wikipedia.org/wiki/Gnosticism https://en.wikipedia.org/wiki/Gnosticism
- deleted 11y ago
  [deleted]
philh 11y ago
> Ugh, there's a lot of argument about that in the cfar alumni community. I take it you're involved with parts of the community outside the CFAR mailing list? I haven't seen it on there. There's certainly argument about it within the broader LW community, but the CFAR alumni community is distinct (but overlapping).
- lahwran 11y ago
  primarily in person, yes. presumably primarily from people who frequent less wrong. I don't, so I don't really know.

by lahwran 11y ago

edanm 11y ago

You seem to assume that the belief that humanity "loses by default" is wrong. I'm curious to hear why, since you seem to be know about this topic. (And because I disagree with your view, so hearing might educate me). "Some folks take it for granted (why??? Take things for granted about such an uncertain subject???) that we're just doomed unless you Give Miri Money(tm)." Are there really people who would put it that way? I know plenty of people (including myself) who think it's a good idea to donate to MIRI, but I certainly wouldn't put it in the terms you did.

lahwran 11y ago
So as I was writing a reply, I found that my opinion doesn't actually differ in "do we need to do a bunch of stuff"; it's more that I don't think miri is taking a useful approach. After talking to a friend who is also in ml about this really, I think the key point I should make is that it's more a matter of engineering control tools that ensure we can spy on its thoughts. If we can build an agi, we can also build safeguards that it doesn't realize are there until too late. Anyway, here is the original comment I was going to write. You can read it and extract your thoughts; I'm fairly confident I got the theory right, but I've reduced my confidence in the point/counterpoint. Regardless, I definitely think miri's position is unreasonably extreme, and this is a pretty ok explanation of why. https://gist.github.com/lahwran/99d84c3f8461ece9153f https://gist.github.com/lahwran/99d84c3f8461ece9153f
- edanm 11y ago
  Thanks for the detailed reply! So firstly, I'm glad we agree on the core idea - some effort/funding should be put into this problem. You don't think MIRI is the best vehicle for such funding - that's understandable, although I'm semi-biased in their favor as they probably brought the most attention to this issue, and have been working on it for a long time. I assume that they have good reasons for the approaches they're taking right now. But again, not believing in MIRI or in a foom event is totally compatible with still believing we should be doing something about this problem, so you and I are pretty closely in sync. Having said that, I read through your notes, and I do have a major object. (You seem to understand more of the technical details than me, so take with a grain of salt). I think your argument boils down to one main thing - that humans are pretty much "efficient" with respect to data usage already, because of evolution. If we take away that belief, then your reasons for thinking we can't build an AI that's superintelligent goes away - we would be able to squeeze more efficiency out of it. So here's the problem - I don't understand why you believe that humans are data efficient. I understand the idea behind it, but it seemed to me like the only explanation for data efficiency is that evolution would've taken care of it. You even mention that the default argument agains this point is that eveolution might get stuck in local optimums, but it's very hard to do better. But there are 2 counterarguments to this: 1. I don't think it's so much difficult to do better than evolution, since we do it regularly. We defeat diseases. We build machines that are faster/stronger/etc than anything evolution has "come up with". Etc. 2. I don't think evolution was "optimizing" for intelligence anyway. Our intelligence, however limited, is enough to become the most powerful species on the planet. Biology faces very hard trade-offs in squeezing more intelligence into us, but this isn't true of digital equipment, which doesn't have to be based on hardware that, e.g. has to be born, therefore has to squeeze through the birth canal, etc. These are limits that evolution was "working under", but that we don't have. Note: All anthropomorphizing of evolution done figuratively.

someguydave 11y ago

Okay so: 1.) preaches about new destroyer god 2.) saviors who possess the secret knowledge of salvation 3.) give them money 4.) think correctly, not incorrectly Yep, it's a cult.

lahwran 11y ago
Eh... Sort of? The two organisations together would form a cult, but the community split I mention makes it a bit confusing. Overall, I agree that miri's level-of-cult is too damn high.
- someguydave 11y ago
  Interestingly, it's an ancient form of cult called Gnosticism. Gnostics teach that the material world is evil ("gives rise to the AI"), and only through hidden knowledge ("correct thinking") can one find the true path to spiritual salvation ("become an immortal human") https://en.wikipedia.org/wiki/Gnosticism https://en.wikipedia.org/wiki/Gnosticism
deleted 11y ago
[deleted]

philh 11y ago

> Ugh, there's a lot of argument about that in the cfar alumni community. I take it you're involved with parts of the community outside the CFAR mailing list? I haven't seen it on there. There's certainly argument about it within the broader LW community, but the CFAR alumni community is distinct (but overlapping).

lahwran 11y ago
primarily in person, yes. presumably primarily from people who frequent less wrong. I don't, so I don't really know.