4 ms·
> What, concretely, makes you think that any of those companies wouldn't place a focus on safety and value alignment? Competitive pressure, and the "if we don'
by Analemma_ 9y ago
> What, concretely, makes you think that any of those companies wouldn't place a focus on safety and value alignment?
Competitive pressure, and the "if we don't, someone else will" effect (or Moloch, if you like). AGI- particularly recursively self-improving AGI- is the ultimate first-mover advantage: the first company or country to have AGI will very likely be able to leverage that into keeping anyone else from getting it (if it doesn't, y'know, kill us). This strongly encourages treating all concerns other than "get there first" as secondary.
> Automobile manufacturers and their tier 1 suppliers are the world leaders in automobile safety, after all.
Not by choice they aren't. They are forced to be the way they are by government regulations, which they always bitterly opposed at the time of creation. In fact, capitalism has such a reliable record of "not giving a shit about safety until they are forced to" that I'm perplexed you think AGI would be any different.
- dsacco 9y ago> recursively self-improving AGI- is the ultimate first-mover advantage: the first company or country to have AGI will very likely be able to leverage that into keeping anyone else from getting it How? This doesn't seem axiomatic.
- nunya213 9y agoSeems clear that you could instruct the AGI to do anything it could to interfere with other organizations efforts to build an AGI. Obviously Nation States would have strong reasons for pursuing such a course of action and unscrupulous corporations would likewise have strong capitalistic motivations to do so.
- p1esk 9y agoSeems very unclear if you could "instruct" true AGI to do something.
- nunya213 9y agoMaybe you assume that an AGI would be totally uncontrollable but in this highly speculative exercise I don't think you should assume your position is the only valid one.
- p1esk 9y agoI don't think you should assume your position is the only valid one. I don't, that's why I said "very unclear" (to me). If something is "clear" to you about AGI, then it seems like it's you who makes that assumption.
- nunya213 9y agoSeems very unclear if you could "instruct" true AGI to do something. So you are claiming that by making this statement you are not assuming that it is impossible to control an AGI?
- p1esk 9y agoI'm claiming that it's not clear to me whether we can control it or not. This is similar to having a very advanced bioengineering technology, where you could instantly change anything about your body. Would you make yourself smarter? Sure. Would you change who you are (e.g. turned off your instincts/desires)? Not so sure.
- dsacco 9y agoFirst you said it's clear, now you're saying it's highly speculative. Choose one.
- nunya213 9y agoI see no conflict.
- throwawayjava 9y agoThis is a great response. Thanks. > Not by choice they aren't. This is why I was glad to see this in TFA: >>> ...while increasing the importance of sharing safety, policy, and standards research." Even for Weak AI, history tells us that these factors -- policy and industry collaboration -- tend to be significant ingredients in successful safety reforms.
- eb3c90 9y agoI think there will be at least a minimum of focus on safety and value alignment. They may have to focus on safety and aligning it to the companies values (else test/weak versions are likely to destroy/negatively impact the company or at least be useless). Which is at least something. Only hyperrational/intelligent value unaligned intelligent systems would avoid this pressure. I don't see the development going that way. It would still be sub optimal though.