3 ms·

> Go lurk on alignmentforum.org for a while, and you'll have a different perspective on OpenAI's decisions. No I won't, because the arguably most successful wa

> Go lurk on alignmentforum.org for a while, and you'll have a different perspective on OpenAI's decisions. No I won't, because the arguably most successful way of detecting, preventing and/or fixing problems with almost all complex systems, is to have as many eyeballs on them as possible. This has been known in software engineering for quite some time: "Given enough eyeballs, all bugs are shallow." -Eric S. Raymond, The Cathedral and the Bazaar, 1999

gwd 4y ago
And you're so sure that this maxim applies to AI alignment, that you're not interested in even hearing what people actually working in the AI field might have to say? (To post on alignmentforum.org, you actually have to demonstrate that you are actively working in AI research.) AND, you're so certain that it applies, that you're willing to potentially risk the fate of the entire human race on it? I wasn't actually suggesting that you lurk there to change your mind; I was just saying that if you see what kinds of discussions the OpenAI engineers are reading, you'll understand better some of the decisions they're making. However, the people posting there do actually have a lot of experience with actual AI, and have done a lot of thinking on the subject -- almost certainly a lot more than you have. Before you make policy recommendations based on ideology (like recommending we just do all AI development open-source style), you should at least try to understand why they think the way they think and engage with it.
- usrbinbash 4y ago
  > Before you make policy recommendations based on ideology Ideologies are belief systems. The fact that open sourcing something is a good way to find errors in complex systems is a proven fact. AI isn't magical in that regard. It is a complex system, and experience with such systems, from economics, to climate mechanisms, to software, teaches us that predictions about them, including error detection, risk management and fixing problems, works the better the more people have a chance to look at its internal workings.
  - gwd 4y ago
    > The fact that open sourcing something is a good way to find errors in complex systems is a proven fact. Is your evidence for this anything other than anecdotal? That ESR quote was given after one particular LKML interaction back in the 90's. And sure, sometimes an interaction like that happens. But just as often I've seen an email or bug report to an open-source project get completely lost. Not to mention that 1) new security-related bugs are still introduced into Linux, despite the number of eyeballs looking at them 2) people are still finding security-related bugs in Linux which are over a decade old. (Not to pick on Linux here -- but Linux probably has the most eyeballs, so according to this theory it should have the least bugs.) Asserting that open-sourcing has always reduced bugs in all software isn't supported by those anecdotes; you'd need to do some sort of actual study comparing various types of development. > AI isn't magical in that regard. It is a complex system, and experience with such systems, from economics, to climate mechanisms, to software, teaches us that predictions about them, including error detection, risk management and fixing problems, works the better the more people have a chance to look at its internal workings. What "AI alignment" people are saying is that from a risk management perspective, AI is different. Namely, the fear is that we'll get an AI which is much more intelligent than us -- an AI intelligent enough to 1) gain a technological superiority over us 2) anticipate and counter anything we could try to do to stop it; and that this AI might end up with its own random "paperclip-maximizing" plans, and care no more about us than we care about ants; and that it might come into being before we have any idea how dangerous it is. The worst think that can happen with a bug that gets into the Linux kernel is you may lose some data, or some other human steals some of your secrets. The worst thing that can happen from an AI alignment catastrophe is the extermination of the human race. Now, maybe those fears are overblown. Or maybe you're right, that open-sourcing everything would be the best way to avert an AI catastrophe in any case. But to assert it's true, based on some thing a random guy said after a random email discussion 25 years ago, without even bothering to engage with what people actively working in the AI field are saying, certainly is an ideology.
    - usrbinbash 4y ago
      > Is your evidence for this anything other than anecdotal? And what is the evidence for closed source being the safest option in AI? > Namely, the fear is that we'll get an AI which is much more intelligent than us Which is probably quite some time away, given the fact that LLMs and other generative models have neither intentionality, nor agency. So right now, and probably for a long time to come, we are not talking about existential threats in the form of Skynet. But we ARE talking about a revolutionary technology, that will change the economic landscape for potentially hundreds of millions of people. I am pretty convinced that a majority of them will not be comfortable with decisions about these technologies being made behind closed (corporate) doors. We are also talking about the much more immediate dangers of AI, which don't arise from super intelligent machines, but from how humans use them, how they are trained, on what, and for what purposes they are used and by whom. These as well are issues that society will want to have more eyeballs on, not less. Other than concerns that someday an AGI might endanger humanity, both of these issues are here, right now, and we have to deal with that fact.

gwd 4y ago

And you're so sure that this maxim applies to AI alignment, that you're not interested in even hearing what people actually working in the AI field might have to say? (To post on alignmentforum.org, you actually have to demonstrate that you are actively working in AI research.) AND, you're so certain that it applies, that you're willing to potentially risk the fate of the entire human race on it? I wasn't actually suggesting that you lurk there to change your mind; I was just saying that if you see what kinds of discussions the OpenAI engineers are reading, you'll understand better some of the decisions they're making. However, the people posting there do actually have a lot of experience with actual AI, and have done a lot of thinking on the subject -- almost certainly a lot more than you have. Before you make policy recommendations based on ideology (like recommending we just do all AI development open-source style), you should at least try to understand why they think the way they think and engage with it.

usrbinbash 4y ago
> Before you make policy recommendations based on ideology Ideologies are belief systems. The fact that open sourcing something is a good way to find errors in complex systems is a proven fact. AI isn't magical in that regard. It is a complex system, and experience with such systems, from economics, to climate mechanisms, to software, teaches us that predictions about them, including error detection, risk management and fixing problems, works the better the more people have a chance to look at its internal workings.
- gwd 4y ago
  > The fact that open sourcing something is a good way to find errors in complex systems is a proven fact. Is your evidence for this anything other than anecdotal? That ESR quote was given after one particular LKML interaction back in the 90's. And sure, sometimes an interaction like that happens. But just as often I've seen an email or bug report to an open-source project get completely lost. Not to mention that 1) new security-related bugs are still introduced into Linux, despite the number of eyeballs looking at them 2) people are still finding security-related bugs in Linux which are over a decade old. (Not to pick on Linux here -- but Linux probably has the most eyeballs, so according to this theory it should have the least bugs.) Asserting that open-sourcing has always reduced bugs in all software isn't supported by those anecdotes; you'd need to do some sort of actual study comparing various types of development. > AI isn't magical in that regard. It is a complex system, and experience with such systems, from economics, to climate mechanisms, to software, teaches us that predictions about them, including error detection, risk management and fixing problems, works the better the more people have a chance to look at its internal workings. What "AI alignment" people are saying is that from a risk management perspective, AI is different. Namely, the fear is that we'll get an AI which is much more intelligent than us -- an AI intelligent enough to 1) gain a technological superiority over us 2) anticipate and counter anything we could try to do to stop it; and that this AI might end up with its own random "paperclip-maximizing" plans, and care no more about us than we care about ants; and that it might come into being before we have any idea how dangerous it is. The worst think that can happen with a bug that gets into the Linux kernel is you may lose some data, or some other human steals some of your secrets. The worst thing that can happen from an AI alignment catastrophe is the extermination of the human race. Now, maybe those fears are overblown. Or maybe you're right, that open-sourcing everything would be the best way to avert an AI catastrophe in any case. But to assert it's true, based on some thing a random guy said after a random email discussion 25 years ago, without even bothering to engage with what people actively working in the AI field are saying, certainly is an ideology.
  - usrbinbash 4y ago
    > Is your evidence for this anything other than anecdotal? And what is the evidence for closed source being the safest option in AI? > Namely, the fear is that we'll get an AI which is much more intelligent than us Which is probably quite some time away, given the fact that LLMs and other generative models have neither intentionality, nor agency. So right now, and probably for a long time to come, we are not talking about existential threats in the form of Skynet. But we ARE talking about a revolutionary technology, that will change the economic landscape for potentially hundreds of millions of people. I am pretty convinced that a majority of them will not be comfortable with decisions about these technologies being made behind closed (corporate) doors. We are also talking about the much more immediate dangers of AI, which don't arise from super intelligent machines, but from how humans use them, how they are trained, on what, and for what purposes they are used and by whom. These as well are issues that society will want to have more eyeballs on, not less. Other than concerns that someday an AGI might endanger humanity, both of these issues are here, right now, and we have to deal with that fact.