4 ms·

Let's be honest here, a lot of HN users have a conflict of interest on this topic. AI entrepreneurs trying to get rich from LLMs benefit from LLMs being open s

Let's be honest here, a lot of HN users have a conflict of interest on this topic. AI entrepreneurs trying to get rich from LLMs benefit from LLMs being open source. But the downside risk from e.g. bioweapons is spread across all of society. It's the same sort of asymmetrical cost/benefit that tobacco companies and greenhouse gas emitters face. Of course if you went to an online forum for oil companies, they'd be hopping mad if they're prevented from drilling, and dismissive of global warming risks. It's no different here.

mattnewton 3y ago
It is worth calling out the motivations of most entrepreneurs here. But I think that analogy you used is very uncharitable - drilling and burning fossil fuels necessarily harms the environment, but the track record on big companies handling alignment/safety in house, rather than open source with the whole research community working on it is still very much up in the air. Sidney (bing assistant) was easy to prompt inject and ask for bad things, and the research that people have been able to do on forcing the output of llama to confirm to certain rules will likely prove invaluable in the future.
- 0xDEAFBEAD 3y ago
  >the track record on big companies handling alignment/safety in house, rather than open source with the whole research community working on it is still very much up in the air. Sidney (bing assistant) was easy to prompt inject and ask for bad things Yep, Microsoft did a terrible job, and they should've been punished. I'm not claiming that Big AI rocks at safety. I'm claiming that Big AI is also a big target for regulators and public ire. There's at least a chance they will get their act together in response to external pressure. But if cutting-edge models are open sourced indefinitely, they'll effectively be impossible to control. >research that people have been able to do on forcing the output of llama to confirm to certain rules will likely prove invaluable in the future. You may be correct that releasing llama was beneficial from the point of view of safety. But the "conform to certain rules" strategy can basically only work if (a) there's a way to enforce rules that can't be fine-tuned away, or (b) we stop releasing models at some point.
MaKey 3y ago
> But the downside risk from e.g. bioweapons is spread across all of society. It gets old hearing about these "risks" in the context of AI. It's just an excuse used by companies to keep as much power as possible to themselves. The real risk is AI being applied in decision making where it affects humans.
- 0xDEAFBEAD 3y ago
  Concern with AI risk actually originated decades ago, way before current AI hype. Here are some of Eliezer Yudkowsky's papers from 2006/2008: https://www.stat.berkeley.edu/~aldous/157/Papers/yudkowsky.pdf https://www.stat.berkeley.edu/~aldous/157/Papers/yudkowsky.p... https://intelligence.org/files/AIPosNegFactor.pdf https://intelligence.org/files/AIPosNegFactor.pdf I am concerned with AI companies keeping all the power to themselves. The recent announcement from the OpenAI board was encouraging on that front, because it makes me believe that maybe they aren't focused on pursuing profit at all costs. Even so, in some cases we want power to be restricted. For example, I'm not keen on democratizing access to nuclear weapons. >The real risk is AI being applied in decision making where it affects humans. I agree. But almost any decision it makes can affect humans directly or indirectly. In any case, the more widespread the access to these models, the greater the probability of a bad actor abusing the model. Perhaps the current generation of models won't allow them to do much damage, but the next generation might, or the one after that. It seems like on our current path, the only way for us to learn about LLM dangers is the hard way.
  - nradov 3y ago
    But Eliezer Yudkowsky just makes shit up. There's no valid scientific reason to take his concerns seriously.
    - 0xDEAFBEAD 3y ago
      Can you be more specific about which stuff you read from Eliezer? Did you read this post for example? https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... It sounds like maybe you're saying: "It's not scientifically valid to suggest that AGI could kill everyone until it actually does so. At that point we would have sufficient evidence." Am I stating your position accurately? If so, can you see why others might find this approach unappealing?
RcouF1uZ4gsC 3y ago
I heard that reading is very dangerous. Reading allows people to for example learn how to build bio weapons. In addition, reading can spread ideas that are dangerous. Many people have died because they were influenced by what they read. It would be much safer if reading were strictly controlled. The companies would have “reading as a service” where regular people could bring their books to have them read. The reader would ensure that the book aligns with the ethics of the company and would refuse to read any work that either does. It align with their ethics or teaches people anything dangerous (like chemistry or physics which can be used to build bombs and other weapons).
- smegger001 3y ago
  Stop that your giving Florida ideas.
- 0xDEAFBEAD 3y ago
  You might be interested in these comments: https://news.ycombinator.com/item?id=38335089 https://news.ycombinator.com/item?id=38335089 https://news.ycombinator.com/item?id=38336054 https://news.ycombinator.com/item?id=38336054

mattnewton 3y ago

It is worth calling out the motivations of most entrepreneurs here. But I think that analogy you used is very uncharitable - drilling and burning fossil fuels necessarily harms the environment, but the track record on big companies handling alignment/safety in house, rather than open source with the whole research community working on it is still very much up in the air. Sidney (bing assistant) was easy to prompt inject and ask for bad things, and the research that people have been able to do on forcing the output of llama to confirm to certain rules will likely prove invaluable in the future.

0xDEAFBEAD 3y ago
>the track record on big companies handling alignment/safety in house, rather than open source with the whole research community working on it is still very much up in the air. Sidney (bing assistant) was easy to prompt inject and ask for bad things Yep, Microsoft did a terrible job, and they should've been punished. I'm not claiming that Big AI rocks at safety. I'm claiming that Big AI is also a big target for regulators and public ire. There's at least a chance they will get their act together in response to external pressure. But if cutting-edge models are open sourced indefinitely, they'll effectively be impossible to control. >research that people have been able to do on forcing the output of llama to confirm to certain rules will likely prove invaluable in the future. You may be correct that releasing llama was beneficial from the point of view of safety. But the "conform to certain rules" strategy can basically only work if (a) there's a way to enforce rules that can't be fine-tuned away, or (b) we stop releasing models at some point.

MaKey 3y ago

> But the downside risk from e.g. bioweapons is spread across all of society. It gets old hearing about these "risks" in the context of AI. It's just an excuse used by companies to keep as much power as possible to themselves. The real risk is AI being applied in decision making where it affects humans.

0xDEAFBEAD 3y ago
Concern with AI risk actually originated decades ago, way before current AI hype. Here are some of Eliezer Yudkowsky's papers from 2006/2008: https://www.stat.berkeley.edu/~aldous/157/Papers/yudkowsky.pdf https://www.stat.berkeley.edu/~aldous/157/Papers/yudkowsky.p... https://intelligence.org/files/AIPosNegFactor.pdf https://intelligence.org/files/AIPosNegFactor.pdf I am concerned with AI companies keeping all the power to themselves. The recent announcement from the OpenAI board was encouraging on that front, because it makes me believe that maybe they aren't focused on pursuing profit at all costs. Even so, in some cases we want power to be restricted. For example, I'm not keen on democratizing access to nuclear weapons. >The real risk is AI being applied in decision making where it affects humans. I agree. But almost any decision it makes can affect humans directly or indirectly. In any case, the more widespread the access to these models, the greater the probability of a bad actor abusing the model. Perhaps the current generation of models won't allow them to do much damage, but the next generation might, or the one after that. It seems like on our current path, the only way for us to learn about LLM dangers is the hard way.
- nradov 3y ago
  But Eliezer Yudkowsky just makes shit up. There's no valid scientific reason to take his concerns seriously.
  - 0xDEAFBEAD 3y ago
    Can you be more specific about which stuff you read from Eliezer? Did you read this post for example? https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... It sounds like maybe you're saying: "It's not scientifically valid to suggest that AGI could kill everyone until it actually does so. At that point we would have sufficient evidence." Am I stating your position accurately? If so, can you see why others might find this approach unappealing?

RcouF1uZ4gsC 3y ago

I heard that reading is very dangerous. Reading allows people to for example learn how to build bio weapons. In addition, reading can spread ideas that are dangerous. Many people have died because they were influenced by what they read. It would be much safer if reading were strictly controlled. The companies would have “reading as a service” where regular people could bring their books to have them read. The reader would ensure that the book aligns with the ethics of the company and would refuse to read any work that either does. It align with their ethics or teaches people anything dangerous (like chemistry or physics which can be used to build bombs and other weapons).

smegger001 3y ago
Stop that your giving Florida ideas.
0xDEAFBEAD 3y ago
You might be interested in these comments: https://news.ycombinator.com/item?id=38335089 https://news.ycombinator.com/item?id=38335089 https://news.ycombinator.com/item?id=38336054 https://news.ycombinator.com/item?id=38336054