4 ms·

The Entire OpenAI Explosion Was Ignited by This Research Paper

15 points by pg_1234 3y ago

ilaksh 3y ago
I was really hoping this post would be about Q*.
- theGeatZhopa 3y ago
  Q* really never was able to influence the fleet command so that they fire Picard against the better knowledge. So. Q* is powerless compared to Toner on that matter. Or, it didn't want to spend time on such a, admittedly fun, unrevealing and developing story. Q* knows?
rramadass 3y ago
Quickly browsed the paper and my conclusion is that Sam Altman's anger at Helen Toner for this paper is completely unwarranted. It is well written and the authors merely explore how "costly signals" laid out as policy affects organizations. This should be obvious but in the case of AI research/companies there are multiple dimensions which complicate the matter. OpenAI is one of the example case studies and there is no explicit "criticism" anywhere. Read pages 1-3 for summary of the paper and pages 27-30 (under "private sector signaling") for comparison of OpenAI vs. Anthropic approaches to signaling.
- gausswho 3y ago
  I read the summary and started the main paper. It rather disappoints. The phrase 'costly signals' is spread around to so many meanings. The dangers are vague and not backed up. Very military and geopolitical focus. Its use of the Cuban Missile crisis to compare to this moment AI is bizarre and alarmist. All for a concusion of... we must control the narrative. Spook juice.
  - rramadass 3y ago
    I am afraid you have not read the report and hence have the wrong impression. It is well written, timely and worth a full read. This is the first time i see a complete picture of policy implications of AI safety/non-safety in one reputed report rather than in uninformed articles/blogs. First off, it is published by CSET whose mission is to look at "security implications of emerging technologies" and "to inform research at the intersection of technology and statecraft." (https://cset.georgetown.edu/about-us/ https://cset.georgetown.edu/about-us/) So the focus on Military/Geopolitics/Strategy is mandatory and natural. The term "Costly Signal" is well known and used in Game Theory/International Relations/etc. fields (eg. https://www.youtube.com/watch?v=yfVTKG8O2bI&list=PLB5965C13F4B0B2DA https://www.youtube.com/watch?v=yfVTKG8O2bI&list=PLB5965C13F...) and thus has different aspects to its usage. The origin of the specific four used by the authors in the report is given by the very first citation in the "Endnotes" section. The paper is titled "Decoding Intentions Artificial Intelligence and Costly Signals" and thus 1) the importance of figuring out the intention behind a signal that is being received and 2) the importance of clearly and accurately conveying intention on the sending side is what is being emphasized by the example of "Cuban Missile Crisis", and not that AI safety is of the same category as the "Cuban Missile Crisis". In summary, everybody needs to read this report in entirety to really "grok" the importance and implications of "AI policy making".
    - deleted 3y ago
      [deleted]
  - deleted 3y ago
    [deleted]
- jasontaylor7 3y ago
  I also read much of the paper and agree that there's nothing too bad about OpenAI in it. I question the claims the dispute was about this paper. Perhaps we are seeing the result of zero information, so like one inside of an isolation tank you have millions of people piecing together theories from what they can actually see, such as the fact that a paper exists by a board member and there's a sentence here or two which give a rationale. IMO the dispute wasn't about anything in the paper unless Altman is crazy, which seems ruled out based on his fan base. IMO it's about AGI. The board really doesn't want it and was willing to axe Altman because he is flirting with releasing a "dumbed down" version of it. This is under the assumption that AGI exists and is being kept secret, something most everyone thinks is totally insane.
edandersen 3y ago
Why would you publish research about the activities of the company you are a board member of? Did she see OpenAI itself as an experiment to study? What's the mentality?
- rramadass 3y ago
  https://news.ycombinator.com/item?id=38391591 https://news.ycombinator.com/item?id=38391591
gausswho 3y ago
Blogspam. Actual paper: https://cset.georgetown.edu/wp-content/uploads/CSET-Decoding-Intentions.pdf https://cset.georgetown.edu/wp-content/uploads/CSET-Decoding...

15 points by pg_1234 3y ago

ilaksh 3y ago

I was really hoping this post would be about Q*.

theGeatZhopa 3y ago
Q* really never was able to influence the fleet command so that they fire Picard against the better knowledge. So. Q* is powerless compared to Toner on that matter. Or, it didn't want to spend time on such a, admittedly fun, unrevealing and developing story. Q* knows?

rramadass 3y ago

Quickly browsed the paper and my conclusion is that Sam Altman's anger at Helen Toner for this paper is completely unwarranted. It is well written and the authors merely explore how "costly signals" laid out as policy affects organizations. This should be obvious but in the case of AI research/companies there are multiple dimensions which complicate the matter. OpenAI is one of the example case studies and there is no explicit "criticism" anywhere. Read pages 1-3 for summary of the paper and pages 27-30 (under "private sector signaling") for comparison of OpenAI vs. Anthropic approaches to signaling.

gausswho 3y ago
I read the summary and started the main paper. It rather disappoints. The phrase 'costly signals' is spread around to so many meanings. The dangers are vague and not backed up. Very military and geopolitical focus. Its use of the Cuban Missile crisis to compare to this moment AI is bizarre and alarmist. All for a concusion of... we must control the narrative. Spook juice.
- rramadass 3y ago
  I am afraid you have not read the report and hence have the wrong impression. It is well written, timely and worth a full read. This is the first time i see a complete picture of policy implications of AI safety/non-safety in one reputed report rather than in uninformed articles/blogs. First off, it is published by CSET whose mission is to look at "security implications of emerging technologies" and "to inform research at the intersection of technology and statecraft." (https://cset.georgetown.edu/about-us/ https://cset.georgetown.edu/about-us/) So the focus on Military/Geopolitics/Strategy is mandatory and natural. The term "Costly Signal" is well known and used in Game Theory/International Relations/etc. fields (eg. https://www.youtube.com/watch?v=yfVTKG8O2bI&list=PLB5965C13F4B0B2DA https://www.youtube.com/watch?v=yfVTKG8O2bI&list=PLB5965C13F...) and thus has different aspects to its usage. The origin of the specific four used by the authors in the report is given by the very first citation in the "Endnotes" section. The paper is titled "Decoding Intentions Artificial Intelligence and Costly Signals" and thus 1) the importance of figuring out the intention behind a signal that is being received and 2) the importance of clearly and accurately conveying intention on the sending side is what is being emphasized by the example of "Cuban Missile Crisis", and not that AI safety is of the same category as the "Cuban Missile Crisis". In summary, everybody needs to read this report in entirety to really "grok" the importance and implications of "AI policy making".
  - deleted 3y ago
    [deleted]
- deleted 3y ago
  [deleted]
jasontaylor7 3y ago
I also read much of the paper and agree that there's nothing too bad about OpenAI in it. I question the claims the dispute was about this paper. Perhaps we are seeing the result of zero information, so like one inside of an isolation tank you have millions of people piecing together theories from what they can actually see, such as the fact that a paper exists by a board member and there's a sentence here or two which give a rationale. IMO the dispute wasn't about anything in the paper unless Altman is crazy, which seems ruled out based on his fan base. IMO it's about AGI. The board really doesn't want it and was willing to axe Altman because he is flirting with releasing a "dumbed down" version of it. This is under the assumption that AGI exists and is being kept secret, something most everyone thinks is totally insane.

edandersen 3y ago

Why would you publish research about the activities of the company you are a board member of? Did she see OpenAI itself as an experiment to study? What's the mentality?

rramadass 3y ago
https://news.ycombinator.com/item?id=38391591 https://news.ycombinator.com/item?id=38391591

gausswho 3y ago

Blogspam. Actual paper: https://cset.georgetown.edu/wp-content/uploads/CSET-Decoding-Intentions.pdf https://cset.georgetown.edu/wp-content/uploads/CSET-Decoding...