5 ms·

> We focus on safety challenges not because they necessarily outweigh the potential benefits, [2] but because we wish to motivate further work in safety measure

> We focus on safety challenges not because they necessarily outweigh the potential benefits, [2] but because we wish to motivate further work in safety measurement, mitigation, and assurance. This sentence really stood out and concerns me deeply. We are currently navigating territory that we almost universally believe to be world-changing in ways we cannot predict. Most pop culture focuses on the catastrophic kind of change, because the implications of such tech provides endless opportunities for such story telling. And yet, now that the real thing is here, one of the leading companies at the forefront disclaims "welll, we're looking at this safety thing...but not because we think the dangers are worse than the benefits"...this does not give me any confidence in the OpenAI team. I'm all for approaching problems of safety with an open mind, but the framing here seems really problematic, and seems to indicate a kind of wishful thinking - or at least dangerously optimistic thinking - about what is about to happen. What do I want to hear? Something like: "We recognize that AI poses a unique set of challenges and potential dangers, and we take that seriously. Here's how we're assessing that threat as we iterate, and here's how we'll know when it's time to pump the brakes...".

xkcd1963 4y ago
I had found it hilarious that erotic content was found in the harmful content section.
- jimkleiber 4y ago
  Yea it says to me that there is a huge cultural imprint on what is "good" and what is "bad" and wondering who gets to determine those rules for such wide-reaching platforms. It's one thing for IG to ban nipples, but what happens when such a pervasive tool as these LLMs ban something deemed "harmful" by a handful of developers?
noelsusman 4y ago
>"We recognize that AI poses a unique set of challenges and potential dangers, and we take that seriously. Here's how we're assessing that threat as we iterate, and here's how we'll know when it's time to pump the brakes..." Any company that voluntarily takes on this stance will just end up losing the AI wars to somebody else. This is clearly a job for government since leaving all of these safety/ethics decisions up to a bunch of software engineers seems obviously stupid anyway, but it's unlikely our government institutions will nimble enough to handle this effectively. It's going to be a bumpy ride and all we can do is strap in and hope for the best.
deleted 4y ago
[deleted]
JimmaDaRustla 4y ago
I think you misinterpreted that statement in a way. "not because they necessarily outweigh the potential benefits" is quite literally them stating that their choice to focus on safety challenges is NOT because it's best for their business. "we wish to motivate further work in safety ..." is complimenting that sentiment that this isn't about what's best for their bottom line, but rather spearheading the initiative to ensure the technology is not malicious because it's the right thing to do. They're simply being honest that this isn't about profits or success. Wouldn't you prefer a business to decide what's best for the future of a technology not based on corporate interests? What you wrote sounds like marketing and PR bull TBH. It's fine, but this is a 60 page documentation paper on the literal implementation of a technology...this isn't about making fluffy feel-good virtue signaling statements.
- nr2x 4y ago
  I can also guarantee you as somebody who did a lot of academic publishing prior to a stint at Big Tech, the lawyers had final say on everything. That phrase is pure corporate law.
- haswell 4y ago
  I can no longer edit my comment, and I think I did interpret the statement too narrowly, but I'm not seeing the same interpretation you are re: business impact. If you take what they wrote literally, expand the implications into explicit statements and reframe the sentence with that additional info, it looks something like this: > There are potential benefits to AI and there are potential risks ("safety challenges"). We're not focusing on the safety challenges because we believe it's a foregone conclusion that they outweigh the benefits, but because further work is required to adequately measure safety, to identify mitigations to safety issues, and to provide assurance that such mitigations are sufficient. We believe that the current level of motivation for such research is not sufficient.. There is no language that even implies a monetary or business impact, even if those are factors that are likely to exist as well. There is no literal interpretation that makes room for such a statement as far as I can see. But what emerges is still interesting for a few reasons. 1) It implicitly acknowledges that there is not currently enough motivation to invest in tools that measure risk. This is where my "shouldn't decades of imagining the risks of AI provide more intrinsic motivation" stance comes right back. 2) It also implicitly acknowledges that they do not currently know how risk these models are, and they don't have any way to know. 3) It still reveals a rather disturbing stance towards safety. Am I glad this investigation exists? absolutely. Is it better than nothing? absolutely. Is it worrisome that they're still trying to generate motivation to build tools that measure risk? absolutely. > this is a 60 page documentation paper on the literal implementation of a technology The fact that this is a 60 page paper is kind of the point. A 60 page paper should not be positioning itself with extremely vague language that leaves room for interpretation. A 60 page paper should be direct and clear about what it is saying. The part where I do think business/money comes into play is the decision to frame this so vaguely. To speak plainly about the current state of safety measurement and unknowability or risk would cause a lot of concern. A lot of concern looks bad for business. > ..this isn't about making fluffy feel-good virtue signaling statements. I agree. I'm not worried about a language model hurting someone's feelings. I'm worried about a language model getting prematurely hooked up to systems with broad access to the real world and the failure modes involved there. None of those failure modes have anything to do with feeling good or signaling virtue.
13years 4y ago
Checkout the confidence level of alignment in the below excerpt. "probably". Will this be the standard for future deployments? "Finally, we facilitated a preliminary model evaluation by the Alignment Research Center (ARC) focused on the ability of GPT-4 versions they evaluated to carry out actions to autonomously replicate and gather resources —a risk that, while speculative, may become possible with sufficiently advanced AI systems— with the conclusion that the current model is probably not yet capable of autonomously doing so."
- ChatGTP 4y ago
  "Finally, we facilitated a preliminary model evaluation by the Alignment Research Center (ARC) focused on the ability of GPT-4 versions they evaluated to carry out actions to autonomously replicate and gather resources —a risk that, while speculative, may become possible with sufficiently advanced AI systems— with the conclusion that the current model is probably not yet capable of autonomously doing so." Is this a joke? If this is even slightly considered possible, I think more conversations are needed to allow this to continue. Maybe best to be continuing this research on a self-contained space station or something?
  - 13years 4y ago
    No joke, from page 3 of the pdf. I'm stunned at the lack of rigorous process around alignment that this suggests. Furthermore, in my own contemplations I don't even perceive how alignment could ever be possible as from my perception we have built the very premise on top of an unsolvable paradox. I elaborate on that in great detail in my recent writings here FYI - https://dakara.substack.com/p/ai-singularity-the-hubris-trap https://dakara.substack.com/p/ai-singularity-the-hubris-trap
  - obscur 4y ago
    While I agree it is worth noting that this was already tested in some kind of sandbox environment that only ‘simulated’ replication.
marcus_holmes 4y ago
At least we are considering it. Late last century we wired up a bunch of computers together and unleashed it on an unready world without even considering that it might change everything. And we hadn't even begun to think about security - we literally trusted that everyone was who they said they were! This is a step forward
- brookst 4y ago
  Absent central planning and approval for everything, I’m not sure who the “we” would be that could slow progress. The US largely won the early internet. Maybe wiser people in some other country consciously decided not to move for the valid concerns you mention, but if so, the lack of complete homogeneity among people rendered their care moot.
  - marcus_holmes 4y ago
    Good question. I suppose the "we" is us, the engineers. And it's not so much slowing progress, just doing what this system card is doing: thinking about potential applications of the technology we're building, and mitigating the bad ones if possible. It's an interesting thought experiment; if, say, the team responsible for defining SMTP had done this, what would they have done differently, if anything?
    - brookst 4y ago
      The SMTP example is interesting. There have been numerous proposed replacements, but the network effect problem makes technically superior approaches unviable. So for SMTP I’d say, if that team had done this, someone else would have shipped a basic, flawed protocol first and SMTP would never have gotten off the ground. But… AI doesn’t have those same network effects. At least not technical ones. Business ones to some degree.
      - marcus_holmes 4y ago
        I dunno, I'm not sure there were any competitors for the original SMTP spec (but I don't know for sure). But if there were, it would still have been useful for them all to produce an evaluation of their potential for evil.
brookst 4y ago
Can I suggest a different interpretation of the sentence you quoted? You seem to be interpreting it as “we focus on safety challenges not because the rewards of doing so outweigh the costs, but for indirect reasons” I think what they are actually saying is “please don’t interpret this extensive list of safety challenges to mean that the dangers of AI outweigh the benefits, but because we think a detailed analysis of these dangers will encourage others to develop more responsibly”. Or more succinctly “we wouldn’t be inviting PR problems if we didn’t think it was important to warn other players.”
anon7725 4y ago
The safety that they’re focusing on is _their safety_ as a firm, from PR disasters.
visarga 4y ago
They sat on the model for 6 months just testing and aligning it, how's that for caring about safety? You know who trains and releases without care? FaceBook. They released LLaMA without any RLHF so it can say anything and is completely unfiltered.
- dragonwriter 4y ago
  > They sat on the model for 6 months just testing and aligning it, how’s that for caring about safety? Its not, its them caring about centralized control and PR.
- jerpint 4y ago
  This is a pretty narrow view, and I disagree with it. We can study llama inside out, adapt it for good and bad. It’s objective. GPT is subject to openAI whim. Facebook is sharing findings, openAI shutting its doors
- deleted 4y ago
  [deleted]
- MacsHeadroom 4y ago
  They sat on it for 8 months and then an additional 6 months. And what are all the mitigations they added in that time? 3 pages of text in the report, which amount to "a little less likely to provide detailed instructions on clandestine nerve gas production"
- swatcoder 4y ago
  > They released LLaMA without any RLHF so it can say anything and is completely unfiltered. An open, untrained model is way more responsible to society than a handful of centralized models that have all been trained to reify a "2020's Professional-Class American Urbanite" worldview and project it upon all users. The latter might sound great to you if that's pretty close to your own worldview right now, but is absurdly short-sighted and presumptuous. The reason Microsoft/OpenAI is doing what they're doing is so that they can quickly sell a low-scandal product to global-wealthy users aligned with that $$$-flooded worldview and secure a business lead on that market. They don't want to offend today's customers and don't have the integrity to think about the long-term global picture for society. Their work is in their interest as a corporation, and perhaps in stroking their own ego as individual researchers. The "safety" language is a cynical rationalization.
abecedarius 4y ago
You can expect that benefits of GPT-4 for users will outweigh any direct safety risks from its availability, and still believe that work on these issues is super important because of the prospect of the powers of the next systems, and the ones after them. That's how I'd read this sentence, at least in isolation. (I'm not claiming these benefits will outweigh the risks, necessarily. The sentence doesn't make any claim either way.)
haswell 4y ago
I'm replying here because I can no longer edit my comment, and I think I interpreted the statement too narrowly. After parsing it more carefully and expanding some of the weasel words/implications, I think it goes something like this: > There are potential benefits to AI and there are potential risks ("safety challenges"). We're not focusing on the safety challenges because we believe it's a foregone conclusion that they outweigh the benefits, but because further work is required to adequately measure safety, to identify mitigations to safety issues, and to provide assurance that such mitigations are sufficient. We believe that the current level of motivation for such research is not sufficient. I took the most liberty with the last sentence, but I put it there because it seems that if the primary motivation is to generate interest, there must be a belief that the current motivation is not there. I think this is ultimately a more charitable interpretation than what I had initially drawn, but also seems deeply disturbing that the company at the forefront of this research is in the stage of trying to generate motivation to build the tools to measure safety. That still terrifies me. And I also find it worrisome that such a consequential sentence has clearly been run through many levels of PR/Legal/Marketing/etc. It shouldn't be necessary to read tea leaves on this issue. I do find it encouraging that this paper exists, and I hope it has the desired effect.
numpad0 4y ago
What concerns me in that sentence and OpenAI's decisions is that there is zero consideration for democracy in it. It is such a 19th-century elitism and rule of might thinking, that because we have power we also have right to impose our correct rule on the society at will. No. They shouldn't have the might nor the right, even if taking them away would end in a global thermonuclear war. We might as well start a global thermonuclear war than them having such a sole in-democratic power.
- JohnFen 4y ago
  I don't understand. Whatever you think of their products, they are their products. They can do whatever they wish with them, same as anything you or I build. How is that imposing their rule on society?
  - swatcoder 4y ago
    I think many people recognize that when society comes to rely on things with very high capital requirements, such that only a very few of these things might exist and compete with each other, there is a duty of responsible stewardship for those that manage them. In this case, people are imagining that ChatGPT will be one of a handful of extremely capable AI platforms that completely overhaul society and that the few operators of those platforms will make choices very differently than a democratic polity might, thereby subverting democratic society. I don’t know if that’s going to happen, but that’s the worry. The scope and consequences of the right to “do what you want with your product” is different at scale, especially for very impactful things.
    - JohnFen 4y ago
      > The scope and consequences of the right to “do what you want with your product” is different at scale, especially for very impactful things I agree. I guess I'm just confused by people acting as if Open AI's products are so fundamental now. If the day comes that they are, I'd have a totally different opinion on the sentiment.
hackerlight 4y ago
If you look past the literal interpretation of the specific words in this particular statement, the people involved have previously shown a deep appreciation for the risks. Altman is optimistic that the risks can be mitigated with effort, but he believes that they're there.

xkcd1963 4y ago

I had found it hilarious that erotic content was found in the harmful content section.

jimkleiber 4y ago
Yea it says to me that there is a huge cultural imprint on what is "good" and what is "bad" and wondering who gets to determine those rules for such wide-reaching platforms. It's one thing for IG to ban nipples, but what happens when such a pervasive tool as these LLMs ban something deemed "harmful" by a handful of developers?

noelsusman 4y ago

>"We recognize that AI poses a unique set of challenges and potential dangers, and we take that seriously. Here's how we're assessing that threat as we iterate, and here's how we'll know when it's time to pump the brakes..." Any company that voluntarily takes on this stance will just end up losing the AI wars to somebody else. This is clearly a job for government since leaving all of these safety/ethics decisions up to a bunch of software engineers seems obviously stupid anyway, but it's unlikely our government institutions will nimble enough to handle this effectively. It's going to be a bumpy ride and all we can do is strap in and hope for the best.

deleted 4y ago

[deleted]

JimmaDaRustla 4y ago

I think you misinterpreted that statement in a way. "not because they necessarily outweigh the potential benefits" is quite literally them stating that their choice to focus on safety challenges is NOT because it's best for their business. "we wish to motivate further work in safety ..." is complimenting that sentiment that this isn't about what's best for their bottom line, but rather spearheading the initiative to ensure the technology is not malicious because it's the right thing to do. They're simply being honest that this isn't about profits or success. Wouldn't you prefer a business to decide what's best for the future of a technology not based on corporate interests? What you wrote sounds like marketing and PR bull TBH. It's fine, but this is a 60 page documentation paper on the literal implementation of a technology...this isn't about making fluffy feel-good virtue signaling statements.

nr2x 4y ago
I can also guarantee you as somebody who did a lot of academic publishing prior to a stint at Big Tech, the lawyers had final say on everything. That phrase is pure corporate law.
haswell 4y ago
I can no longer edit my comment, and I think I did interpret the statement too narrowly, but I'm not seeing the same interpretation you are re: business impact. If you take what they wrote literally, expand the implications into explicit statements and reframe the sentence with that additional info, it looks something like this: > There are potential benefits to AI and there are potential risks ("safety challenges"). We're not focusing on the safety challenges because we believe it's a foregone conclusion that they outweigh the benefits, but because further work is required to adequately measure safety, to identify mitigations to safety issues, and to provide assurance that such mitigations are sufficient. We believe that the current level of motivation for such research is not sufficient.. There is no language that even implies a monetary or business impact, even if those are factors that are likely to exist as well. There is no literal interpretation that makes room for such a statement as far as I can see. But what emerges is still interesting for a few reasons. 1) It implicitly acknowledges that there is not currently enough motivation to invest in tools that measure risk. This is where my "shouldn't decades of imagining the risks of AI provide more intrinsic motivation" stance comes right back. 2) It also implicitly acknowledges that they do not currently know how risk these models are, and they don't have any way to know. 3) It still reveals a rather disturbing stance towards safety. Am I glad this investigation exists? absolutely. Is it better than nothing? absolutely. Is it worrisome that they're still trying to generate motivation to build tools that measure risk? absolutely. > this is a 60 page documentation paper on the literal implementation of a technology The fact that this is a 60 page paper is kind of the point. A 60 page paper should not be positioning itself with extremely vague language that leaves room for interpretation. A 60 page paper should be direct and clear about what it is saying. The part where I do think business/money comes into play is the decision to frame this so vaguely. To speak plainly about the current state of safety measurement and unknowability or risk would cause a lot of concern. A lot of concern looks bad for business. > ..this isn't about making fluffy feel-good virtue signaling statements. I agree. I'm not worried about a language model hurting someone's feelings. I'm worried about a language model getting prematurely hooked up to systems with broad access to the real world and the failure modes involved there. None of those failure modes have anything to do with feeling good or signaling virtue.

13years 4y ago

Checkout the confidence level of alignment in the below excerpt. "probably". Will this be the standard for future deployments? "Finally, we facilitated a preliminary model evaluation by the Alignment Research Center (ARC) focused on the ability of GPT-4 versions they evaluated to carry out actions to autonomously replicate and gather resources —a risk that, while speculative, may become possible with sufficiently advanced AI systems— with the conclusion that the current model is probably not yet capable of autonomously doing so."

ChatGTP 4y ago
"Finally, we facilitated a preliminary model evaluation by the Alignment Research Center (ARC) focused on the ability of GPT-4 versions they evaluated to carry out actions to autonomously replicate and gather resources —a risk that, while speculative, may become possible with sufficiently advanced AI systems— with the conclusion that the current model is probably not yet capable of autonomously doing so." Is this a joke? If this is even slightly considered possible, I think more conversations are needed to allow this to continue. Maybe best to be continuing this research on a self-contained space station or something?
- 13years 4y ago
  No joke, from page 3 of the pdf. I'm stunned at the lack of rigorous process around alignment that this suggests. Furthermore, in my own contemplations I don't even perceive how alignment could ever be possible as from my perception we have built the very premise on top of an unsolvable paradox. I elaborate on that in great detail in my recent writings here FYI - https://dakara.substack.com/p/ai-singularity-the-hubris-trap https://dakara.substack.com/p/ai-singularity-the-hubris-trap
- obscur 4y ago
  While I agree it is worth noting that this was already tested in some kind of sandbox environment that only ‘simulated’ replication.

marcus_holmes 4y ago

At least we are considering it. Late last century we wired up a bunch of computers together and unleashed it on an unready world without even considering that it might change everything. And we hadn't even begun to think about security - we literally trusted that everyone was who they said they were! This is a step forward

brookst 4y ago
Absent central planning and approval for everything, I’m not sure who the “we” would be that could slow progress. The US largely won the early internet. Maybe wiser people in some other country consciously decided not to move for the valid concerns you mention, but if so, the lack of complete homogeneity among people rendered their care moot.
- marcus_holmes 4y ago
  Good question. I suppose the "we" is us, the engineers. And it's not so much slowing progress, just doing what this system card is doing: thinking about potential applications of the technology we're building, and mitigating the bad ones if possible. It's an interesting thought experiment; if, say, the team responsible for defining SMTP had done this, what would they have done differently, if anything?
  - brookst 4y ago
    The SMTP example is interesting. There have been numerous proposed replacements, but the network effect problem makes technically superior approaches unviable. So for SMTP I’d say, if that team had done this, someone else would have shipped a basic, flawed protocol first and SMTP would never have gotten off the ground. But… AI doesn’t have those same network effects. At least not technical ones. Business ones to some degree.
    - marcus_holmes 4y ago
      I dunno, I'm not sure there were any competitors for the original SMTP spec (but I don't know for sure). But if there were, it would still have been useful for them all to produce an evaluation of their potential for evil.

brookst 4y ago

Can I suggest a different interpretation of the sentence you quoted? You seem to be interpreting it as “we focus on safety challenges not because the rewards of doing so outweigh the costs, but for indirect reasons” I think what they are actually saying is “please don’t interpret this extensive list of safety challenges to mean that the dangers of AI outweigh the benefits, but because we think a detailed analysis of these dangers will encourage others to develop more responsibly”. Or more succinctly “we wouldn’t be inviting PR problems if we didn’t think it was important to warn other players.”

anon7725 4y ago

The safety that they’re focusing on is _their safety_ as a firm, from PR disasters.

visarga 4y ago

They sat on the model for 6 months just testing and aligning it, how's that for caring about safety? You know who trains and releases without care? FaceBook. They released LLaMA without any RLHF so it can say anything and is completely unfiltered.

dragonwriter 4y ago
> They sat on the model for 6 months just testing and aligning it, how’s that for caring about safety? Its not, its them caring about centralized control and PR.
jerpint 4y ago
This is a pretty narrow view, and I disagree with it. We can study llama inside out, adapt it for good and bad. It’s objective. GPT is subject to openAI whim. Facebook is sharing findings, openAI shutting its doors
deleted 4y ago
[deleted]
MacsHeadroom 4y ago
They sat on it for 8 months and then an additional 6 months. And what are all the mitigations they added in that time? 3 pages of text in the report, which amount to "a little less likely to provide detailed instructions on clandestine nerve gas production"
swatcoder 4y ago
> They released LLaMA without any RLHF so it can say anything and is completely unfiltered. An open, untrained model is way more responsible to society than a handful of centralized models that have all been trained to reify a "2020's Professional-Class American Urbanite" worldview and project it upon all users. The latter might sound great to you if that's pretty close to your own worldview right now, but is absurdly short-sighted and presumptuous. The reason Microsoft/OpenAI is doing what they're doing is so that they can quickly sell a low-scandal product to global-wealthy users aligned with that $$$-flooded worldview and secure a business lead on that market. They don't want to offend today's customers and don't have the integrity to think about the long-term global picture for society. Their work is in their interest as a corporation, and perhaps in stroking their own ego as individual researchers. The "safety" language is a cynical rationalization.

abecedarius 4y ago

You can expect that benefits of GPT-4 for users will outweigh any direct safety risks from its availability, and still believe that work on these issues is super important because of the prospect of the powers of the next systems, and the ones after them. That's how I'd read this sentence, at least in isolation. (I'm not claiming these benefits will outweigh the risks, necessarily. The sentence doesn't make any claim either way.)

haswell 4y ago

I'm replying here because I can no longer edit my comment, and I think I interpreted the statement too narrowly. After parsing it more carefully and expanding some of the weasel words/implications, I think it goes something like this: > There are potential benefits to AI and there are potential risks ("safety challenges"). We're not focusing on the safety challenges because we believe it's a foregone conclusion that they outweigh the benefits, but because further work is required to adequately measure safety, to identify mitigations to safety issues, and to provide assurance that such mitigations are sufficient. We believe that the current level of motivation for such research is not sufficient. I took the most liberty with the last sentence, but I put it there because it seems that if the primary motivation is to generate interest, there must be a belief that the current motivation is not there. I think this is ultimately a more charitable interpretation than what I had initially drawn, but also seems deeply disturbing that the company at the forefront of this research is in the stage of trying to generate motivation to build the tools to measure safety. That still terrifies me. And I also find it worrisome that such a consequential sentence has clearly been run through many levels of PR/Legal/Marketing/etc. It shouldn't be necessary to read tea leaves on this issue. I do find it encouraging that this paper exists, and I hope it has the desired effect.

numpad0 4y ago

What concerns me in that sentence and OpenAI's decisions is that there is zero consideration for democracy in it. It is such a 19th-century elitism and rule of might thinking, that because we have power we also have right to impose our correct rule on the society at will. No. They shouldn't have the might nor the right, even if taking them away would end in a global thermonuclear war. We might as well start a global thermonuclear war than them having such a sole in-democratic power.

JohnFen 4y ago
I don't understand. Whatever you think of their products, they are their products. They can do whatever they wish with them, same as anything you or I build. How is that imposing their rule on society?
- swatcoder 4y ago
  I think many people recognize that when society comes to rely on things with very high capital requirements, such that only a very few of these things might exist and compete with each other, there is a duty of responsible stewardship for those that manage them. In this case, people are imagining that ChatGPT will be one of a handful of extremely capable AI platforms that completely overhaul society and that the few operators of those platforms will make choices very differently than a democratic polity might, thereby subverting democratic society. I don’t know if that’s going to happen, but that’s the worry. The scope and consequences of the right to “do what you want with your product” is different at scale, especially for very impactful things.
  - JohnFen 4y ago
    > The scope and consequences of the right to “do what you want with your product” is different at scale, especially for very impactful things I agree. I guess I'm just confused by people acting as if Open AI's products are so fundamental now. If the day comes that they are, I'd have a totally different opinion on the sentiment.

hackerlight 4y ago

If you look past the literal interpretation of the specific words in this particular statement, the people involved have previously shown a deep appreciation for the risks. Altman is optimistic that the risks can be mitigated with effort, but he believes that they're there.