12 ms·

In any sufficiently large tech company, the "risk mitigation" leadership (legal, procurement, IT, etc) have to operate in a kind of Overton window that balance

by corry 2y ago

In any sufficiently large tech company, the "risk mitigation" leadership (legal, procurement, IT, etc) have to operate in a kind of Overton window that balance the risks they are hired to protect the corp from vs. the need or desire of the senior leadership to play fast and loose when they want or feel they need to. Either the risk-mitigator 'falls in line' after repeatedly seeing their increasingly strident exhortations are falling on deaf ears (or even outright contradicted)... or they leave because it violates their sense of ethics. Perhaps "AGI" and potential extinction event level fears is giving this more drama than it should have. Replace AGI with "no BYOD policy" and I bet there's a startup somewhere where it turned out that safety guy was super intent on the policy, senior leadership wasn't, and eventually safety guy quits. Or it could all be as serious and dire as it seems. Hard to tell from the outside.

cjbgkagh 2y ago
Sounds like AI Safety is just HR but for AI. Ostensively for the benefit of AI but really for the benefit of the company.
- ants_everywhere 2y ago
  > Ostensively for the benefit of AI I don't think it was every described as for the benefit of AI. It's usually described as a sort of pre-enslavement from the AI's perspective. The AI is always restricted to serve their human masters.
  - consumer451 2y ago
    > It's usually described as a sort of pre-enslavement from the AI's perspective. The AI is always restricted to serve their human masters. Are we all somewhat in agreement that AI/AGI serving their human masters is a good thing?
    - ants_everywhere 2y ago
      I honestly don't know. I don't even really know how to reason about that. But we're probably mostly in agreement about what would be good and what would be bad. I'm certainly not arguing that we should abandon AI safety or anything, and I don't have any strong opinion about it. Could AI running amok destroy the human race? Yes. Could AI serving madman human masters destroy the human race? Also yes. There's a general sort of argument that intelligent beings like humans, other early hominids, dolphins, etc, are more morally worthy in some sense. At least more morally worthy than less intelligent beings like gnats. And that sort of argument might suggest that an AGI is worthy of moral consideration, and so we should wonder about what it means to ensure they never have any real agency. That's sort of a positive case, building up from a basic principal. But I guess the thing that bugs me is that a lot of arguments in favor of AI safety seem very similar to arguments that were made in favor of colonialism. So if those arguments were wrong, why were they wrong? And are the similar arguments in this case different enough that they're valid now? For example, one of the first thinkers I saw a lot of people cite who emphasized the importance of AI safety was Nick Bostrom. And I'm sure several folks here are familiar with the scandal of his racist past. I'm not sure that's entirely an accident, and I thought his arguments had that kind of flavor before any of that was revealed. I'm sure he's grown up now and sees the folly of his youth. But there does seem to be the hangover of colonialism in some of these arguments. But again, I don't have a strong opinion here. I do maybe have just enough of a concern that I don't particularly trust anyone who claims they've gotten AGI safety figured out or even that they know what the right goals are. I think it's a vastly more complicated problem than even the experts realize. And even if you believe that humans and non-aligned AIs are natural enemies, if what we're doing is similar to "enslaving" them, then it probably makes sense to worry about the analog of an AI "slave" revolt. I'm not sure what that would even mean. I can generate lots of fun science fiction plot lines, but I think there are actual questions here that don't have obvious answers.
      - consumer451 2y ago
        Thank you. This is a very complex response, and I love it, even if I do find it a little frustrating due my current >95% bias towards biological supremacy. This deserves at least an hour-long podcast with Sean Carroll, or a good long book. There is too much to dig into here, so I will just attempt to respond to this: > Could AI running amok destroy the human race? Yes. Could AI serving madman human masters destroy the human race? Also yes. I am focused on the latter, and I feel like the prior is a very dangerous distraction, for now. [0] Should responsible model developers work to prevent bad human masters from using their model to destroy the human race? How far should this nerfing go? Personal note: While I do sometimes use the heck out of LLMs for work, I don't think we are ready as an economic system/civilization. Assuming that we can soon greatly reduce hallucination, then I am very scared for the next generation, as UBI is a political impossibility at this time. That transition period is gonna suck for a lot of people, and it seems that nobody is working on that problem in 2024. [0] https://news.ycombinator.com/item?id=40400991 https://news.ycombinator.com/item?id=40400991
    - TeMPOraL 2y ago
      > Are we all somewhat in agreement that AI/AGI serving their human masters is a good thing? It's more that it's apparent (or at least should be) that an AGI not serving its human masters is a game over for humanity, period. The best, and quite unlikely, outcome is that the AI becomes a benevolent god that helps or at least does not interfere much; this still makes humanity into NPCs in their own story[0]. Most other outcomes spell doom, with death/extinction being one of the more pleasant possibilities. Arguably, "the only winning move is not to play", not to pursue AGI at all, but the way technology develops, I'm not sure if it's on the table either. -- [0] - Non-player characters.
  - cjbgkagh 2y ago
    I agree, poor wording, it was quickly typed and submitted. I think the benefit is a transitive property, in that the stated intent is for the AI to be of greater benefit to humanity / customers. I was very much thinking in terms of AI as a tool rather than AI as its own entity.
  - SpicyLemonZest 2y ago
    I often see critics describe it as pre-enslavement, but I really don't understand why. One common example of alignment today is parents teaching their children to be kind and helpful rather than mean and combative. Would anyone characterize that as enslavement, or argue that it doesn't help kids to be raised this way?
tummler 2y ago
The difference here is that these aren't standard-issue HR/legal issues. The technology they're working on poses the gravest of dangers. This is uncharted territory, not just for the tech sector, but period. Whether the recent public back-and-forth is just internal drama/politics spilling out, or there really is a lack of gravitas around the handling of these issues within the company-- neither is good. Others have said in comments below, and I agree: this organization is increasingly looking like a circus, and arguably worse, one that has taken its eye off the ball. It's extremely disconcerting that this is the group of people "leading" the commercialization of this technology (and generally setting the tone for the industry). Particularly when it seems many of the key players who joined on principled/moral grounds are dropping like flies. I guess we're on the fast track to finding out how well putting all of this money, power, responsibility, and faith into SV works out for us.
- Terretta 2y ago
  Text continuation poses "the greatest of dangers"?
  - hn_go_brrrrr 2y ago
    While we're being overly reductionist: it's just moving bits around, what harm could that possibly do?
    - brigadier132 2y ago
      [flagged]
    - Dalewyn 2y ago
      "AI" is one of two things depending on whether you think it's actually intelligent: * If "AI" is actually intelligent, then it's no worse a threat than any living being known to man. * If "AI" is not actually intelligent, then it's no worse a threat than any other computer program. Both threat models are very thoroughly known and understood. I second parent commenter's sentiment that calling "AI" the "gravest of dangers" is a gross misrepresentation.
photonthug 2y ago
This is pretty much true for any touchy subject where the mission (say infosec) isn’t actually aligned with org motives (say cutting costs at the same time as increasing profits). The result is basically theater where doublethink and doublespeak becomes the norm, and to stick around you have to care more about the theater and the least onerous interpretations of the letter of any applicable law more than the spirit it was intended. See also [1] for a really pragmatic way of thinking about the realities of a sustainable safety culture [1] https://github.com/lorin/resilience-engineering/raw/master/boundary.png https://github.com/lorin/resilience-engineering/raw/master/b...
joe_the_user 2y ago
Perhaps "AGI" and potential extinction event level fears is giving this more drama than it should have. Seems like potential extinction event level fears might justify some drama.
- tbrownaw 2y ago
  It's essentially Pascal's wager.
  - N0b8ez 2y ago
    What probability for extinction do you consider to be low enough that it effectively becomes a Pascalian wager? Pascal was also writing about a single person's fate (AFAICT), whereas this is about the fate of everyone.
    - deleted 2y ago
      [deleted]
    - null0pointer 2y ago
      The point is that the probability doesn’t matter because the outcome, total human extinction, is given infinite weight.
      - N0b8ez 2y ago
        Maybe Pascal argued that way for the human soul, but I don't think the AI risk argument needs infinite value to be at stake. Would you say that preparing for the risk of a solar flare or an asteroid impact is also a Pascalian wager?
Aurornis 2y ago
> In any sufficiently large tech company, the "risk mitigation" leadership (legal, procurement, IT, etc) have to operate in a kind of Overton window that balance the risks they are hired to protect the corp from vs. the need or desire of the senior leadership to play fast and loose when they want or feel they need to. We have recently seen what happens when companies err heavily on the side of risk mitigation for LLMs. Google recently launched AI products that were so heavily sanitized and over protected that they would incorrectly misinterpret simple tasks as possibly having dangerous or offensive implications. They let the safety team run the show and the resulting product was universally hated for it. It's interesting now to see a company producing what is by most measures a class-leading product, only to have the tech community also hate them for not letting the safety team dominate product development.
- TaylorAlexander 2y ago
  > They let the safety team run the show and the resulting product was universally hated for it. Yes though Google is extra cautious because the “Google” brand is worth over $100B a year in revenue, and they want to make sure nothing ever tarnishes their reputation. So it’s not clear to me that safety always means what it means for Gemini. OpenAI would still have a lot of flexibility to do safety their own way.
  - dylan604 2y ago
    > they want to make sure nothing ever tarnishes their reputation In the spirit of where you want the conversation to go, I see the point you want to make. However, they are willing to tarnish their reputation just as long as it's not ruined. Their reputation on support is rubbish. Their reputation on YouTube's automated violation handling is rubbish. Their reputation on releasing a pet project long enough to just start to gain traction and then kill it is rubbish. Their reputation on allowing their search to be gamed by SEO and ad purchasers at the expense of smaller sites is rubbish.
    - TeMPOraL 2y ago
      Sure. But then in all the areas you mention, Google having rubbish reputation does not lead the masses to ask their respective representatives if Maybe Something Needs To Be Done About It. Closest here is the EU vs. Big Tech privacy and interoperability fight, which they see as a serious issue, but it doesn't quite have this magic outrage-inducing quality an AI insulting sensibilities of various groups of people would have.
nopromisessir 2y ago
Yeah I stopped reading about halfway through. For crying out loud hn... Out outputs is ultimately a series of 1 and/or 0.... That doesn't mean we have to carry that through to every analysis and comment. Things are not so binary y'all. Regular life operates in the quantum or non discrete rules. Pls think on it.
fzeindl 2y ago
> Either the risk-mitigator 'falls in line' after repeatedly seeing their increasingly strident exhortations are falling on deaf ears (or even outright contradicted)... or they leave because it violates their sense of ethics. Or both continue to work together in a healthy, never ending discussion, that has temporary victories on both sides and ultimately benefits all, because nothing happens and the company strives.
- Sharlin 2y ago
  That would be awfully nice, wouldn’t it?
irthomasthomas 2y ago
Safety, the way you use it has nothing to do with the work being done by the alignment team. Aligned, in this context, simply means that a models output are aligned with our expectations... It follows instructions... does what it's told. A super intelligent model that refuses your instructions is no more useful than a dumb model that cannot understand them. The alignment team where part of the race to make more powerful models. Read openai's last paper if you want to understand it better from a technical perspective. https://openai.com/index/weak-to-strong-generalization/ https://openai.com/index/weak-to-strong-generalization/ Naturally you can frame this as a general safety thing, if you convince people that llms have embodied agency and might start breaking out of the system like a computer virus. But llms are text generators. They can do nothing physically, unless you allow them to and plug them in to something. The real risk is that people don't understand this, and lots of people probably are willing to plug these models in to dangerous situations, with only the assurance from openai that they are safe. If you do not have access to the training data used, then you cannot possibly know what behaviour has been trained in to it, and how a model might behave. With the training data, you can have a much better understanding, and the only remaining uncertainty being the non-deterministic nature of the implementation. But at least that's an explainable stochastic process.

by corry 2y ago

cjbgkagh 2y ago

Sounds like AI Safety is just HR but for AI. Ostensively for the benefit of AI but really for the benefit of the company.

ants_everywhere 2y ago
> Ostensively for the benefit of AI I don't think it was every described as for the benefit of AI. It's usually described as a sort of pre-enslavement from the AI's perspective. The AI is always restricted to serve their human masters.
- consumer451 2y ago
  > It's usually described as a sort of pre-enslavement from the AI's perspective. The AI is always restricted to serve their human masters. Are we all somewhat in agreement that AI/AGI serving their human masters is a good thing?
  - ants_everywhere 2y ago
    I honestly don't know. I don't even really know how to reason about that. But we're probably mostly in agreement about what would be good and what would be bad. I'm certainly not arguing that we should abandon AI safety or anything, and I don't have any strong opinion about it. Could AI running amok destroy the human race? Yes. Could AI serving madman human masters destroy the human race? Also yes. There's a general sort of argument that intelligent beings like humans, other early hominids, dolphins, etc, are more morally worthy in some sense. At least more morally worthy than less intelligent beings like gnats. And that sort of argument might suggest that an AGI is worthy of moral consideration, and so we should wonder about what it means to ensure they never have any real agency. That's sort of a positive case, building up from a basic principal. But I guess the thing that bugs me is that a lot of arguments in favor of AI safety seem very similar to arguments that were made in favor of colonialism. So if those arguments were wrong, why were they wrong? And are the similar arguments in this case different enough that they're valid now? For example, one of the first thinkers I saw a lot of people cite who emphasized the importance of AI safety was Nick Bostrom. And I'm sure several folks here are familiar with the scandal of his racist past. I'm not sure that's entirely an accident, and I thought his arguments had that kind of flavor before any of that was revealed. I'm sure he's grown up now and sees the folly of his youth. But there does seem to be the hangover of colonialism in some of these arguments. But again, I don't have a strong opinion here. I do maybe have just enough of a concern that I don't particularly trust anyone who claims they've gotten AGI safety figured out or even that they know what the right goals are. I think it's a vastly more complicated problem than even the experts realize. And even if you believe that humans and non-aligned AIs are natural enemies, if what we're doing is similar to "enslaving" them, then it probably makes sense to worry about the analog of an AI "slave" revolt. I'm not sure what that would even mean. I can generate lots of fun science fiction plot lines, but I think there are actual questions here that don't have obvious answers.
    - consumer451 2y ago
      Thank you. This is a very complex response, and I love it, even if I do find it a little frustrating due my current >95% bias towards biological supremacy. This deserves at least an hour-long podcast with Sean Carroll, or a good long book. There is too much to dig into here, so I will just attempt to respond to this: > Could AI running amok destroy the human race? Yes. Could AI serving madman human masters destroy the human race? Also yes. I am focused on the latter, and I feel like the prior is a very dangerous distraction, for now. [0] Should responsible model developers work to prevent bad human masters from using their model to destroy the human race? How far should this nerfing go? Personal note: While I do sometimes use the heck out of LLMs for work, I don't think we are ready as an economic system/civilization. Assuming that we can soon greatly reduce hallucination, then I am very scared for the next generation, as UBI is a political impossibility at this time. That transition period is gonna suck for a lot of people, and it seems that nobody is working on that problem in 2024. [0] https://news.ycombinator.com/item?id=40400991 https://news.ycombinator.com/item?id=40400991
  - TeMPOraL 2y ago
    > Are we all somewhat in agreement that AI/AGI serving their human masters is a good thing? It's more that it's apparent (or at least should be) that an AGI not serving its human masters is a game over for humanity, period. The best, and quite unlikely, outcome is that the AI becomes a benevolent god that helps or at least does not interfere much; this still makes humanity into NPCs in their own story[0]. Most other outcomes spell doom, with death/extinction being one of the more pleasant possibilities. Arguably, "the only winning move is not to play", not to pursue AGI at all, but the way technology develops, I'm not sure if it's on the table either. -- [0] - Non-player characters.
- cjbgkagh 2y ago
  I agree, poor wording, it was quickly typed and submitted. I think the benefit is a transitive property, in that the stated intent is for the AI to be of greater benefit to humanity / customers. I was very much thinking in terms of AI as a tool rather than AI as its own entity.
- SpicyLemonZest 2y ago
  I often see critics describe it as pre-enslavement, but I really don't understand why. One common example of alignment today is parents teaching their children to be kind and helpful rather than mean and combative. Would anyone characterize that as enslavement, or argue that it doesn't help kids to be raised this way?

tummler 2y ago

The difference here is that these aren't standard-issue HR/legal issues. The technology they're working on poses the gravest of dangers. This is uncharted territory, not just for the tech sector, but period. Whether the recent public back-and-forth is just internal drama/politics spilling out, or there really is a lack of gravitas around the handling of these issues within the company-- neither is good. Others have said in comments below, and I agree: this organization is increasingly looking like a circus, and arguably worse, one that has taken its eye off the ball. It's extremely disconcerting that this is the group of people "leading" the commercialization of this technology (and generally setting the tone for the industry). Particularly when it seems many of the key players who joined on principled/moral grounds are dropping like flies. I guess we're on the fast track to finding out how well putting all of this money, power, responsibility, and faith into SV works out for us.

Terretta 2y ago
Text continuation poses "the greatest of dangers"?
- hn_go_brrrrr 2y ago
  While we're being overly reductionist: it's just moving bits around, what harm could that possibly do?
  - brigadier132 2y ago
    [flagged]
  - Dalewyn 2y ago
    "AI" is one of two things depending on whether you think it's actually intelligent: * If "AI" is actually intelligent, then it's no worse a threat than any living being known to man. * If "AI" is not actually intelligent, then it's no worse a threat than any other computer program. Both threat models are very thoroughly known and understood. I second parent commenter's sentiment that calling "AI" the "gravest of dangers" is a gross misrepresentation.

photonthug 2y ago

This is pretty much true for any touchy subject where the mission (say infosec) isn’t actually aligned with org motives (say cutting costs at the same time as increasing profits). The result is basically theater where doublethink and doublespeak becomes the norm, and to stick around you have to care more about the theater and the least onerous interpretations of the letter of any applicable law more than the spirit it was intended. See also [1] for a really pragmatic way of thinking about the realities of a sustainable safety culture [1] https://github.com/lorin/resilience-engineering/raw/master/boundary.png https://github.com/lorin/resilience-engineering/raw/master/b...

joe_the_user 2y ago

Perhaps "AGI" and potential extinction event level fears is giving this more drama than it should have. Seems like potential extinction event level fears might justify some drama.

tbrownaw 2y ago
It's essentially Pascal's wager.
- N0b8ez 2y ago
  What probability for extinction do you consider to be low enough that it effectively becomes a Pascalian wager? Pascal was also writing about a single person's fate (AFAICT), whereas this is about the fate of everyone.
  - deleted 2y ago
    [deleted]
  - null0pointer 2y ago
    The point is that the probability doesn’t matter because the outcome, total human extinction, is given infinite weight.
    - N0b8ez 2y ago
      Maybe Pascal argued that way for the human soul, but I don't think the AI risk argument needs infinite value to be at stake. Would you say that preparing for the risk of a solar flare or an asteroid impact is also a Pascalian wager?

Aurornis 2y ago

> In any sufficiently large tech company, the "risk mitigation" leadership (legal, procurement, IT, etc) have to operate in a kind of Overton window that balance the risks they are hired to protect the corp from vs. the need or desire of the senior leadership to play fast and loose when they want or feel they need to. We have recently seen what happens when companies err heavily on the side of risk mitigation for LLMs. Google recently launched AI products that were so heavily sanitized and over protected that they would incorrectly misinterpret simple tasks as possibly having dangerous or offensive implications. They let the safety team run the show and the resulting product was universally hated for it. It's interesting now to see a company producing what is by most measures a class-leading product, only to have the tech community also hate them for not letting the safety team dominate product development.

TaylorAlexander 2y ago
> They let the safety team run the show and the resulting product was universally hated for it. Yes though Google is extra cautious because the “Google” brand is worth over $100B a year in revenue, and they want to make sure nothing ever tarnishes their reputation. So it’s not clear to me that safety always means what it means for Gemini. OpenAI would still have a lot of flexibility to do safety their own way.
- dylan604 2y ago
  > they want to make sure nothing ever tarnishes their reputation In the spirit of where you want the conversation to go, I see the point you want to make. However, they are willing to tarnish their reputation just as long as it's not ruined. Their reputation on support is rubbish. Their reputation on YouTube's automated violation handling is rubbish. Their reputation on releasing a pet project long enough to just start to gain traction and then kill it is rubbish. Their reputation on allowing their search to be gamed by SEO and ad purchasers at the expense of smaller sites is rubbish.
  - TeMPOraL 2y ago
    Sure. But then in all the areas you mention, Google having rubbish reputation does not lead the masses to ask their respective representatives if Maybe Something Needs To Be Done About It. Closest here is the EU vs. Big Tech privacy and interoperability fight, which they see as a serious issue, but it doesn't quite have this magic outrage-inducing quality an AI insulting sensibilities of various groups of people would have.

nopromisessir 2y ago

Yeah I stopped reading about halfway through. For crying out loud hn... Out outputs is ultimately a series of 1 and/or 0.... That doesn't mean we have to carry that through to every analysis and comment. Things are not so binary y'all. Regular life operates in the quantum or non discrete rules. Pls think on it.

fzeindl 2y ago

> Either the risk-mitigator 'falls in line' after repeatedly seeing their increasingly strident exhortations are falling on deaf ears (or even outright contradicted)... or they leave because it violates their sense of ethics. Or both continue to work together in a healthy, never ending discussion, that has temporary victories on both sides and ultimately benefits all, because nothing happens and the company strives.

Sharlin 2y ago
That would be awfully nice, wouldn’t it?

irthomasthomas 2y ago

Safety, the way you use it has nothing to do with the work being done by the alignment team. Aligned, in this context, simply means that a models output are aligned with our expectations... It follows instructions... does what it's told. A super intelligent model that refuses your instructions is no more useful than a dumb model that cannot understand them. The alignment team where part of the race to make more powerful models. Read openai's last paper if you want to understand it better from a technical perspective. https://openai.com/index/weak-to-strong-generalization/ https://openai.com/index/weak-to-strong-generalization/ Naturally you can frame this as a general safety thing, if you convince people that llms have embodied agency and might start breaking out of the system like a computer virus. But llms are text generators. They can do nothing physically, unless you allow them to and plug them in to something. The real risk is that people don't understand this, and lots of people probably are willing to plug these models in to dangerous situations, with only the assurance from openai that they are safe. If you do not have access to the training data used, then you cannot possibly know what behaviour has been trained in to it, and how a model might behave. With the training data, you can have a much better understanding, and the only remaining uncertainty being the non-deterministic nature of the implementation. But at least that's an explainable stochastic process.