4 ms·

AI safety researcher here. I've interacted with several folks at OpenAI, DeepMind, and other such groups, but have no affiliation with any of them. While I'm n

AI safety researcher here. I've interacted with several folks at OpenAI, DeepMind, and other such groups, but have no affiliation with any of them. While I'm not privy to OpenAI's internal motivations, it's worth keeping in mind that as a nonprofit they face a pair of hard organizational constraints: 1) Training state-of-the-art ML models costs huge amounts of money. That means OpenAI's capex is forcibly high. 2) Their charter (https://openai.com/charter/ https://openai.com/charter/) commits them to a "fiduciary duty to humanity", so among other things they can't accept donations which are subject to conditions that may be misaligned with their mission. That means OpenAI's effective funding pool is forcibly low. The combination of 1) and 2) means that, absent an absolutely massive yet hands-off private donor — a rare thing indeed in the nonprofit world — it's hard to imagine a solution to this dilemma that doesn't involve some kind of for-profit spinoff. And the outcome here really is pretty binary: if you believe OpenAI will build safe AGI, then 100x returns for investors really are immaterial compared to the wealth they can expect to generate for everyone else. While if you don't believe they'll build safe AGI, then you must either believe that A) they'll fail to build AGI (in which case investors get nothing anyway); or that B) they'll fail to build it safely (in which case your concerns are presumably technical rather than organizational). Incidentally, it's quite likely that in the beginning, OpenAI's own researchers were by no means sure they'd be able to get results like GPT-3 through scaling alone. But that turned out to be the world we live in, so they're now playing the hand they've been dealt.

catillac 5y ago
What does AI Safety mean in this context?
- ceejayoz 5y ago
  Ensuring AIs don't wind up wanting to destroy humanity.
  - adventured 5y ago
    Wanting is the easier of the two - the other is ensuring AIs don't inadvertently wind up destroying humanity. The road to hell being paved with good intentions and all that.
  - deleted 5y ago
    [deleted]
  - d13 5y ago
    “Want” is a problem for me here. GPT-3, for example, is just a dumb, brute force Markhov chain. Any danger it might pose is no more than that of a mechanical trigger that sits between a finger and gun barrel. The only real intelligence in the system, and where the only danger lies, is from the person behind the gun.
    - danpalmer 5y ago
      GPT-3 is a bit more than a Markhov chain. There is a lot more to it than just being big. Same for GPT-2.
    - suyjuris 5y ago
      As soon as you put the intelligence in AI this changes. It is easy to say that current AI systems are “dumb” (although note that the precise meaning of dumb has changed significantly over the last few decades). You can say that about any AI with sub-human intelligence. But if you reach human levels, you can likely reach super-human as well, so you need to start worrying much earlier.
    - Jenda_ 5y ago
      GPT-3 is better than everything what was achieved so far in multiple metrics, though yes, it's of course not an "intelligence". However, this is ignoring possible future developments - things may get better and more advanced and that's what AI-safety people are concerned about.
skybrian 5y ago
This idea that either they build AGI or nothing seems like a false binary to me? Like other startups, there’s a chance that that they build something interesting and profitable that isn’t AGI. (I’m thinking that the odds of that are higher than those for AGI.)
- edouard-harris 5y ago
  I agree that's a legitimate third possibility, but I don't see it as likely compared to the others. Pivoting is extraordinarily hard when you have 100+ employees and a very visible public commitment to your mission. Given that OAI filters recruits partly for their alignment with the charter, a full pivot to, e.g., a for-profit SaaS or something seems pretty much out of the question — if only because they'd probably lose a ton of employees in the process.
joe_the_user 5y ago
A couple serious, unbiased questions: 1) How seriously is AI safety research taken by academia? Are there grants? Academic positions? Sponsored studies etc? 2) How much interaction is there people who study the long terms dangers of strong AI and people who look at the problems and biases of existing AI systems (say, analyzes of racial in AI bond-granting systems, etc). ... Full disclosure, I have something of a love-hate relationship with "AI safety" positions. It seems you have a series of intertwined problems. So now some biased, editorializing points. * Exactly what General AI is, what it can do and so-forth, is not known, not understood. So it's hard to either know or articulate potential dangers. Often the stand-in for this unknown is taking an AI as a god-like with an unknown agenda. This often results in contradictory and implausible worries that it's hard to take serious, so-and-so's basilisk , the "terrifying" that's Pascal's Wager warmed-over. * A primary focus of AI research is "alignment" between AIs and humans. This has the fundamental problem that humans are often not aligned with other humans. Humans don't always engage in genocidal world wars but we certainly know times when they have. AI researchers like Paul Christiano don't seem particular concerned with problem of "aligning" humans with each other but seems like a pressing question. * But let's take "AI dangers" seriously. Let's look at an analogous situation. Suppose there's a new kind of energy with the potential for massive payoff but with some kind of dangers (pollution, explosions, whatever). It's weird to say "to solve this danger, we'll create a safe form first". A rational response to some dangerous activity would be regulation for everyone creating it.
Widdershin 5y ago
There are other ways around the cost. They could build a distributed training system along the lines of Folding@Home, I’m sure a lot of AI folks would donate GPU time.
- boulos 5y ago
  The optimization problem at the inner loop of these methods doesn't handle latency very well. This is covered a bit in the DOTA 2 rerun work [1] (e.g., Figure 5b on staleness). So for both large models (which are split up amongst multiple accelerators but effectively one "computer"), as well as learning from "your friends" you want to be more tightly coupled than something like seti/folding@home. There are plenty of workloads that are great for embarrassingly parallel (and now "pleasingly parallel") workloads, but training a single model isn't one of them. [1] https://cdn.openai.com/dota-2.pdf https://cdn.openai.com/dota-2.pdf
alpineidyll3 5y ago
The claim that OpenAI needed the money seems flimsy in light of gptj

catillac 5y ago

What does AI Safety mean in this context?

ceejayoz 5y ago
Ensuring AIs don't wind up wanting to destroy humanity.
- adventured 5y ago
  Wanting is the easier of the two - the other is ensuring AIs don't inadvertently wind up destroying humanity. The road to hell being paved with good intentions and all that.
- deleted 5y ago
  [deleted]
- d13 5y ago
  “Want” is a problem for me here. GPT-3, for example, is just a dumb, brute force Markhov chain. Any danger it might pose is no more than that of a mechanical trigger that sits between a finger and gun barrel. The only real intelligence in the system, and where the only danger lies, is from the person behind the gun.
  - danpalmer 5y ago
    GPT-3 is a bit more than a Markhov chain. There is a lot more to it than just being big. Same for GPT-2.
  - suyjuris 5y ago
    As soon as you put the intelligence in AI this changes. It is easy to say that current AI systems are “dumb” (although note that the precise meaning of dumb has changed significantly over the last few decades). You can say that about any AI with sub-human intelligence. But if you reach human levels, you can likely reach super-human as well, so you need to start worrying much earlier.
  - Jenda_ 5y ago
    GPT-3 is better than everything what was achieved so far in multiple metrics, though yes, it's of course not an "intelligence". However, this is ignoring possible future developments - things may get better and more advanced and that's what AI-safety people are concerned about.

skybrian 5y ago

This idea that either they build AGI or nothing seems like a false binary to me? Like other startups, there’s a chance that that they build something interesting and profitable that isn’t AGI. (I’m thinking that the odds of that are higher than those for AGI.)

edouard-harris 5y ago
I agree that's a legitimate third possibility, but I don't see it as likely compared to the others. Pivoting is extraordinarily hard when you have 100+ employees and a very visible public commitment to your mission. Given that OAI filters recruits partly for their alignment with the charter, a full pivot to, e.g., a for-profit SaaS or something seems pretty much out of the question — if only because they'd probably lose a ton of employees in the process.

joe_the_user 5y ago

A couple serious, unbiased questions: 1) How seriously is AI safety research taken by academia? Are there grants? Academic positions? Sponsored studies etc? 2) How much interaction is there people who study the long terms dangers of strong AI and people who look at the problems and biases of existing AI systems (say, analyzes of racial in AI bond-granting systems, etc). ... Full disclosure, I have something of a love-hate relationship with "AI safety" positions. It seems you have a series of intertwined problems. So now some biased, editorializing points. * Exactly what General AI is, what it can do and so-forth, is not known, not understood. So it's hard to either know or articulate potential dangers. Often the stand-in for this unknown is taking an AI as a god-like with an unknown agenda. This often results in contradictory and implausible worries that it's hard to take serious, so-and-so's basilisk , the "terrifying" that's Pascal's Wager warmed-over. * A primary focus of AI research is "alignment" between AIs and humans. This has the fundamental problem that humans are often not aligned with other humans. Humans don't always engage in genocidal world wars but we certainly know times when they have. AI researchers like Paul Christiano don't seem particular concerned with problem of "aligning" humans with each other but seems like a pressing question. * But let's take "AI dangers" seriously. Let's look at an analogous situation. Suppose there's a new kind of energy with the potential for massive payoff but with some kind of dangers (pollution, explosions, whatever). It's weird to say "to solve this danger, we'll create a safe form first". A rational response to some dangerous activity would be regulation for everyone creating it.

Widdershin 5y ago

There are other ways around the cost. They could build a distributed training system along the lines of Folding@Home, I’m sure a lot of AI folks would donate GPU time.

boulos 5y ago
The optimization problem at the inner loop of these methods doesn't handle latency very well. This is covered a bit in the DOTA 2 rerun work [1] (e.g., Figure 5b on staleness). So for both large models (which are split up amongst multiple accelerators but effectively one "computer"), as well as learning from "your friends" you want to be more tightly coupled than something like seti/folding@home. There are plenty of workloads that are great for embarrassingly parallel (and now "pleasingly parallel") workloads, but training a single model isn't one of them. [1] https://cdn.openai.com/dota-2.pdf https://cdn.openai.com/dota-2.pdf

alpineidyll3 5y ago

The claim that OpenAI needed the money seems flimsy in light of gptj