5 ms·

I often wonder if OpenAIs decision not to open gpt-3 was because it was to expensive to train relative to its real value. They’ve hidden the model behind an ap

by lumost 5y ago

I often wonder if OpenAIs decision not to open gpt-3 was because it was to expensive to train relative to its real value. They’ve hidden the model behind an api where they can filter out most of the dumb behaviors, while everyone believes they are working on something entirely different.

teaearlgraycold 5y ago
> They’ve hidden the model behind an api where they can filter out most of the dumb behaviors What do you mean by this?
- codebolt 5y ago
  Things like cobbling on a bunch of heuristic rule-based behaviours that wouldn't look good in the public repo of a supposed quasi-AGI system?
- lumost 5y ago
  There is some evidence that the OpenAI GPT-3 APIs have a human in the loop for bad examples. They may also have a number of filters to exclude certain words/patterns/other rules. The challenge with such rule and human in the loop systems is that the long-tail of these problems is huge, and fat. Meaning that you generally can't make a product which doesn't have full generalization. That it took ~1.5 years to open the GPT-3 API inclines me to think that they've run into similar problems. We're also not seeing the long pitched swarm of GPT enabled content despite the API being open for ~10 months.
  - teaearlgraycold 5y ago
    There’s no way they have a human in the loop. The model spits out tokens one at a time. You can see that with the stream flag set to true. The latency doesn’t allow for human intervention. They do have API parameters for tweaking repetitiveness. That might be what you’re talking about - but it’s fair to call the model and an external repetition filter part of the same product. As for word filters - no. If they did they’d not be sending back explicit content. But they do. If you have a gpt-3 product you’re obligated to run each result through their content filter to filter out anything nsfw. We don’t see a ton of gpt-3 enabled content because writing good gpt-3 prompts is hard. You’re trying to learn how this black box works with almost no examples to go off of. I worked for a gpt-3 startup and we put someone on prompt writing full time to get the most out of it. Most startups wouldn’t think to do that and won’t want to.
alimov 5y ago
I would like to know about the reason behind this as well.
zarzavat 5y ago
Didn’t they sell an exclusive license to Microsoft? It’s probably just a contractural issue.
- lumost 5y ago
  That happened after they decided not to release the model.
  - Jensson 5y ago
    So their goal was to become the next IBM Watson? Parade around tech and try to create hype and hope for the future around it, while hiding all the dirty secrets that shows how limited the technology really is. Their original reasoning for not releasing it "this model is too dangerous to be released to the public" felt very much like a marketing stunt.
    - jaimex2 5y ago
      It does feel like the Tesla FSD playbook citing "pending regulatory approval"
  - cortesoft 5y ago
    Well, the decision not to release the model might have been made so that they could license it instead.
    - Jensson 5y ago
      They gave a reason why they didn't release it to the public, they said it was too dangerous: https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction https://www.theguardian.com/technology/2019/feb/14/elon-musk... But of course then they started selling it to the highest bidder, so I wouldn't really trust what they say. They aren't "OpenAI", at this point they are just regular "ProprietaryAI". I really wonder what goal Elon Musk have with it.
      - 5y ago

by lumost 5y ago

teaearlgraycold 5y ago

> They’ve hidden the model behind an api where they can filter out most of the dumb behaviors What do you mean by this?

codebolt 5y ago
Things like cobbling on a bunch of heuristic rule-based behaviours that wouldn't look good in the public repo of a supposed quasi-AGI system?
lumost 5y ago
There is some evidence that the OpenAI GPT-3 APIs have a human in the loop for bad examples. They may also have a number of filters to exclude certain words/patterns/other rules. The challenge with such rule and human in the loop systems is that the long-tail of these problems is huge, and fat. Meaning that you generally can't make a product which doesn't have full generalization. That it took ~1.5 years to open the GPT-3 API inclines me to think that they've run into similar problems. We're also not seeing the long pitched swarm of GPT enabled content despite the API being open for ~10 months.
- teaearlgraycold 5y ago
  There’s no way they have a human in the loop. The model spits out tokens one at a time. You can see that with the stream flag set to true. The latency doesn’t allow for human intervention. They do have API parameters for tweaking repetitiveness. That might be what you’re talking about - but it’s fair to call the model and an external repetition filter part of the same product. As for word filters - no. If they did they’d not be sending back explicit content. But they do. If you have a gpt-3 product you’re obligated to run each result through their content filter to filter out anything nsfw. We don’t see a ton of gpt-3 enabled content because writing good gpt-3 prompts is hard. You’re trying to learn how this black box works with almost no examples to go off of. I worked for a gpt-3 startup and we put someone on prompt writing full time to get the most out of it. Most startups wouldn’t think to do that and won’t want to.

alimov 5y ago

I would like to know about the reason behind this as well.

zarzavat 5y ago

Didn’t they sell an exclusive license to Microsoft? It’s probably just a contractural issue.

lumost 5y ago
That happened after they decided not to release the model.
- Jensson 5y ago
  So their goal was to become the next IBM Watson? Parade around tech and try to create hype and hope for the future around it, while hiding all the dirty secrets that shows how limited the technology really is. Their original reasoning for not releasing it "this model is too dangerous to be released to the public" felt very much like a marketing stunt.
  - jaimex2 5y ago
    It does feel like the Tesla FSD playbook citing "pending regulatory approval"
- cortesoft 5y ago
  Well, the decision not to release the model might have been made so that they could license it instead.
  - Jensson 5y ago
    They gave a reason why they didn't release it to the public, they said it was too dangerous: https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction https://www.theguardian.com/technology/2019/feb/14/elon-musk... But of course then they started selling it to the highest bidder, so I wouldn't really trust what they say. They aren't "OpenAI", at this point they are just regular "ProprietaryAI". I really wonder what goal Elon Musk have with it.
    - 5y ago