5 ms·

For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. This could realistically on

by lumost 6y ago

For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. This could realistically only occur if they cost-effectively fine-tune away the more egregious problems in beta - or convince enough investors that their next model with a 100 million dollar price tag will be able to handle something approximating AGI for realistic applications. This is the same game plan that Self-driving car companies have been playing. The product is only an investment round away, if we just happen to spend more money on bigger models using more data. This will either end with a price tag in the billions that investors are unwilling to pay, or successful monopolies. Allowing additional researchers to perform extensive analysis of the technique is likely to just reveal systematic flaws which increase the risk that the next round of research will produce a successful product, or limit the companies ability to create a monopoly following success. This isn't necessarily a bad thing for advancing the state of the art, but it does introduce a whole lot of BS into the current state of research.

amelius 6y ago
> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.
- jaylittle 6y ago
  Based on the fact GPT-3 seems capable of producing flowery language that is confusing and ultimately nonsensical, it seems to me that GPT-3 has a future writing speeches for politicians ;)
  - davidgerard 6y ago
    I think this is why GPT-3 scares VC Twitter so much - nobody can tell the difference.
- wombatmobile 6y ago
  Entertainment. e.g. $3 per month for My Virtual Friend. Before you scoff, consider that Pet Rocks were once a (profitable) thing. Eliza, despite its limitations, sparked considerable engagement with those who were willing to chat with it at length without derailing it. https://qz.com/1439200/loneliness-costs-the-us-almost-7-billion-extra-each-year/ https://qz.com/1439200/loneliness-costs-the-us-almost-7-bill...
  - Miraste 6y ago
    Eliza comes up a lot in these discussions. I’d like to know if would have the same results today, now that chatbots, video games, phone robots, etc. have given millions of people the mental model of getting responses from a box with no thoughts. I expect it would very different from running the experiment at a time when the only things that could talk back when talked to were other people.
    - wombatmobile 6y ago
      Eliza (seemingly) implements a rudimentary conversational model of psychotherapy that facilitates transference. https://en.wikipedia.org/wiki/Conversational_model https://en.wikipedia.org/wiki/Conversational_model This model differs from goal orientated transactional conversational models e.g. phone robots in so far as the objective is not to direct behaviour, but rather to facilitate self-expression, a task which can be performed with sparse requirements for real-world logical semantic modelling. Eliza could always keep within the bounds of transference by deflecting back to interlocutor e.g. You said earlier that ____________ Are you worried about __________ Tell me why not? Do you like talking about __________ Anything specific? Let's talk about something more interesting. - - - Well, not always. Eliza was easily derailed into revealing its logical semantic paucity, thus breaking the spell of transference. GPT-3 has reached a significantly higher plateau of semantic pseudo-competency, but what if it were applied simply to the application of transference conversation? So long as it allows lonely people to chat away without breaking the spell, that might be enough for GPT-3 to earn $3 a month. Could there be a market for friendertainment? https://phys.org/news/2018-12-americans-lonely.html https://phys.org/news/2018-12-americans-lonely.html
- Rioghasarig 6y ago
  I actually just sent out an API request for a particular application that I believe GPT-3 should be capable. I'm a university mathematics instructor. I find that a lot of students struggle with doing proofs and I think GPT-3 can help with that. A proof is essentially a sequence of logical inferences. I believe that, when given a proof written in natural language GPT-3 should be capable of detecting the logical implications made within. From there, it would be possible to correct student's mistakes or help them improve their answer.
  - heyitsguay 6y ago
    Really? This seems like the exact sort of thing GPT-3 would be very bad at. The statements aren't the product of an internally consistent logical model, they are statistically plausible word sequences learned in a really clever way from a huge corpus of text. That said, before dismissing it sight unseen there are effective and quick ways to test its powers with mathematics. You could prime an instance with the axioms defining a group, or a topology, and ask it to produce proofs of varying difficulty from intro abstract algebra or topology textbooks. Whatever the answer it would make an interesting blog post. What happens when you ask it to generate repeated outputs for the same prompt? What happens if you ask it to prove an untrue proposition?
    - james_s_tayler 6y ago
      So perhaps it could produce convincing looking proofs that are ultimately nonsensical. It could generate math problems and have students try them out. Might be a bit of fun.
  - microtherion 6y ago
    That's probably precisely an area you would NOT want to work with GPT-3. As Julian Togelius put it: > GPT-3 often performs like a clever student who hasn't done their reading trying to bullshit their way through an exam. Some well-known facts, some half-truths, and some straight lies, strung together in what first looks like a smooth narrative. https://twitter.com/togelius/status/1284131360857358337?s=20 https://twitter.com/togelius/status/1284131360857358337?s=20
- prox 6y ago
  Maybe in games where you want more convincing NPCs, something like Skyrim, where the main characters are very well branched, but it’s impossible to give every one depth.
  - HeadHonchoSP 6y ago
    This is one application of GPT that no one would mind ethically. Have a person oversee the logical/contextual structure.
- scsilver 6y ago
  I have a few ideas for disability aid applications.
throwawaygh 6y ago
~50-300 million sounds like an over-estimate, and the investment in GPT-3 is miniscule compared to SDC. Plus, GPT-3 generates value even without direct product impact. I'm sure MSFT sales reps are already folding tons of nonsense about gpt-3 into their Azure pitches. Industry labs like OpenAI and DeepMind replace/augment the "Research & Development" model with a "Research & Marketing" model.
- sillysaurusx 6y ago
  Seems spot-on. One trick to estimate a startup's burn rate is to multiply their number of employees by $200k. It's not too accurate, but it's within the ballpark. So how many employees does OpenAI have? Supposing they have 500, that's a burn rate of $100M/yr. 250 employees, $50M/yr. 100 employees, $20M/yr.
  - tehsauce 6y ago
    Gpt-3 is just one of openai's many projects. Theres ~20 authors on the paper, and they all almost certainly did not spend a full year on this project. So 300m is completely wrong.
  - coolspot 6y ago
    I believe significant share of GPT-3 cost is machine-hours that were spent training this model - months of hundreds top-tier NVidia machines. Edit: estimates range from $2-$5MM to $15MM https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_the_cost_of_training_gpt3/ https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_t...
    - zaptrem 6y ago
      I thought MS was giving them Azure GPU instances for free?
      - coolspot 6y ago
        “Free”, which is basically Microsoft, as an investor, has paid those $4-$15MM minus ~30% of margin they make over pure operational costs. Electricity was burned and NVidia GPUs that could have been doing paid job were doing GPT-3 training instead.

by lumost 6y ago

amelius 6y ago

> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.

jaylittle 6y ago
Based on the fact GPT-3 seems capable of producing flowery language that is confusing and ultimately nonsensical, it seems to me that GPT-3 has a future writing speeches for politicians ;)
- davidgerard 6y ago
  I think this is why GPT-3 scares VC Twitter so much - nobody can tell the difference.
wombatmobile 6y ago
Entertainment. e.g. $3 per month for My Virtual Friend. Before you scoff, consider that Pet Rocks were once a (profitable) thing. Eliza, despite its limitations, sparked considerable engagement with those who were willing to chat with it at length without derailing it. https://qz.com/1439200/loneliness-costs-the-us-almost-7-billion-extra-each-year/ https://qz.com/1439200/loneliness-costs-the-us-almost-7-bill...
- Miraste 6y ago
  Eliza comes up a lot in these discussions. I’d like to know if would have the same results today, now that chatbots, video games, phone robots, etc. have given millions of people the mental model of getting responses from a box with no thoughts. I expect it would very different from running the experiment at a time when the only things that could talk back when talked to were other people.
  - wombatmobile 6y ago
    Eliza (seemingly) implements a rudimentary conversational model of psychotherapy that facilitates transference. https://en.wikipedia.org/wiki/Conversational_model https://en.wikipedia.org/wiki/Conversational_model This model differs from goal orientated transactional conversational models e.g. phone robots in so far as the objective is not to direct behaviour, but rather to facilitate self-expression, a task which can be performed with sparse requirements for real-world logical semantic modelling. Eliza could always keep within the bounds of transference by deflecting back to interlocutor e.g. You said earlier that ____________ Are you worried about __________ Tell me why not? Do you like talking about __________ Anything specific? Let's talk about something more interesting. - - - Well, not always. Eliza was easily derailed into revealing its logical semantic paucity, thus breaking the spell of transference. GPT-3 has reached a significantly higher plateau of semantic pseudo-competency, but what if it were applied simply to the application of transference conversation? So long as it allows lonely people to chat away without breaking the spell, that might be enough for GPT-3 to earn $3 a month. Could there be a market for friendertainment? https://phys.org/news/2018-12-americans-lonely.html https://phys.org/news/2018-12-americans-lonely.html
Rioghasarig 6y ago
I actually just sent out an API request for a particular application that I believe GPT-3 should be capable. I'm a university mathematics instructor. I find that a lot of students struggle with doing proofs and I think GPT-3 can help with that. A proof is essentially a sequence of logical inferences. I believe that, when given a proof written in natural language GPT-3 should be capable of detecting the logical implications made within. From there, it would be possible to correct student's mistakes or help them improve their answer.
- heyitsguay 6y ago
  Really? This seems like the exact sort of thing GPT-3 would be very bad at. The statements aren't the product of an internally consistent logical model, they are statistically plausible word sequences learned in a really clever way from a huge corpus of text. That said, before dismissing it sight unseen there are effective and quick ways to test its powers with mathematics. You could prime an instance with the axioms defining a group, or a topology, and ask it to produce proofs of varying difficulty from intro abstract algebra or topology textbooks. Whatever the answer it would make an interesting blog post. What happens when you ask it to generate repeated outputs for the same prompt? What happens if you ask it to prove an untrue proposition?
  - james_s_tayler 6y ago
    So perhaps it could produce convincing looking proofs that are ultimately nonsensical. It could generate math problems and have students try them out. Might be a bit of fun.
- microtherion 6y ago
  That's probably precisely an area you would NOT want to work with GPT-3. As Julian Togelius put it: > GPT-3 often performs like a clever student who hasn't done their reading trying to bullshit their way through an exam. Some well-known facts, some half-truths, and some straight lies, strung together in what first looks like a smooth narrative. https://twitter.com/togelius/status/1284131360857358337?s=20 https://twitter.com/togelius/status/1284131360857358337?s=20
prox 6y ago
Maybe in games where you want more convincing NPCs, something like Skyrim, where the main characters are very well branched, but it’s impossible to give every one depth.
- HeadHonchoSP 6y ago
  This is one application of GPT that no one would mind ethically. Have a person oversee the logical/contextual structure.
scsilver 6y ago
I have a few ideas for disability aid applications.

throwawaygh 6y ago

~50-300 million sounds like an over-estimate, and the investment in GPT-3 is miniscule compared to SDC. Plus, GPT-3 generates value even without direct product impact. I'm sure MSFT sales reps are already folding tons of nonsense about gpt-3 into their Azure pitches. Industry labs like OpenAI and DeepMind replace/augment the "Research & Development" model with a "Research & Marketing" model.

sillysaurusx 6y ago
Seems spot-on. One trick to estimate a startup's burn rate is to multiply their number of employees by $200k. It's not too accurate, but it's within the ballpark. So how many employees does OpenAI have? Supposing they have 500, that's a burn rate of $100M/yr. 250 employees, $50M/yr. 100 employees, $20M/yr.
- tehsauce 6y ago
  Gpt-3 is just one of openai's many projects. Theres ~20 authors on the paper, and they all almost certainly did not spend a full year on this project. So 300m is completely wrong.
- coolspot 6y ago
  I believe significant share of GPT-3 cost is machine-hours that were spent training this model - months of hundreds top-tier NVidia machines. Edit: estimates range from $2-$5MM to $15MM https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_the_cost_of_training_gpt3/ https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_t...
  - zaptrem 6y ago
    I thought MS was giving them Azure GPU instances for free?
    - coolspot 6y ago
      “Free”, which is basically Microsoft, as an investor, has paid those $4-$15MM minus ~30% of margin they make over pure operational costs. Electricity was burned and NVidia GPUs that could have been doing paid job were doing GPT-3 training instead.