4 ms·

Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the

Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the data only once“ approach which is just wasteful. Compute challenges are more real, but we are seeing for the first time huge amounts of global capital being allocated to solve specifically these problems, so I am curious what fruit that will bear in a few years. I mean already the stuff that some of these low level people are doing is absolutely nuts. Tim Dettmers work training with only 4 bits means only 16 possible values per weight and still getting great results.

nextworddev 3y ago
Yep. We are in the very early innings of capital being deployed to all this.
- sgt101 3y ago
  Ok - what's the ROI on the $10bn (++) that OpenAI have had? So far I reckon <$10m in actual revenue. This isn't what VC's (or microsoft) dream of.
  - nextworddev 3y ago
    I think Azure OpenAI service is growing at 1000% per quarter according to last earnings call
    - TeMPOraL 3y ago
      Indeed. Azure OpenAI service is how you get corporate-blessed ChatGPT that you can use with proprietary information, among other things. There's a huge demand for it.
  - Quarrelsome 3y ago
    The level of exposure since chatGPT has/will result in a lot of money turning up, especially for applications of the existing technology (whether they succeed or fail). That stats on usage demonstrate that the thundering herd has noticed and that attention can be extremely valuable. I think its quite likely that OpenAI will make that money back and more, as both the industry leader and with the power of their brand (chatGPT).
  - letitgo12345 3y ago
    OAI revenue is 100m+ at least right now iirc and probably gonna grow quite a bit
  - justrealist 3y ago
    That would be 500,000 people buying 1 month of GPT-4. I think they blew past that in the first week.
  - deleted 3y ago
    [deleted]
PartiallyTyped 3y ago
> using the “show the model the data only once“ approach which is just wasteful. According to the InstructGPT paper, that is not the case, showing the data multiple times results in overfitting.
- whimsicalism 3y ago
  1. You are just referring to fine tuning, I am referring to training the base model. 2. They still saw performance improvements which is why they did train on the data multiple times, you can see in the paper. 3. there was a recent paper demonstrating that reusing data still saw continued improvements in perplexity, i am on my ipad so cannot find it now
  - PartiallyTyped 3y ago
    > 1. You are just referring to fine tuning, I am referring to training the base model. Ahh mb! Sorry.

nextworddev 3y ago

Yep. We are in the very early innings of capital being deployed to all this.

sgt101 3y ago
Ok - what's the ROI on the $10bn (++) that OpenAI have had? So far I reckon <$10m in actual revenue. This isn't what VC's (or microsoft) dream of.
- nextworddev 3y ago
  I think Azure OpenAI service is growing at 1000% per quarter according to last earnings call
  - TeMPOraL 3y ago
    Indeed. Azure OpenAI service is how you get corporate-blessed ChatGPT that you can use with proprietary information, among other things. There's a huge demand for it.
- Quarrelsome 3y ago
  The level of exposure since chatGPT has/will result in a lot of money turning up, especially for applications of the existing technology (whether they succeed or fail). That stats on usage demonstrate that the thundering herd has noticed and that attention can be extremely valuable. I think its quite likely that OpenAI will make that money back and more, as both the industry leader and with the power of their brand (chatGPT).
- letitgo12345 3y ago
  OAI revenue is 100m+ at least right now iirc and probably gonna grow quite a bit
- justrealist 3y ago
  That would be 500,000 people buying 1 month of GPT-4. I think they blew past that in the first week.
- deleted 3y ago
  [deleted]

PartiallyTyped 3y ago

> using the “show the model the data only once“ approach which is just wasteful. According to the InstructGPT paper, that is not the case, showing the data multiple times results in overfitting.

whimsicalism 3y ago
1. You are just referring to fine tuning, I am referring to training the base model. 2. They still saw performance improvements which is why they did train on the data multiple times, you can see in the paper. 3. there was a recent paper demonstrating that reusing data still saw continued improvements in perplexity, i am on my ipad so cannot find it now
- PartiallyTyped 3y ago
  > 1. You are just referring to fine tuning, I am referring to training the base model. Ahh mb! Sorry.