3 ms·

Is it possible to download and play with the trained models? I know that training the model would be absurdly expensive, so I'm curious if it is possible to do

Is it possible to download and play with the trained models? I know that training the model would be absurdly expensive, so I'm curious if it is possible to download one of the already trained artifacts of one of the models. I can't find anything online or in either of the HN threads on GPT-3

IceWreck 6y ago
They dont release them saying they would be too dangerous in a world where people believe everything on social media.
- tasogare 6y ago
  Who will believe that? They don’t release it because it’s not up to the hype. If a research is successful one should have nothing to hide.
  - wizzwizz4 6y ago
    People don't release the plans to nuclear bombs. Is that because they're not up to the hype, or because they could have devastating effects on society if built?
    - tasogare 6y ago
      GTP has nothing near the devastating power of nuclear weapon...
      - wizzwizz4 6y ago
        A nuke can't change your mind.
stephenroller 6y ago
No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future. On another note: At 175B parameters, with float16 representations, the in memory footprint is about 350GB plus activations would take it to another 400GB. You would need 12 or 13 V100GB GPUs to hold it in memory, or three p3.8xlarge. Meaning loading it on AWS would cost around $35-40/hr. Though if you didn't care about speed, you could load up the weights from disk one at a time and forward through it a few layers at a time on a single GPU.
- freeqaz 6y ago
  $35-40 an hour is well within the range of a "that sounds fun to grab my friends and mess around with it for a few hours on the weekend" budget! Especially if you can use spot instances or a cheaper cloud host. But I guess without the weights, the floor for this is several thousand dollars to play around with. Do you know if the data set is being released?
  - stephenroller 6y ago
    The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia. You can find a very comparable corpus open sourced and easy to use on the [T5 repo](https://github.com/google-research/text-to-text-transfer-transformer#c4 https://github.com/google-research/text-to-text-transfer-tra...)

IceWreck 6y ago

They dont release them saying they would be too dangerous in a world where people believe everything on social media.

tasogare 6y ago
Who will believe that? They don’t release it because it’s not up to the hype. If a research is successful one should have nothing to hide.
- wizzwizz4 6y ago
  People don't release the plans to nuclear bombs. Is that because they're not up to the hype, or because they could have devastating effects on society if built?
  - tasogare 6y ago
    GTP has nothing near the devastating power of nuclear weapon...
    - wizzwizz4 6y ago
      A nuke can't change your mind.

stephenroller 6y ago

No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future. On another note: At 175B parameters, with float16 representations, the in memory footprint is about 350GB plus activations would take it to another 400GB. You would need 12 or 13 V100GB GPUs to hold it in memory, or three p3.8xlarge. Meaning loading it on AWS would cost around $35-40/hr. Though if you didn't care about speed, you could load up the weights from disk one at a time and forward through it a few layers at a time on a single GPU.

freeqaz 6y ago
$35-40 an hour is well within the range of a "that sounds fun to grab my friends and mess around with it for a few hours on the weekend" budget! Especially if you can use spot instances or a cheaper cloud host. But I guess without the weights, the floor for this is several thousand dollars to play around with. Do you know if the data set is being released?
- stephenroller 6y ago
  The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia. You can find a very comparable corpus open sourced and easy to use on the [T5 repo](https://github.com/google-research/text-to-text-transfer-transformer#c4 https://github.com/google-research/text-to-text-transfer-tra...)