4 ms·
What they did is cool and impressive but that's an extreme oversimplification. Building a small version of something that already exits, has code you can look
by Tenoke 5y ago
What they did is cool and impressive but that's an extreme oversimplification.
Building a small version of something that already exits, has code you can look at (gpt2), paper etc. is going to always be cheaper and easier even if we ignore that OpenAI work on and publish more than gpt. This is before even mentioning that they have an enormous amount of free compute from Google (and others) while Azure credits is a lot of what the 1bn investment to OA included.
- Eridrus 5y agoThis is clearly true, but this is also far cheaper that we could get this capability if we were to really staff a project. If there are volunteers who are capable of and want to contribute to the information commons and are merely constrained by compute, we should clearly help them.
- sivakon 5y agoGPT-J is trained on TPU VM generously offered by Google as part of their TPU research cloud. It’s a small version true but the results are almost on par with reported GPT-3 Source - https://github.com/kingoflolz/mesh-transformer-jax https://github.com/kingoflolz/mesh-transformer-jax
- nmfisher 5y agoIt's more that, if "amateurs"/volunteers can replicate this, it means OpenAI doesn't have much of a moat. (OK, I admit that it wouldn't have been possible without Google's donated TPUs. But the cost isn't so outrageous that it's out of reach for any reasonably funded startup).
- stellaathena 5y agoIf anyone has 300k and wants to buy a 20B parameter language model, custom trained on your data, let me know.