4 ms·

> is anyone doing so to your knowledge? current barrier as far as i understand is that it costs an estimated 4.6MM USD in compute to recreate. That I know of,

> is anyone doing so to your knowledge? current barrier as far as i understand is that it costs an estimated 4.6MM USD in compute to recreate. That I know of, no one is looking into doing this in the open source community no. I think it's simply too hard and costly. As I said in another comment, even if an open source org. managed to raise 4.6M$, they still wouldn't be able to do it as the infrastructure needed to train a model of that size is not available publicly on any cloud. The cluster Microsoft built for OpenAI is about 285k CPUs and 10k V100 GPUs, this is absolutly huge. From GPT-3's paper we know the training required 3640 PFLOPs/days of compute. A single V100 GPU is about 14 TFLOPs in half precision, so if you had 75 of those GPUs (which is already massive by open source standard, that's about 5 DGX-2 nodes), it would take about 10 years of training (by that same calculation, it still took about a month on their hypercluster if they used all the GPUs). And that is probably an optimistic estimate since I am using Nvidia's marketing slide to evaluate a V100 FLOPs. Even if somehow an open source organisation was able to get their hand on a cluster half the size of the one OpenAI used, by the time they figure out all the data scraping/cleaning part, found the correct hyper parameters by experimenting on smaller models, and developed the many other things and tricks you need to do to train those models (optimizers, correct checkpointing/resuming strategy etc.) I believe one of OpenAI/Microsoft/Google/Nvidia would already have (or be very close to) announcing the next bigger model, and then it's back to square 1. I think it's just too big a project (with too little usefuleness) to justify undertaking if you are not a massive organization. The closest open source org that could pull this off that I can think of would be HuggingFace. They have the skill and descend funding, and they have the incentive (I believe they also want to enter that NLP API space), but they don't have access to that kind of compute infrastructure (that I know of). They also have been open source so far, but nothing guarantees that would share the checkpoints of such a model should they train it.

chosen1111 6y ago
Great points. Thank you. Is something like this capable of being run as a distributed computing project, akin to something like folding at home? Are there any obstacles that would make recreating a GPT-3 via a distributed system impossible...other than GPT-4 coming out before completion of such a system?
- Voloskaya 6y ago
  I'm not an expert in distributed computing so take what I say with a grain of salt, but I believe it would be really hard. The premise of all the @home projects is that each compute unit can do some relatively large amount of work in isolation and then communicate the result back to a central server. If one of the client disconnects in the middle of it's computation, it can just be redistributed to another client. And it doesn't really matter if a client is very slow. For the same reason network latency barely matters in those scenarios. On the other hand, to train GPT-3 the model is split into many GPUs at every level, even a simple matrix multiplication goes accross the network, and there is no way to continue the computation until every GPU has communicated their result. So the issue here is that if a compute unit is very slow, then all the other ones* will block waiting for it, so your total cluster speed is defined by the slowest unit + slowest network time. And if a unit of compute disconnects (which happens constantly in @home scenarios), then its work need to be attributed to another unit and re-executed, while every other unit is idle waiting for that to happen. I'm sure there are smarter ways to split such a model than just trying to replicate what we currently do on big compute clusters, but it's definitly a hard problem. Finding better ways to do distributed training with less communication between units is an active area of research *: Not exactly all the other ones, because you could divide your cluster into chunks, each chunk handling one batch of data. So the chunks would be independents to a certain degree. But the size of the batch has an impact on the final performance of the model (because of batch normalization operation) so you still want it to be quite big, which requires many clients for a single chunk.
  - chosen1111 6y ago
    Very interesting thank you.

chosen1111 6y ago

Great points. Thank you. Is something like this capable of being run as a distributed computing project, akin to something like folding at home? Are there any obstacles that would make recreating a GPT-3 via a distributed system impossible...other than GPT-4 coming out before completion of such a system?

Voloskaya 6y ago
I'm not an expert in distributed computing so take what I say with a grain of salt, but I believe it would be really hard. The premise of all the @home projects is that each compute unit can do some relatively large amount of work in isolation and then communicate the result back to a central server. If one of the client disconnects in the middle of it's computation, it can just be redistributed to another client. And it doesn't really matter if a client is very slow. For the same reason network latency barely matters in those scenarios. On the other hand, to train GPT-3 the model is split into many GPUs at every level, even a simple matrix multiplication goes accross the network, and there is no way to continue the computation until every GPU has communicated their result. So the issue here is that if a compute unit is very slow, then all the other ones* will block waiting for it, so your total cluster speed is defined by the slowest unit + slowest network time. And if a unit of compute disconnects (which happens constantly in @home scenarios), then its work need to be attributed to another unit and re-executed, while every other unit is idle waiting for that to happen. I'm sure there are smarter ways to split such a model than just trying to replicate what we currently do on big compute clusters, but it's definitly a hard problem. Finding better ways to do distributed training with less communication between units is an active area of research *: Not exactly all the other ones, because you could divide your cluster into chunks, each chunk handling one batch of data. So the chunks would be independents to a certain degree. But the size of the batch has an impact on the final performance of the model (because of batch normalization operation) so you still want it to be quite big, which requires many clients for a single chunk.
- chosen1111 6y ago
  Very interesting thank you.