3 ms·

What about some kind of sharding, parts of the computation that could be executed in isolation for a longer period of time?

• Filligree 6y ago
  An ongoing research problem. OpenAI would certainly like being able to use smaller GPUs, instead of having to fit the entire model into one.
  • jne2356 6y ago
    GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.