3 ms·

There are other ways around the cost. They could build a distributed training system along the lines of Folding@Home, I’m sure a lot of AI folks would donate GP

• boulos 5y ago
  The optimization problem at the inner loop of these methods doesn't handle latency very well. This is covered a bit in the DOTA 2 rerun work [1] (e.g., Figure 5b on staleness). So for both large models (which are split up amongst multiple accelerators but effectively one "computer"), as well as learning from "your friends" you want to be more tightly coupled than something like seti/folding@home. There are plenty of workloads that are great for embarrassingly parallel (and now "pleasingly parallel") workloads, but training a single model isn't one of them. [1] https://cdn.openai.com/dota-2.pdf https://cdn.openai.com/dota-2.pdf