5 ms·

Time to get serious about competitive open source models. Can't we do a seti at home sort of thing to distribute the training?

renonce 4y ago
Training is a bandwidth-intensive operation and requires huge (20Gbps+ stable and uninterrupted) bandwidth between all peers.
avereveard 4y ago
"GPT-3 175B model required 3.14E23 flops" according to their marketing material. Seti at home was about 1PetaFlops iirc so about 3 years training, possibly less if you can generate enough attention to the project that the people with the beefy devices will partecipate. The problem is that you need to train the full model you can't train aspect of it and even with each node doing independent tiny batches the network bandwidth for sinchronization would be massive. Seti was massively parallelizable because of the nature of the job oddnt require sinchronization between every peer.
- dylan604 4y ago
  is there something to be said that seti@home was CPU only? would the GPU give a performance benefit that seti did not have? are people still using the GPUs to mine coins, or is that GPU compute at home available now?
  - lyu07282 4y ago
    The problem is not compute power the problem is weight and data synchronization. Each iteration or epoch builds on the previous, you either need to run the full model on each node with part of the data and you synchronize every epoch or you run part of the model but then you need to synchronize weights after each iteration. In proof of work mining you don't need to synchronize between each iteration, that's why in mining rigs the GPUs are connected to the CPU with only a few pcie lanes (2x instead of 16x or something) and they are unsuitable for machine learning. The clusters that big orgs are using to parallelize training use extremely expensive infiniband interconnections with tens of gigabytes per second and low latency (400 Gbps in the latest oai cluster) for that reason. Unfortunately not something you can democratize anytime soon.
    - shagie 4y ago
      While not likely used here, a fun chip for doing ML - https://www.cerebras.net/product-chip/ https://www.cerebras.net/product-chip/ From Tom's Hardware: https://www.tomshardware.com/news/cerebras-wafer-scale-engine-2-worlds-largest-chip-7nm-850000-cores https://www.tomshardware.com/news/cerebras-wafer-scale-engin... > Power Consumption (System/Chip) 20kW / 15kW Putting 15kW into one chip is really impressive. The power and cooling for that gets rather interesting. The piece that reminded me of this is the comment on the bandwidth: > Fabric Bandwidth 220 Pb/s
- ChatGTP 4y ago
  How does something of this scale impact climate change? Like when there are 5-6-7 OpenAIs, what does that look like, is this just a huge amount of energy consumption ?
  - pixl97 4y ago
    Depends how they source the power. If they are getting it out of nukes or renewables, well, not that much at all.
  - avereveard 4y ago
    The estimate I've found say GPT-3 released an estimated 552 tons of CO2, and so about two large lorries being driven for a year, wouldn't lose my sleep over it
    - ChatGTP 4y ago
      Yup, let's just keep saying that I guess?

renonce 4y ago

Training is a bandwidth-intensive operation and requires huge (20Gbps+ stable and uninterrupted) bandwidth between all peers.

avereveard 4y ago

"GPT-3 175B model required 3.14E23 flops" according to their marketing material. Seti at home was about 1PetaFlops iirc so about 3 years training, possibly less if you can generate enough attention to the project that the people with the beefy devices will partecipate. The problem is that you need to train the full model you can't train aspect of it and even with each node doing independent tiny batches the network bandwidth for sinchronization would be massive. Seti was massively parallelizable because of the nature of the job oddnt require sinchronization between every peer.

dylan604 4y ago
is there something to be said that seti@home was CPU only? would the GPU give a performance benefit that seti did not have? are people still using the GPUs to mine coins, or is that GPU compute at home available now?
- lyu07282 4y ago
  The problem is not compute power the problem is weight and data synchronization. Each iteration or epoch builds on the previous, you either need to run the full model on each node with part of the data and you synchronize every epoch or you run part of the model but then you need to synchronize weights after each iteration. In proof of work mining you don't need to synchronize between each iteration, that's why in mining rigs the GPUs are connected to the CPU with only a few pcie lanes (2x instead of 16x or something) and they are unsuitable for machine learning. The clusters that big orgs are using to parallelize training use extremely expensive infiniband interconnections with tens of gigabytes per second and low latency (400 Gbps in the latest oai cluster) for that reason. Unfortunately not something you can democratize anytime soon.
  - shagie 4y ago
    While not likely used here, a fun chip for doing ML - https://www.cerebras.net/product-chip/ https://www.cerebras.net/product-chip/ From Tom's Hardware: https://www.tomshardware.com/news/cerebras-wafer-scale-engine-2-worlds-largest-chip-7nm-850000-cores https://www.tomshardware.com/news/cerebras-wafer-scale-engin... > Power Consumption (System/Chip) 20kW / 15kW Putting 15kW into one chip is really impressive. The power and cooling for that gets rather interesting. The piece that reminded me of this is the comment on the bandwidth: > Fabric Bandwidth 220 Pb/s
ChatGTP 4y ago
How does something of this scale impact climate change? Like when there are 5-6-7 OpenAIs, what does that look like, is this just a huge amount of energy consumption ?
- pixl97 4y ago
  Depends how they source the power. If they are getting it out of nukes or renewables, well, not that much at all.
- avereveard 4y ago
  The estimate I've found say GPT-3 released an estimated 552 tons of CO2, and so about two large lorries being driven for a year, wouldn't lose my sleep over it
  - ChatGTP 4y ago
    Yup, let's just keep saying that I guess?