3 ms·

How long will it take to train GPT-4 on Fugaku supercomputer

• ThrowawayR2 4y ago
  Not sure I get the point of this? It doesn't seem particularly interesting to say that a compute cluster optimized for FP64 jobs is less effective at handling FP16 jobs.
  • ftxbro 4y ago
    I think the point is that Sam Altman the CEO of OpenAI has been in Japan visiting the prime minister and looking for collaboration opportunities. The Fugaku supercomputer is the flagship public supercomputer in Japan, so it's interesting to see how long it would take it to train GPT-4. According to that article it looks like it's on the order of 10x slower (as opposed to equally fast or like 100x slower). To me the most interesting thing is that they seem to be revealing (or assuming?) that GPT-4 costs 10x the computer power of GPT-3.5. I'm not sure that was public information before. Maybe it wasn't.
    • MichaelZuo 4y ago
      It still doesn't seem clear what the comparison is for. Unless someone in Japan didn't believe that OpenAI/Micrsoft had access to 100x the FP16 compute that Fugaku can perform?