3 ms·
Yeah I also think that the impact of better hardware generates a visible lead in quality in the models that are released, for example: companies like OpenAI hav
by GaggiX 3y ago
Yeah I also think that the impact of better hardware generates a visible lead in quality in the models that are released, for example: companies like OpenAI have had access to large quantities of H100 for a few months now and Sora is being presented, something I would not have believed a year ago, I would also believe that Claude 3 models were trained on H100, DBRX was trained on 12T tokens, a big difference compared to the 300B for the original GPT-3, the new NovelAI image generation was trained on H100 and compared to the previous model is like day and night. It seems to create a generational jump.
- declaredapple 3y ago> companies like OpenAI have had access to large quantities of H100 for a few months now and Sora is being presented From what I could tell from Nvidia's recent presentation, Nvidia works directly with OpenAI to test their next gen hardware. IIRC they had some slides showing the throughput comparisons with Hopper and Blackwell, suggesting they used OpenAI's workload for testing. H100's have been generally available (not a long waitlist) for only several months, but all the big players had them already 1 year ago. I agree with you, but I think you might be 1 generation behind. > OpenAI used H100’s predecessor — NVIDIA A100 GPUs — to train and run ChatGPT, an AI system optimized for dialogue, which has been used by hundreds of millions of people worldwide in record time. OpenAI will be using H100 on its Azure supercomputer to power its continuing AI research. March 21, 2023 https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand-reach-as-demand-for-ai-grows https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand...
- GaggiX 3y agoVery interesting, I guess it does make sense that GPT-4 was also trained on the Hopper architecture.