4 ms·

Can this run GPT3? Also a 6.5kW computer must need quite some cooling system...

• fennecfoxen 6y ago
  Let's see. According to https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/ https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo... GPT-3 was run on V100s, which are the previous generation server GPUs ("Volta" vs "Ampere"). "GPT-3 uses half-precision floating-point variables at 16 bits per parameter... At around $130,000, DGX-1 is short on VRAM (8×16 GB), but has all the other components for a solid performance on GPT-3." (DGX-1 is the previous chassis NVIDIA offered for hosting V100s.) So I tentatively think the answer is yes, a single DGX-A100 probably can run GPT-3 comfortably. You'll still want a cluster of them to train it, though, because they note training takes about 355 V100-years.
  • p1esk 6y ago
    You don’t need any GPUs to run gpt-3. You can run it on a regular server with >512GB RAM.
    • fennecfoxen 6y ago
      I can only comment that this is at odds with the information at the link I supplied, and I cannot tell you for certain whether that information is actually accurate.
      • qayxc 6y ago
        The link only mentions requirements to run inference at "decent speeds", without going into details about what they consider to be decent speeds. In principle you can of course run any model on any hardware that has enough RAM. Whether the inference performance is acceptable depends your particular application. I'd argue that for most non-interactive use cases, inference speed doesn't really matter and the cost benefit from running on CPUs vs GPUs might be worth it.