4 ms·

But you also run inference orders of magnitudes more times, so it should still amount to more compute than training?

by demaga 2y ago

teaearlgraycold 2y ago
That matters more to the electricity company than the silicon company. The profit margins on the datacenter training hardware are stupidly high compared to an AMD APU.
- binary132 2y ago
  If there are tens of thousands of training GPUs but billions of APUs, then what? BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there.
  - k__ 2y ago
    This. Most will probably use something like Llama as base.
  - talldayo 2y ago
    > If there are tens of thousands of training GPUs but billions of APUs, then what? Believe it or not, we've actually been grappling with this scenario for almost a decade at this point. Originally the answer was to unite hardware manufacturers around a common featureset that could compete with (albeit not replace) CUDA. Khronos was prepared to elevate OpenCL to an industry standard, but Apple pulled their support for it and let the industry collapse into proprietary competition again. I bet they're kicking themselves over that one, if they still hold a stronger grudge against Nvidia than Khronos at least. So - logically, there's actually a one-size-fits-all solution for this problem. It was even going to get managed by the same people handling Vulkan. The problem was corporate greed and shortsighted investment that let OpenCL languish while CUDA was under active heavy development. > BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there. Eh, that's kinda like saying "app development is so expensive that consumers will eventually care". Consumers just buy the end product; they are never exposed to building the software or concerned with the cost of the development. This is especially true with businesses like OpenAI that just give you free access to a decent LLM (or Apple and their "it's free for now" mentality).
marcosdumay 2y ago
Besides, if you separate them, the people doing the training will put way more effort into optimizing their hardware ROI than the ones doing inference.

But you also run inference orders of magnitudes more times, so it should still amount to more compute than training?

by demaga 2y ago

But you also run inference orders of magnitudes more times, so it should still amount to more compute than training?

That matters more to the electricity company than the silicon company. The profit margins on the datacenter training hardware are stupidly high compared to an AMD APU.

binary132 2y ago
If there are tens of thousands of training GPUs but billions of APUs, then what? BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there.
- k__ 2y ago
  This. Most will probably use something like Llama as base.
- talldayo 2y ago
  > If there are tens of thousands of training GPUs but billions of APUs, then what? Believe it or not, we've actually been grappling with this scenario for almost a decade at this point. Originally the answer was to unite hardware manufacturers around a common featureset that could compete with (albeit not replace) CUDA. Khronos was prepared to elevate OpenCL to an industry standard, but Apple pulled their support for it and let the industry collapse into proprietary competition again. I bet they're kicking themselves over that one, if they still hold a stronger grudge against Nvidia than Khronos at least. So - logically, there's actually a one-size-fits-all solution for this problem. It was even going to get managed by the same people handling Vulkan. The problem was corporate greed and shortsighted investment that let OpenCL languish while CUDA was under active heavy development. > BTW, training is such a high cost that it seems like a major motive for the customer to reduce costs there. Eh, that's kinda like saying "app development is so expensive that consumers will eventually care". Consumers just buy the end product; they are never exposed to building the software or concerned with the cost of the development. This is especially true with businesses like OpenAI that just give you free access to a decent LLM (or Apple and their "it's free for now" mentality).

marcosdumay 2y ago

Besides, if you separate them, the people doing the training will put way more effort into optimizing their hardware ROI than the ones doing inference.