4 ms·

I think this is the big point of uncertainty in Nvidia’s future: will we find new training techniques which require significantly less compute, and/or are bette

I think this is the big point of uncertainty in Nvidia’s future: will we find new training techniques which require significantly less compute, and/or are better suited to some fundamentally different architecture than GPUs? I’m reluctant to bet no on that long term, and “long term” for ML right now is not very long.

brigadier132 2y ago
If we find a new training technique that is that much more efficient why do you think we wont just increase the amount of training we do be n times? (or even more since it's now accessible to train custom models for smaller businesses)
- mistercow 2y ago
  We might, but it’s also plausible that it would change the ecosystem so much that centralized models are no longer so prominent. For example, suppose that with much cheaper training, most training is on your specific data and behaviors so that you have a model (or ensemble of models) tailored to your own needs. You still need a foundation model, but those are much smaller so that they can run on device, so even with overparameterization and distillation, the training costs are orders of magnitude smaller. Or, in the small business case (mind you, “long term” for tech reaching small businesses is looooong), these businesses again need much smaller models because a) they don’t need a model well versed in Shakespeare and multi variable calculus, and b) they want inference to be as low cost as possible. These are just scenarios off the top of my head. The broader point is that a dramatic drop in training cost is a wildcard whose effects are really hard to predict.
marcosdumay 2y ago
I'd bet that any AI that is really useful for the tasks people want to push LLMs into will answer "yes" to both parts of your question. But I don't know what "long term" is exactly, and have no idea how to time this thing. Besides, I'd bet the sibling evoking the Jevon's paradox is correct.
acchow 2y ago
I’m betting the opposite: new model architectures will unlock greater abilities at the cost of massive compute.
- mistercow 2y ago
  Even so, massive compute doesn’t necessarily mean GPU-friendly compute. We could see a breakthrough in analog or neuromorphic hardware, for example, where Nvidia isn’t well positioned. Or we could see a training breakthrough which is far more efficient, but bottlenecked on single core performance, or just branch-heavy performance. You can imagine scenarios like that where GPUs still play a role, but where even today’s top of the line GPUs are way over the top compared to the CPU bottleneck. If one of those scenarios happens, maybe Nvidia can pivot, or if we see analog take over, we could see something really bizarre like a dark horse like Seagate taking over by pivoting from SSDs, just because their manufacturing pipeline is more compatible.

brigadier132 2y ago

If we find a new training technique that is that much more efficient why do you think we wont just increase the amount of training we do be n times? (or even more since it's now accessible to train custom models for smaller businesses)

mistercow 2y ago
We might, but it’s also plausible that it would change the ecosystem so much that centralized models are no longer so prominent. For example, suppose that with much cheaper training, most training is on your specific data and behaviors so that you have a model (or ensemble of models) tailored to your own needs. You still need a foundation model, but those are much smaller so that they can run on device, so even with overparameterization and distillation, the training costs are orders of magnitude smaller. Or, in the small business case (mind you, “long term” for tech reaching small businesses is looooong), these businesses again need much smaller models because a) they don’t need a model well versed in Shakespeare and multi variable calculus, and b) they want inference to be as low cost as possible. These are just scenarios off the top of my head. The broader point is that a dramatic drop in training cost is a wildcard whose effects are really hard to predict.

marcosdumay 2y ago

I'd bet that any AI that is really useful for the tasks people want to push LLMs into will answer "yes" to both parts of your question. But I don't know what "long term" is exactly, and have no idea how to time this thing. Besides, I'd bet the sibling evoking the Jevon's paradox is correct.

acchow 2y ago

I’m betting the opposite: new model architectures will unlock greater abilities at the cost of massive compute.

mistercow 2y ago
Even so, massive compute doesn’t necessarily mean GPU-friendly compute. We could see a breakthrough in analog or neuromorphic hardware, for example, where Nvidia isn’t well positioned. Or we could see a training breakthrough which is far more efficient, but bottlenecked on single core performance, or just branch-heavy performance. You can imagine scenarios like that where GPUs still play a role, but where even today’s top of the line GPUs are way over the top compared to the CPU bottleneck. If one of those scenarios happens, maybe Nvidia can pivot, or if we see analog take over, we could see something really bizarre like a dark horse like Seagate taking over by pivoting from SSDs, just because their manufacturing pipeline is more compatible.