3 ms·

"Inference costs will dominate" seems very short-sighted. I've kind of laughed at any startup saying "oh we will use AI to do <X>" for years but seeing what LLM

"Inference costs will dominate" seems very short-sighted. I've kind of laughed at any startup saying "oh we will use AI to do <X>" for years but seeing what LLMs can do suddenly hardware seems like the limiting factor. I can think of endless applications if I had a few GPUs with a petabyte of onboard ram each that also run about 10000x the FLOPs of current GPUs and a few petabytes of storage for training data. I would be training models left and right to tackle whatever problem I thought of. Of course, it's hard to say if such hardware will be available in my lifetime at a price point that I can get for personal use. In the meantime providing hardware for training will still underpin massive businesses. And especially as hardware costs come down I suspect "fine-tuning" will become more and more common and there will even be use cases where running inference on a large number of tokens looks a lot more like fine-tuning, which is to say you're going to want the best GPU you can find and CPUs are just not going to work very well if at all.

015a 3y ago
One thing I think about: over time, more and more training will probably move closer to users devices. - Client-side training carries a lot of financial advantages; you can push the cost of the silicon, storage, and electricity onto the user. - There's privacy benefits, which while not a major driver in adoption is something people think about. - Apple does this already. They're going to keep doing this. When Apple makes a decision, it instantly impacts a massive plurality of the human population in a way that no other company can; and, it tangentially influences other companies. I think you're right that "inference costs will dominate" is a short-sighted take. But: I think the better point is to think about where training will happen. Nvidia is weirdly poorly positioned to have a strong hand in client-side training. They don't have a cost-efficient and electricity-efficient strategy in any of their products; except for Tegra, which has seen zero consumer uptake outside of the Nintendo Switch. There's no hundred-billion-dollar client side AI training strategy anywhere approximate to the RTX 3070 in my gaming Windows PC, that ain't happening. I'm doubtful they can make that pivot; there's a lot of entrenched interest, and legitimately great products, from the existing computer & smartphone manufacturers. Apple has their chips. Google has their chips, and a really strong relationship with Samsung. Microsoft will be an ally, but they have very little power toward convincing their Windows users that a $1400 laptop is better than a $800 one because it has local AI training capability. But, I mean: server-side training is still going to be huge, and Nvidia will still be an extremely successful company. Its just when you consider their percent ownership of the net total of all AI training that will happen in 2030; its going to drop, and the biggest factor behind that drop isn't going to be AMD; its going to be client-side training on chips made by Apple, Google, and others.

015a 3y ago

One thing I think about: over time, more and more training will probably move closer to users devices. - Client-side training carries a lot of financial advantages; you can push the cost of the silicon, storage, and electricity onto the user. - There's privacy benefits, which while not a major driver in adoption is something people think about. - Apple does this already. They're going to keep doing this. When Apple makes a decision, it instantly impacts a massive plurality of the human population in a way that no other company can; and, it tangentially influences other companies. I think you're right that "inference costs will dominate" is a short-sighted take. But: I think the better point is to think about where training will happen. Nvidia is weirdly poorly positioned to have a strong hand in client-side training. They don't have a cost-efficient and electricity-efficient strategy in any of their products; except for Tegra, which has seen zero consumer uptake outside of the Nintendo Switch. There's no hundred-billion-dollar client side AI training strategy anywhere approximate to the RTX 3070 in my gaming Windows PC, that ain't happening. I'm doubtful they can make that pivot; there's a lot of entrenched interest, and legitimately great products, from the existing computer & smartphone manufacturers. Apple has their chips. Google has their chips, and a really strong relationship with Samsung. Microsoft will be an ally, but they have very little power toward convincing their Windows users that a $1400 laptop is better than a $800 one because it has local AI training capability. But, I mean: server-side training is still going to be huge, and Nvidia will still be an extremely successful company. Its just when you consider their percent ownership of the net total of all AI training that will happen in 2030; its going to drop, and the biggest factor behind that drop isn't going to be AMD; its going to be client-side training on chips made by Apple, Google, and others.