4 ms·

i don’t think they need to win the on device market. we need to separate inference and training - the real winners are those who have the training compute. you

i don’t think they need to win the on device market. we need to separate inference and training - the real winners are those who have the training compute. you can always have other companies help with inference

dagmx 2y ago
At what point does the on device stuff eat into their market share though? As on device gets better, who will pay for cloud compute? Other than enterprise use. I’m not saying on device will ever truly compete at quality, but I believe it’ll be good enough that most people don’t care to pay for cloud services.
- whimsicalism 2y ago
  You're still focused about inference :) inference basically does not matter, it is a commodity
  - dagmx 2y ago
    You’re still focused about training :) training doesn’t matter if inference costs are high and people don’t pay for them
    - whimsicalism 2y ago
      but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference
      - dagmx 2y ago
        Inference costs per invocation aren’t high. Scale it out to billions of users and it’s a different story. Training is amortized over each inference, so the cost of inference also needs to include the cost of training to break even unless made up elsewhere
  - rowanG077 2y ago
    That makes no sense. Inference cost dwarf training cost if you have a succesfull product pretty quickly. Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1.
    - whimsicalism 2y ago
      > Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1. Stack enough GPUs and any of them can run o1. Building a chip to infer LLMs is much easier than building a training chip. Just because one cost dwarfs another does not mean that this is where the most marginal value from developing a better chip will be, especially if other people are just doing it for you. Google gets a good model, inference providers will be begging to be able to run it on their platform, or to just sell google their chips - and as I said, inference chips are much easier.
      - menaerus 2y ago
        Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.
        whimsicalism 2y ago
        more like ~$16k for 16 3090s. AMD chips can also run these models. The parts are expensive but there is a competitive market in processors that can do LLM inference. Less so in training.
        menaerus 2y ago
        > more like ~$16k for 16 3090s I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000. > The parts are expensive Now that we invested ~$30k in GPUs, we only need to find a motherboard that can accommodate 16x pcie4 x16 GPUs, right? And we also need a CPU that can drive that many pcie4 x16 lanes? Well, none of them exist, not even in the server parts sector let alone client commodity hardware. In any case, you'd need two CPUs so even with this imaginary motherboard we are already entering the server rack design space. And that costs 100's of thousands of $$$. > but there is a competitive market in processors that can do LLM inference Nothing but the smallest and smallish models. If that existed then why would you set yourself out building a 16x RTX 3090 machine? Sorry, but you're just spitting out non-sense.
maeil 2y ago
> i don’t think they need to win the on device market. The second Apple comes out with strong on-device AI - and it very much looks like they will - Google will have to respond on Android. They can't just sit and pray that e.g. Samsung makes a competitive chip for this purpose.
- petra 2y ago
  But given inference time compute, to give a strong reply reasonably fast, you'll need a lot of compute, very rarely used. Economically this fits the cloud much better.
- reportingsjr 2y ago
  The Android on chip AI is and has been leagues better than what is available on iOS. If anything, I think the upcoming iOS AI update will bring them to a similar level as android/google.
- SimianSci 2y ago
  I think Apple is uniquely disadvantaged in the AI race to a point people dont realize. They have less training data to use, having famously been focused on privacy for its users and thus having no particular advantage in this space due to not having customer data to train on. They have little to no cloud business, and while they operate a couple of services for their users, they do not have the infrastructure scale to compete with hyperscaler cloud vendors such as Google and Microsoft. Most of what they would need to spend on training new models would require that they hand over lots of money to the very companies that already have their own models, supercharging their competition. While there is a chance that Apple might come out with a very sophisticate on-device model. The problem here is that they would only be able to compete with other on-device models. The magnitude of compute needed to keep pace with SOA models is not achievable on a single device. It will take many generations of Apple silicon in order to compete with the compute of existing datacenters. Google also already has competitive silicon in this space with the Tensor series processors, which are being fabbed at Samsung plants today. There is no sitting and praying necessary on their part as they already compete. Apple is a very distant competitor in the space of AI, and I see no reason to assume this will change, they are uniquely disadvantaged by several of the choices they made on their way to mobile supremacy. The only thing they currently have going for them is the development of their own ARM silicon which may give them the ability to compete with Google's TPU chips, but there is far more needed to be competitive here than the ability to avoid the Nvidia tax.
vineyardmike 2y ago
I don’t think the AI market will ever really be a healthy one until inference vastly outnumbers training. What does it say about AI if training is done more than inference? I agree that the in-device inference market is not important yet.
- whimsicalism 2y ago
  done more != where the value is at inference hardware is a commodity in a way that training is not

dagmx 2y ago

At what point does the on device stuff eat into their market share though? As on device gets better, who will pay for cloud compute? Other than enterprise use. I’m not saying on device will ever truly compete at quality, but I believe it’ll be good enough that most people don’t care to pay for cloud services.

whimsicalism 2y ago
You're still focused about inference :) inference basically does not matter, it is a commodity
- dagmx 2y ago
  You’re still focused about training :) training doesn’t matter if inference costs are high and people don’t pay for them
  - whimsicalism 2y ago
    but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference
    - dagmx 2y ago
      Inference costs per invocation aren’t high. Scale it out to billions of users and it’s a different story. Training is amortized over each inference, so the cost of inference also needs to include the cost of training to break even unless made up elsewhere
- rowanG077 2y ago
  That makes no sense. Inference cost dwarf training cost if you have a succesfull product pretty quickly. Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1.
  - whimsicalism 2y ago
    > Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1. Stack enough GPUs and any of them can run o1. Building a chip to infer LLMs is much easier than building a training chip. Just because one cost dwarfs another does not mean that this is where the most marginal value from developing a better chip will be, especially if other people are just doing it for you. Google gets a good model, inference providers will be begging to be able to run it on their platform, or to just sell google their chips - and as I said, inference chips are much easier.
    - menaerus 2y ago
      Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.
      - whimsicalism 2y ago
        more like ~$16k for 16 3090s. AMD chips can also run these models. The parts are expensive but there is a competitive market in processors that can do LLM inference. Less so in training.
        menaerus 2y ago
        > more like ~$16k for 16 3090s I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000. > The parts are expensive Now that we invested ~$30k in GPUs, we only need to find a motherboard that can accommodate 16x pcie4 x16 GPUs, right? And we also need a CPU that can drive that many pcie4 x16 lanes? Well, none of them exist, not even in the server parts sector let alone client commodity hardware. In any case, you'd need two CPUs so even with this imaginary motherboard we are already entering the server rack design space. And that costs 100's of thousands of $$$. > but there is a competitive market in processors that can do LLM inference Nothing but the smallest and smallish models. If that existed then why would you set yourself out building a 16x RTX 3090 machine? Sorry, but you're just spitting out non-sense.

maeil 2y ago

> i don’t think they need to win the on device market. The second Apple comes out with strong on-device AI - and it very much looks like they will - Google will have to respond on Android. They can't just sit and pray that e.g. Samsung makes a competitive chip for this purpose.

petra 2y ago
But given inference time compute, to give a strong reply reasonably fast, you'll need a lot of compute, very rarely used. Economically this fits the cloud much better.
reportingsjr 2y ago
The Android on chip AI is and has been leagues better than what is available on iOS. If anything, I think the upcoming iOS AI update will bring them to a similar level as android/google.
SimianSci 2y ago
I think Apple is uniquely disadvantaged in the AI race to a point people dont realize. They have less training data to use, having famously been focused on privacy for its users and thus having no particular advantage in this space due to not having customer data to train on. They have little to no cloud business, and while they operate a couple of services for their users, they do not have the infrastructure scale to compete with hyperscaler cloud vendors such as Google and Microsoft. Most of what they would need to spend on training new models would require that they hand over lots of money to the very companies that already have their own models, supercharging their competition. While there is a chance that Apple might come out with a very sophisticate on-device model. The problem here is that they would only be able to compete with other on-device models. The magnitude of compute needed to keep pace with SOA models is not achievable on a single device. It will take many generations of Apple silicon in order to compete with the compute of existing datacenters. Google also already has competitive silicon in this space with the Tensor series processors, which are being fabbed at Samsung plants today. There is no sitting and praying necessary on their part as they already compete. Apple is a very distant competitor in the space of AI, and I see no reason to assume this will change, they are uniquely disadvantaged by several of the choices they made on their way to mobile supremacy. The only thing they currently have going for them is the development of their own ARM silicon which may give them the ability to compete with Google's TPU chips, but there is far more needed to be competitive here than the ability to avoid the Nvidia tax.

vineyardmike 2y ago

I don’t think the AI market will ever really be a healthy one until inference vastly outnumbers training. What does it say about AI if training is done more than inference? I agree that the in-device inference market is not important yet.

whimsicalism 2y ago
done more != where the value is at inference hardware is a commodity in a way that training is not