4 ms·

Regarding TPU’s, sure for the stuff that’s running on the cloud. However their on device TPUs lag behind the competition and Google still seem to struggle to m

by dagmx 2y ago

Regarding TPU’s, sure for the stuff that’s running on the cloud. However their on device TPUs lag behind the competition and Google still seem to struggle to move significant parts of Gemini to run on device as a result. Of course, Gemini is provided as a subscription service as well so perhaps they’re not incentivized to move things locally. I am curious if they’ll introduce something like Apple’s private cloud compute.

whimsicalism 2y ago
i don’t think they need to win the on device market. we need to separate inference and training - the real winners are those who have the training compute. you can always have other companies help with inference
- dagmx 2y ago
  At what point does the on device stuff eat into their market share though? As on device gets better, who will pay for cloud compute? Other than enterprise use. I’m not saying on device will ever truly compete at quality, but I believe it’ll be good enough that most people don’t care to pay for cloud services.
  - whimsicalism 2y ago
    You're still focused about inference :) inference basically does not matter, it is a commodity
    - dagmx 2y ago
      You’re still focused about training :) training doesn’t matter if inference costs are high and people don’t pay for them
      - whimsicalism 2y ago
        but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference
        dagmx 2y ago
        Inference costs per invocation aren’t high. Scale it out to billions of users and it’s a different story. Training is amortized over each inference, so the cost of inference also needs to include the cost of training to break even unless made up elsewhere
    - rowanG077 2y ago
      That makes no sense. Inference cost dwarf training cost if you have a succesfull product pretty quickly. Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1.
      - whimsicalism 2y ago
        > Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1. Stack enough GPUs and any of them can run o1. Building a chip to infer LLMs is much easier than building a training chip. Just because one cost dwarfs another does not mean that this is where the most marginal value from developing a better chip will be, especially if other people are just doing it for you. Google gets a good model, inference providers will be begging to be able to run it on their platform, or to just sell google their chips - and as I said, inference chips are much easier.
        menaerus 2y ago
        Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.
        whimsicalism 2y ago
        more like ~$16k for 16 3090s. AMD chips can also run these models. The parts are expensive but there is a competitive market in processors that can do LLM inference. Less so in training.
        menaerus 2y ago
        > more like ~$16k for 16 3090s I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000. > The parts are expensive Now that we invested ~$30k in GPUs, we only need to find a motherboard that can accommodate 16x pcie4 x16 GPUs, right? And we also need a CPU that can drive that many pcie4 x16 lanes? Well, none of them exist, not even in the server parts sector let alone client commodity hardware. In any case, you'd need two CPUs so even with this imaginary motherboard we are already entering the server rack design space. And that costs 100's of thousands of $$$. > but there is a competitive market in processors that can do LLM inference Nothing but the smallest and smallish models. If that existed then why would you set yourself out building a 16x RTX 3090 machine? Sorry, but you're just spitting out non-sense.
- maeil 2y ago
  > i don’t think they need to win the on device market. The second Apple comes out with strong on-device AI - and it very much looks like they will - Google will have to respond on Android. They can't just sit and pray that e.g. Samsung makes a competitive chip for this purpose.
  - petra 2y ago
    But given inference time compute, to give a strong reply reasonably fast, you'll need a lot of compute, very rarely used. Economically this fits the cloud much better.
  - reportingsjr 2y ago
    The Android on chip AI is and has been leagues better than what is available on iOS. If anything, I think the upcoming iOS AI update will bring them to a similar level as android/google.
  - SimianSci 2y ago
    I think Apple is uniquely disadvantaged in the AI race to a point people dont realize. They have less training data to use, having famously been focused on privacy for its users and thus having no particular advantage in this space due to not having customer data to train on. They have little to no cloud business, and while they operate a couple of services for their users, they do not have the infrastructure scale to compete with hyperscaler cloud vendors such as Google and Microsoft. Most of what they would need to spend on training new models would require that they hand over lots of money to the very companies that already have their own models, supercharging their competition. While there is a chance that Apple might come out with a very sophisticate on-device model. The problem here is that they would only be able to compete with other on-device models. The magnitude of compute needed to keep pace with SOA models is not achievable on a single device. It will take many generations of Apple silicon in order to compete with the compute of existing datacenters. Google also already has competitive silicon in this space with the Tensor series processors, which are being fabbed at Samsung plants today. There is no sitting and praying necessary on their part as they already compete. Apple is a very distant competitor in the space of AI, and I see no reason to assume this will change, they are uniquely disadvantaged by several of the choices they made on their way to mobile supremacy. The only thing they currently have going for them is the development of their own ARM silicon which may give them the ability to compete with Google's TPU chips, but there is far more needed to be competitive here than the ability to avoid the Nvidia tax.
- vineyardmike 2y ago
  I don’t think the AI market will ever really be a healthy one until inference vastly outnumbers training. What does it say about AI if training is done more than inference? I agree that the in-device inference market is not important yet.
  - whimsicalism 2y ago
    done more != where the value is at inference hardware is a commodity in a way that training is not
mupuff1234 2y ago
Majority of people want better performance, running locally is just a nice to have feature.
- dagmx 2y ago
  They’ll care though when they have to pay for it, or when they’re in an area with poor reception.
  - mupuff1234 2y ago
    They pay to run it locally as well (more expensive hardware) And sure, poor reception will be an issue, but most people would still absolutely take a helpful remote assistant over a dumb local assistant. And you don't exactly see people complaining that they can't run Google/YouTube/etc locally.
    - dagmx 2y ago
      Your first sentence has the fallacy that you’re attributing the cost of the device to a single feature against the cost of that single feature. Most people are unlikely to buy the device for the AI features alone. It’s a value add to the device they’d buy anyway. So you need the paid for option to be significantly better than the free one that comes with the device. Your second sentence assumes the local one is dumb. What happens when local ones get better? Again how much better is the cloud one to compete on cost? To your last sentence, it assumes data fetching from the cloud. Which is valid but a lot of data is local too. Are people really going to pay for what Google search is giving them for free?
      - mupuff1234 2y ago
        I think it's a more likely assumption that on device performance will trail off device models by a significant margin for at least the next few years - of course if magically you can make it work locally with the same level of performance it would be better. Plus a lot of the "agentic" stuff is interaction with the outside world, connectivity is a must regardless.
YetAnotherNick 2y ago
If the model weights is not open, you can't run it on device anyways.
- kridsdale1 2y ago
  The Pixel 9 runs many small proprietary Gemini models on the internal TPU.
  - griomnib 2y ago
    And yet these new models still haven’t reached feature parity with Google Assistant, which can turn my flashlight on, but with all the power of burning down a rainforest, Gemini still cannot interact with my actual phone.
    - lern_too_spel 2y ago
      I just tried asking my phone to turn on the flashlight using Gemini. It worked. https://9to5google.com/2024/11/07/gemini-utilities-extension/ https://9to5google.com/2024/11/07/gemini-utilities-extension...
      - griomnib 2y ago
        Ok I tried literally last week on Pixel 7a and it didn’t work. What model do you have? Maybe it requires a phone that can do on-device models?
        staticman2 2y ago
        I just tried it on my Galaxy Ultra s23 and it worked. I then disconnected internet and it did not work.
        _puk 2y ago
        Works on a Pixel 4A 5G.. Pretty sure that's not doing any fancy on-device models! That said, there was a popup today saying that assistant is now using Gemini, so I just enabled it to try. Could well have changed in the last week.

by dagmx 2y ago

whimsicalism 2y ago

i don’t think they need to win the on device market. we need to separate inference and training - the real winners are those who have the training compute. you can always have other companies help with inference

dagmx 2y ago
At what point does the on device stuff eat into their market share though? As on device gets better, who will pay for cloud compute? Other than enterprise use. I’m not saying on device will ever truly compete at quality, but I believe it’ll be good enough that most people don’t care to pay for cloud services.
- whimsicalism 2y ago
  You're still focused about inference :) inference basically does not matter, it is a commodity
  - dagmx 2y ago
    You’re still focused about training :) training doesn’t matter if inference costs are high and people don’t pay for them
    - whimsicalism 2y ago
      but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference
      - dagmx 2y ago
        Inference costs per invocation aren’t high. Scale it out to billions of users and it’s a different story. Training is amortized over each inference, so the cost of inference also needs to include the cost of training to break even unless made up elsewhere
  - rowanG077 2y ago
    That makes no sense. Inference cost dwarf training cost if you have a succesfull product pretty quickly. Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1.
    - whimsicalism 2y ago
      > Afaik there is no commodity hardware that can run state of the art models like chatgpt-o1. Stack enough GPUs and any of them can run o1. Building a chip to infer LLMs is much easier than building a training chip. Just because one cost dwarfs another does not mean that this is where the most marginal value from developing a better chip will be, especially if other people are just doing it for you. Google gets a good model, inference providers will be begging to be able to run it on their platform, or to just sell google their chips - and as I said, inference chips are much easier.
      - menaerus 2y ago
        Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.
        whimsicalism 2y ago
        more like ~$16k for 16 3090s. AMD chips can also run these models. The parts are expensive but there is a competitive market in processors that can do LLM inference. Less so in training.
        menaerus 2y ago
        > more like ~$16k for 16 3090s I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000. > The parts are expensive Now that we invested ~$30k in GPUs, we only need to find a motherboard that can accommodate 16x pcie4 x16 GPUs, right? And we also need a CPU that can drive that many pcie4 x16 lanes? Well, none of them exist, not even in the server parts sector let alone client commodity hardware. In any case, you'd need two CPUs so even with this imaginary motherboard we are already entering the server rack design space. And that costs 100's of thousands of $$$. > but there is a competitive market in processors that can do LLM inference Nothing but the smallest and smallish models. If that existed then why would you set yourself out building a 16x RTX 3090 machine? Sorry, but you're just spitting out non-sense.
maeil 2y ago
> i don’t think they need to win the on device market. The second Apple comes out with strong on-device AI - and it very much looks like they will - Google will have to respond on Android. They can't just sit and pray that e.g. Samsung makes a competitive chip for this purpose.
- petra 2y ago
  But given inference time compute, to give a strong reply reasonably fast, you'll need a lot of compute, very rarely used. Economically this fits the cloud much better.
- reportingsjr 2y ago
  The Android on chip AI is and has been leagues better than what is available on iOS. If anything, I think the upcoming iOS AI update will bring them to a similar level as android/google.
- SimianSci 2y ago
  I think Apple is uniquely disadvantaged in the AI race to a point people dont realize. They have less training data to use, having famously been focused on privacy for its users and thus having no particular advantage in this space due to not having customer data to train on. They have little to no cloud business, and while they operate a couple of services for their users, they do not have the infrastructure scale to compete with hyperscaler cloud vendors such as Google and Microsoft. Most of what they would need to spend on training new models would require that they hand over lots of money to the very companies that already have their own models, supercharging their competition. While there is a chance that Apple might come out with a very sophisticate on-device model. The problem here is that they would only be able to compete with other on-device models. The magnitude of compute needed to keep pace with SOA models is not achievable on a single device. It will take many generations of Apple silicon in order to compete with the compute of existing datacenters. Google also already has competitive silicon in this space with the Tensor series processors, which are being fabbed at Samsung plants today. There is no sitting and praying necessary on their part as they already compete. Apple is a very distant competitor in the space of AI, and I see no reason to assume this will change, they are uniquely disadvantaged by several of the choices they made on their way to mobile supremacy. The only thing they currently have going for them is the development of their own ARM silicon which may give them the ability to compete with Google's TPU chips, but there is far more needed to be competitive here than the ability to avoid the Nvidia tax.
vineyardmike 2y ago
I don’t think the AI market will ever really be a healthy one until inference vastly outnumbers training. What does it say about AI if training is done more than inference? I agree that the in-device inference market is not important yet.
- whimsicalism 2y ago
  done more != where the value is at inference hardware is a commodity in a way that training is not

mupuff1234 2y ago

Majority of people want better performance, running locally is just a nice to have feature.

dagmx 2y ago
They’ll care though when they have to pay for it, or when they’re in an area with poor reception.
- mupuff1234 2y ago
  They pay to run it locally as well (more expensive hardware) And sure, poor reception will be an issue, but most people would still absolutely take a helpful remote assistant over a dumb local assistant. And you don't exactly see people complaining that they can't run Google/YouTube/etc locally.
  - dagmx 2y ago
    Your first sentence has the fallacy that you’re attributing the cost of the device to a single feature against the cost of that single feature. Most people are unlikely to buy the device for the AI features alone. It’s a value add to the device they’d buy anyway. So you need the paid for option to be significantly better than the free one that comes with the device. Your second sentence assumes the local one is dumb. What happens when local ones get better? Again how much better is the cloud one to compete on cost? To your last sentence, it assumes data fetching from the cloud. Which is valid but a lot of data is local too. Are people really going to pay for what Google search is giving them for free?
    - mupuff1234 2y ago
      I think it's a more likely assumption that on device performance will trail off device models by a significant margin for at least the next few years - of course if magically you can make it work locally with the same level of performance it would be better. Plus a lot of the "agentic" stuff is interaction with the outside world, connectivity is a must regardless.

YetAnotherNick 2y ago

If the model weights is not open, you can't run it on device anyways.

kridsdale1 2y ago
The Pixel 9 runs many small proprietary Gemini models on the internal TPU.
- griomnib 2y ago
  And yet these new models still haven’t reached feature parity with Google Assistant, which can turn my flashlight on, but with all the power of burning down a rainforest, Gemini still cannot interact with my actual phone.
  - lern_too_spel 2y ago
    I just tried asking my phone to turn on the flashlight using Gemini. It worked. https://9to5google.com/2024/11/07/gemini-utilities-extension/ https://9to5google.com/2024/11/07/gemini-utilities-extension...
    - griomnib 2y ago
      Ok I tried literally last week on Pixel 7a and it didn’t work. What model do you have? Maybe it requires a phone that can do on-device models?
      - staticman2 2y ago
        I just tried it on my Galaxy Ultra s23 and it worked. I then disconnected internet and it did not work.
      - _puk 2y ago
        Works on a Pixel 4A 5G.. Pretty sure that's not doing any fancy on-device models! That said, there was a popup today saying that assistant is now using Gemini, so I just enabled it to try. Could well have changed in the last week.