6 ms·

In general I agree that the tensor hardware has been mostly a disappointment. I mean this more in general performance than in ML. In ML I think the story is a

In general I agree that the tensor hardware has been mostly a disappointment. I mean this more in general performance than in ML. In ML I think the story is a little different. Hardware design/production pipelines are measured in years. The LLM "revolution" is relatively recent. For the Pixel8, I imagine some folks at Google had to make the really tough decision of whether to run weaker, smaller models on device, or run the giant LLM models in the cloud, giving much better results, albeit slower (and with ofc the requirement for an Internet connection).

crest 3y ago
I’m sure Google had to think long and hard if they wanted all those queries forwarded to their data centers to analyse (enrich their user profiles).
- wmf 3y ago
  They could process stuff locally and also send data to the cloud.
  - matrix_overload 3y ago
    There's a big difference. If you process stuff locally, uploading the entire unprocessed data to the cloud would spark a lot of criticism. So, you are stuck running analysis locally and uploading things like relevant ad topics. If you run your processing through the cloud, there is no optics or bandwidth cost of keeping the entire raw data as long as you want, let humans review it, and figure out which metrics to extract from it at a later point. I would say, the latter raises more privacy concerns than the former.
    - vineyardmike 3y ago
      Yea but then they have to process everything. Sending just the relevant information for ads means they save on cost of running compute. Considering Azure has run out of GPUs and is renting from Oracle, it’s not crazy to imagine that they could run out of TPUs and the price of compute out weighs the slightly (?) worse ad targeting.
vineyardmike 3y ago
I totally imagine that they intended to run models locally when this phone was on a whiteboard in an engineering meeting but… (1) the models team just can’t/wont trim the models, instead focusing on bigger and better. This is probably doubly true now that the AI wars have taken over their focus And (2) they realize they probably need to ship these features to every non pixel phone too.
- pmontra 3y ago
  Why point 2 is a problem? They can run locally if there is a chip that can do that and offload to cloud if not, and if there is a connection fast enough to transfer the data both ways in a reasonable time. I'm thinking about photo and (especially) video processing. Translating would probably be OK even on show connections. Furthermore, IMHO shooting photos and videos in the middle of nowhere with no connection is a more common use case than translating something there.
  - rickdeckard 3y ago
    Not disagreeing with you, but when running a product and R&D team implementing and maintaining both routes is a hard sell. From a business and resource-allocation perspective there's not enough ROI, I'd rather have those resources work on improving power management. Also, in the end the whole additional effort would be made for a device which is not well-connected to the internet, a tough acknowledgement for a company whose whole business is based on people connected to the internet. For Qualcomm the story is different: They want to sell the chipset to companies who then build/buy local models to include in their firmware package, just like they're used to for the past 10 years...
  - vineyardmike 3y ago
    Because you have to maintain two code paths that are very different. One of them gets a huge budget (pixel design) and has to rewrite major ML models. Very expensive. All for a small amount of users. The other code path has most of your users, and can use whatever the R&D team throws together. And works for every customer, and can be easily bug-fixed if issues arise. When you’re Google with Billions of users, your long-tail use cases are all common. You have to think about what that behavior looks like regardless.
cesaref 3y ago
I think this is the point - each design team and company bet on what they believe their hardware is likely to need to perform. The performance vs power consumption trade off is really tricky when you have to guess what the software landscape will look like a few years out. Get it right, and you have a winner, get it slightly wrong and you either have 'poor battery life' in reviews, or you offload stuff which ideally would be running locally (which is the suggestion in this case). I'm not sure I want LLMs on my phone anyway - the situations where you want a 'digital assistant' are generally ones where you've got connectivity, so offloading seems like a sensible fit for me. Of course i'm making that assumption and i'll probably look silly, but that's progress right? Of more interest is the heavy lifting for image processing to make the phone camera not suck (it's all smoke and mirrors with phone cameras, and more inference on device helps).
- injidup 3y ago
  I'd love to have an LLM on my phone when no connectivity. There are many situations where a personal assistant would come in handy. In the wilds if Canada somewhere: Hey google. Lead me through the steps to create a splint and stop the bleeding. My buddy just got attacked by a bear and is bleeding out. What? That's not a bear? It's a racoon ? Oh...anyway. Lead me through the steps to start a fire with wet wood and how do I prepare a bear, I mean racoon, carcass for good eating. I have some old copper wire and a magnet. I need to make a phone charger. Lead me through it. Also make sure the fire is smokeless as we don't want to get discovered and sent back to the supermax.
  - withinboredom 3y ago
    "I'm sorry, I can't help with that."
  - thret 3y ago
    Apparently LLM = MacGyver.
  - SketchySeaBeast 3y ago
    Translating that information into an easily understood format is the job of a LLM, but it's not having the data, right? Have we not moved the goalpost a bit if we want a model that contains the majority of human knowledge, including accurate and guaranteed non-hallucinated survival information, to be stored locally on our phone?
silisili 3y ago
99% of smartphone consumers don't care about 'ML performance' as a primary concern. People care how fast apps open, and battery life. From the perspective, the tensor chips are pigs that nobody is excited about.
- stavros 3y ago
  I very much do care if my phone understands my questions accurately when I ask them, and doesn't return irrelevant results, though. Nobody says "yeah Siri is crap, but have you seen that power usage?!".
  - FirmwareBurner 3y ago
    Honestly, I haven't met many people who speak to their phones. [edit: where I live. Can't speak for the north American/Anglophone market] They usually use their phones to speak to other people, hence why signal reception and battery life is one of the most important things I look for. All ML gimmicks are nice to show off at events, pub gatherings and in reviews videos("hey look at what my new phone can do, yeah it looks the same as the one from last year and the year before, but can your phone do THIS? <insert ML gimmick of swapping people's heads in the picture>"), but won't help me if the signal reception quality sucks and drains my battery. Phones need to work good as phones first and foremost. High-end compute abilities come second. My most used ML gimmick is ChatGPT which doesn't run on device anyway, nor is it something I need in case of emergency like you know... being able to call someone to come pick me up/help pe when I need it the most. So IMHO bad modems like the ones in Pixel phones, are inexcusable in a phone no matter how fancy the other features are.
    - matthewmacleod 3y ago
      _They usually use their phones to speak to other people_ Counterpoint – I haven't met many people who use their phones to speak to other people. YMMV.
      - FirmwareBurner 3y ago
croes 3y ago
Speaking of disappointing. What happened to the Soli radar chip in Pixels?
- SketchySeaBeast 3y ago
  You ever go all in on a technology that tries to sell your users on a big bezel in exchange for being able to use their phone without touching it and then have a global emergency come around that largely removes the only reliable way to authenticate the user without them touching the phone?

crest 3y ago

I’m sure Google had to think long and hard if they wanted all those queries forwarded to their data centers to analyse (enrich their user profiles).

wmf 3y ago
They could process stuff locally and also send data to the cloud.
- matrix_overload 3y ago
  There's a big difference. If you process stuff locally, uploading the entire unprocessed data to the cloud would spark a lot of criticism. So, you are stuck running analysis locally and uploading things like relevant ad topics. If you run your processing through the cloud, there is no optics or bandwidth cost of keeping the entire raw data as long as you want, let humans review it, and figure out which metrics to extract from it at a later point. I would say, the latter raises more privacy concerns than the former.
  - vineyardmike 3y ago
    Yea but then they have to process everything. Sending just the relevant information for ads means they save on cost of running compute. Considering Azure has run out of GPUs and is renting from Oracle, it’s not crazy to imagine that they could run out of TPUs and the price of compute out weighs the slightly (?) worse ad targeting.

vineyardmike 3y ago

I totally imagine that they intended to run models locally when this phone was on a whiteboard in an engineering meeting but… (1) the models team just can’t/wont trim the models, instead focusing on bigger and better. This is probably doubly true now that the AI wars have taken over their focus And (2) they realize they probably need to ship these features to every non pixel phone too.

pmontra 3y ago
Why point 2 is a problem? They can run locally if there is a chip that can do that and offload to cloud if not, and if there is a connection fast enough to transfer the data both ways in a reasonable time. I'm thinking about photo and (especially) video processing. Translating would probably be OK even on show connections. Furthermore, IMHO shooting photos and videos in the middle of nowhere with no connection is a more common use case than translating something there.
- rickdeckard 3y ago
  Not disagreeing with you, but when running a product and R&D team implementing and maintaining both routes is a hard sell. From a business and resource-allocation perspective there's not enough ROI, I'd rather have those resources work on improving power management. Also, in the end the whole additional effort would be made for a device which is not well-connected to the internet, a tough acknowledgement for a company whose whole business is based on people connected to the internet. For Qualcomm the story is different: They want to sell the chipset to companies who then build/buy local models to include in their firmware package, just like they're used to for the past 10 years...
- vineyardmike 3y ago
  Because you have to maintain two code paths that are very different. One of them gets a huge budget (pixel design) and has to rewrite major ML models. Very expensive. All for a small amount of users. The other code path has most of your users, and can use whatever the R&D team throws together. And works for every customer, and can be easily bug-fixed if issues arise. When you’re Google with Billions of users, your long-tail use cases are all common. You have to think about what that behavior looks like regardless.

cesaref 3y ago

I think this is the point - each design team and company bet on what they believe their hardware is likely to need to perform. The performance vs power consumption trade off is really tricky when you have to guess what the software landscape will look like a few years out. Get it right, and you have a winner, get it slightly wrong and you either have 'poor battery life' in reviews, or you offload stuff which ideally would be running locally (which is the suggestion in this case). I'm not sure I want LLMs on my phone anyway - the situations where you want a 'digital assistant' are generally ones where you've got connectivity, so offloading seems like a sensible fit for me. Of course i'm making that assumption and i'll probably look silly, but that's progress right? Of more interest is the heavy lifting for image processing to make the phone camera not suck (it's all smoke and mirrors with phone cameras, and more inference on device helps).

injidup 3y ago
I'd love to have an LLM on my phone when no connectivity. There are many situations where a personal assistant would come in handy. In the wilds if Canada somewhere: Hey google. Lead me through the steps to create a splint and stop the bleeding. My buddy just got attacked by a bear and is bleeding out. What? That's not a bear? It's a racoon ? Oh...anyway. Lead me through the steps to start a fire with wet wood and how do I prepare a bear, I mean racoon, carcass for good eating. I have some old copper wire and a magnet. I need to make a phone charger. Lead me through it. Also make sure the fire is smokeless as we don't want to get discovered and sent back to the supermax.
- withinboredom 3y ago
  "I'm sorry, I can't help with that."
- thret 3y ago
  Apparently LLM = MacGyver.
- SketchySeaBeast 3y ago
  Translating that information into an easily understood format is the job of a LLM, but it's not having the data, right? Have we not moved the goalpost a bit if we want a model that contains the majority of human knowledge, including accurate and guaranteed non-hallucinated survival information, to be stored locally on our phone?

silisili 3y ago

99% of smartphone consumers don't care about 'ML performance' as a primary concern. People care how fast apps open, and battery life. From the perspective, the tensor chips are pigs that nobody is excited about.

stavros 3y ago
I very much do care if my phone understands my questions accurately when I ask them, and doesn't return irrelevant results, though. Nobody says "yeah Siri is crap, but have you seen that power usage?!".
- FirmwareBurner 3y ago
  Honestly, I haven't met many people who speak to their phones. [edit: where I live. Can't speak for the north American/Anglophone market] They usually use their phones to speak to other people, hence why signal reception and battery life is one of the most important things I look for. All ML gimmicks are nice to show off at events, pub gatherings and in reviews videos("hey look at what my new phone can do, yeah it looks the same as the one from last year and the year before, but can your phone do THIS? <insert ML gimmick of swapping people's heads in the picture>"), but won't help me if the signal reception quality sucks and drains my battery. Phones need to work good as phones first and foremost. High-end compute abilities come second. My most used ML gimmick is ChatGPT which doesn't run on device anyway, nor is it something I need in case of emergency like you know... being able to call someone to come pick me up/help pe when I need it the most. So IMHO bad modems like the ones in Pixel phones, are inexcusable in a phone no matter how fancy the other features are.
  - matthewmacleod 3y ago
    _They usually use their phones to speak to other people_ Counterpoint – I haven't met many people who use their phones to speak to other people. YMMV.
    - FirmwareBurner 3y ago

croes 3y ago

Speaking of disappointing. What happened to the Soli radar chip in Pixels?

SketchySeaBeast 3y ago
You ever go all in on a technology that tries to sell your users on a big bezel in exchange for being able to use their phone without touching it and then have a global emergency come around that largely removes the only reliable way to authenticate the user without them touching the phone?