4 ms·

Privacy isn't the only benefit of local compute, users also get colossal bandwidth, tiny latency, and high reliability.

fauigerzigerk 4y ago
Agreed. On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time because it would otherwise drain the battery. Where low latency is not an absolute necessity, the economic pull of the cloud will be tremendous, especially if mobile networks become ubiquitous and fast.
- vinay_ys 4y ago
  That's a weak argument. Lots of hardware sits idle in the cloud as well. And on your phone its not expensive. In fact, the $/tflop is cheaper on phone than in the cloud – cloud has to deal with all kinds of complexity that you assume away in your local single-tenant phone context.
  - fauigerzigerk 4y ago
    I wouldn't be so sure. A quick web search brings up average server utilisation numbers for large-scale cloud providers between 45% and 65%. That's probably an order of magnitude or two higher than what you could do on a mobile device without absolutely annihilating the battery.
- nl 4y ago
  Inactive silicon isn't inefficient. And it's not even particularly wasteful to produce. While there are lots of devices the amount of materials in each chip is incredibly small. In terms of financial cost the material cost is so low that most chip vendors include features in low end chips that are disabled but shipped anyway.
  - moffkalast 4y ago
    Aren't they usually disabled because they don't perform up to spec?
  - fauigerzigerk 4y ago
    That's an interesting point, but I wonder how the cost structure will change as AI becomes a bigger part of the feature set we expect. The battery is a key part as well. Doing a lot of on-device processing requires a bigger battery and/or more frequent battery replacements.
- ClumsyPilot 4y ago
  > On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time Expensive hardware like millions of 5G/6G modems, stations and fiber optic cables nessesary to send vast quantity of photos and videos to the cloud for analysis? ait is more expensive to build out bandwidth than to give everyone compute Also average car is unused 95/% of the time, so by this principle everyone should take a bus
  - fauigerzigerk 4y ago
    >ait is more expensive to build out bandwidth than to give everyone compute I'm not sure about that. Storing all data on all devices (not just user data but also the pretrained models) means higher data transfer volumes. On the other hand, syncing data doesn't require low latency and a lot of the data can be transferred over cheaper landline connections rather than mobile towers. Right now though the default appears to be to upload everything to the cloud right away, regardless of where the data is processed. And there's also communication and media streaming, which consumes the vast majority of network resources. Net net I'm not sure whether on-device vs cloud processing will make a big difference for networking costs one way or another. >Also average car is unused 95/% of the time, so by this principle everyone should take a bus Yes, and it is hugely inefficient. People who own a car (I don't) are doing it for the benefits it has, not because it's efficient.
  - gurleen_s 4y ago
    > Also average car is unused 95/% of the time, so by this principle everyone should take a bus That’s exactly right.
crazygringo 4y ago
On the other hand, it kills your battery. Back when dictation was done in the cloud, I could dictate all day on my iPhone no problem. Now that it's on-device it kills my battery in a couple of hours. The latency is absolutely improved, and continuous dictation (not stopping every 30s) is a godsend. But it does absolutely destroy your battery life.
- bibanez 4y ago
  Don't worry too much because there is Moore's law to the rescue. NPUs benefit from new processes
  - mcculley 4y ago
    Moore's Law makes it a good long term strategy for Apple. The GP is complaining about his battery life today.
    - lucideer 4y ago
      I hope it's not disrespectful to point this out, less than 24 hours after his passing, but I don't think Gordon would object to my pointing out that Moore's Law has a finite length. Some have argued it expired up to 13 years ago; Moore himself predicted another 2 years or so.
      - nwienert 4y ago
        I built an always on local OCR system that used ML on CPU/GPU a few years ago and I can say with confidence it doesn’t use much. We literally scanned your entire screen every two seconds and it used less than 1% in total, and this was before CoreML which is far more efficient. I think it’s FUD that it is that significant.
      - jjoonathan 4y ago
        RF transmission and reception have been bumping smack up against physics limits for a long time. Regardless of whether or not Moore's Law is Dead, it is an extremely good bet that there is more juice left in compute than in RF. For instance, even the "specialized" ML hardware of today has a comfortable factor of 10 in further possible optimization and probably considerably more. Also, due to the fact that these innovations tend to compound, there isn't just more juice, there is e to the power of more juice left. "Moore's Law is Dead" is a bit of a joke in the hardware space because a staggering number of smart people have made staggeringly wrong predictions of this nature (well, typically correct in a narrow sense and wrong in a broad sense). Jim Keller frequently talks about this and has a convincing theory as to why it happens: the industry is full of specialists who are all chasing one particular S-curve and fully understand to the point of conservatively believing in another S-curve or two. Inevitably, this gives them the impression that Moore's Law has just a few years of gas left -- however, it's actually a consequence of limits on human communication, curiosity, and cognition that determines the number of promising S-curves a typical engineer is thinking about. It's not a true evaluation of the supply of additional S-curves waiting in the wings. There's a limit somewhere out there but it's not really "in sight."

Agreed. On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time because it would otherwise drain the battery. Where low latency is not an absolute necessity, the economic pull of the cloud will be tremendous, especially if mobile networks become ubiquitous and fast.

vinay_ys 4y ago
That's a weak argument. Lots of hardware sits idle in the cloud as well. And on your phone its not expensive. In fact, the $/tflop is cheaper on phone than in the cloud – cloud has to deal with all kinds of complexity that you assume away in your local single-tenant phone context.
- fauigerzigerk 4y ago
  I wouldn't be so sure. A quick web search brings up average server utilisation numbers for large-scale cloud providers between 45% and 65%. That's probably an order of magnitude or two higher than what you could do on a mobile device without absolutely annihilating the battery.
nl 4y ago
Inactive silicon isn't inefficient. And it's not even particularly wasteful to produce. While there are lots of devices the amount of materials in each chip is incredibly small. In terms of financial cost the material cost is so low that most chip vendors include features in low end chips that are disabled but shipped anyway.
- moffkalast 4y ago
  Aren't they usually disabled because they don't perform up to spec?
- fauigerzigerk 4y ago
  That's an interesting point, but I wonder how the cost structure will change as AI becomes a bigger part of the feature set we expect. The battery is a key part as well. Doing a lot of on-device processing requires a bigger battery and/or more frequent battery replacements.
ClumsyPilot 4y ago
> On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time Expensive hardware like millions of 5G/6G modems, stations and fiber optic cables nessesary to send vast quantity of photos and videos to the cloud for analysis? ait is more expensive to build out bandwidth than to give everyone compute Also average car is unused 95/% of the time, so by this principle everyone should take a bus
- fauigerzigerk 4y ago
  >ait is more expensive to build out bandwidth than to give everyone compute I'm not sure about that. Storing all data on all devices (not just user data but also the pretrained models) means higher data transfer volumes. On the other hand, syncing data doesn't require low latency and a lot of the data can be transferred over cheaper landline connections rather than mobile towers. Right now though the default appears to be to upload everything to the cloud right away, regardless of where the data is processed. And there's also communication and media streaming, which consumes the vast majority of network resources. Net net I'm not sure whether on-device vs cloud processing will make a big difference for networking costs one way or another. >Also average car is unused 95/% of the time, so by this principle everyone should take a bus Yes, and it is hugely inefficient. People who own a car (I don't) are doing it for the benefits it has, not because it's efficient.
- gurleen_s 4y ago
  > Also average car is unused 95/% of the time, so by this principle everyone should take a bus That’s exactly right.

crazygringo 4y ago

On the other hand, it kills your battery. Back when dictation was done in the cloud, I could dictate all day on my iPhone no problem. Now that it's on-device it kills my battery in a couple of hours. The latency is absolutely improved, and continuous dictation (not stopping every 30s) is a godsend. But it does absolutely destroy your battery life.

bibanez 4y ago
Don't worry too much because there is Moore's law to the rescue. NPUs benefit from new processes
- mcculley 4y ago
  Moore's Law makes it a good long term strategy for Apple. The GP is complaining about his battery life today.
  - lucideer 4y ago
    I hope it's not disrespectful to point this out, less than 24 hours after his passing, but I don't think Gordon would object to my pointing out that Moore's Law has a finite length. Some have argued it expired up to 13 years ago; Moore himself predicted another 2 years or so.
    - nwienert 4y ago
      I built an always on local OCR system that used ML on CPU/GPU a few years ago and I can say with confidence it doesn’t use much. We literally scanned your entire screen every two seconds and it used less than 1% in total, and this was before CoreML which is far more efficient. I think it’s FUD that it is that significant.
    - jjoonathan 4y ago
      RF transmission and reception have been bumping smack up against physics limits for a long time. Regardless of whether or not Moore's Law is Dead, it is an extremely good bet that there is more juice left in compute than in RF. For instance, even the "specialized" ML hardware of today has a comfortable factor of 10 in further possible optimization and probably considerably more. Also, due to the fact that these innovations tend to compound, there isn't just more juice, there is e to the power of more juice left. "Moore's Law is Dead" is a bit of a joke in the hardware space because a staggering number of smart people have made staggeringly wrong predictions of this nature (well, typically correct in a narrow sense and wrong in a broad sense). Jim Keller frequently talks about this and has a convincing theory as to why it happens: the industry is full of specialists who are all chasing one particular S-curve and fully understand to the point of conservatively believing in another S-curve or two. Inevitably, this gives them the impression that Moore's Law has just a few years of gas left -- however, it's actually a consequence of limits on human communication, curiosity, and cognition that determines the number of promising S-curves a typical engineer is thinking about. It's not a true evaluation of the supply of additional S-curves waiting in the wings. There's a limit somewhere out there but it's not really "in sight."