3 ms·

> any cloud based device is subject to the health of the company and LLMs are not cheap to run Perhaps the technical angle to this story is the promise of edge

> any cloud based device is subject to the health of the company and LLMs are not cheap to run Perhaps the technical angle to this story is the promise of edge ML. If your language model runs on the device, your cloud inference costs go to zero and the device works as long as it has power. Aside from the financial benefits, there’s a huge privacy upside as well since no audio or text is sent over a wire. Might be notable for a children’s toy. Of course, this is very difficult for large companies and VC-backed startups to care about because 1) it involves hard technical problems rather than API calls and 2) as long as you can keep asking for money the inference costs don’t matter and 3) there are no criminal consequences (prison time) for privacy violations.

anonym29 2y ago
In spite of my hugely biased pro-privacy stance, I'm skeptical that real privacy is a feature or selling point that can contribute to the financial interests of corporations enough to consider, broadly speaking. To the corporations, data is the new oil, and to the vast majority of consumers, there's this very defeatist attitude around privacy, something like "I already don't have any privacy, what's one more recording device going to do?". I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks, and I sadly don't see much hope here at all. In fact, I think a substantial plurality, if not an outright majority, even have the technical aptitude to critically evaluate the accuracy of corporate claims of privacy, like those misleading claims of privacy offered by Apple. Humans are highly adaptable creatures, and sadly, I think most have settled quite comfortably into the panopticon that modern society has mandated for so many of us, embedded deep down in the terms of service that everyone agrees to but never reads.
- burningChrome 2y ago
  >> I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks, Do you see this as a generational thing? I remember working for wireless companies in the 1900's when GPS got huge and companies wanted to start using tracking apps for their fleet management and every company we talked to refused to install it because of privacy concerns and the drivers (and sometimes unions) being 100% opposed to it. Now? We have what? Two to three generations who have never valued their privacy enough to really do anything about it. While I agree with your assertion, I whole heartedly believe the road back to people seeing privacy as important may have effectively died with the Gen Xers.
  - janalsncm 2y ago
    It’s a bit more nuanced than people simply not caring. All things being equal, people will choose not to be spied on. Apple showed that pretty conclusively in my mind with the “allow app to track” toggle. If your generational hypothesis is correct we should see that older people are less likely to allow the app to track. I personally doubt that but I don’t have the data. More likely is that young people are more likely to use newer technologies at all, and those technologies have other side effects. People care, imo. Companies know this so they make it a pain in the butt to opt out. Plus, the harms are very abstract and rarely materialize. So unless a person has a lot of time to spend configuring everything, they usually don’t waste time turning tracking off.
- janalsncm 2y ago
  I agree, I don’t expect privacy to be popular enough to drive the R&D. But models small enough to run on-device will be a byproduct of other cost cutting, which means we will be able to turn off telemetry via other means or even develop more privacy-conscious devices. For the layman, this will allow them to bypass a monthly cloud subscription fee which is a killer feature.
im3w1l 2y ago
I don't think edge ML is competitive for now. It can do simple things but not the big beefy LLMs. State of the art AI chips are so expensive that you can't afford to idle them. They need to be M:N shared - M chips for N users so they have maximum utilization, and that fits perfectly for the cloud. However, there is a middle ground: pluggable AI could potentially be a thing. The device would use an open protocol to access cloud AI. If the original company goes bankrupt then someone else can implement the protocol and the devices can be repointed.
- janalsncm 2y ago
  Yeah definitely, especially for more complicated things like having a conversation. State of the Art will probably always need a server. For simpler things, small models can definitely handle them. Transcription, object detection, simple classification tasks. I expect more and more to fall under the category of “things which ML can do on $X of hardware” as hardware and software get better.

anonym29 2y ago

In spite of my hugely biased pro-privacy stance, I'm skeptical that real privacy is a feature or selling point that can contribute to the financial interests of corporations enough to consider, broadly speaking. To the corporations, data is the new oil, and to the vast majority of consumers, there's this very defeatist attitude around privacy, something like "I already don't have any privacy, what's one more recording device going to do?". I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks, and I sadly don't see much hope here at all. In fact, I think a substantial plurality, if not an outright majority, even have the technical aptitude to critically evaluate the accuracy of corporate claims of privacy, like those misleading claims of privacy offered by Apple. Humans are highly adaptable creatures, and sadly, I think most have settled quite comfortably into the panopticon that modern society has mandated for so many of us, embedded deep down in the terms of service that everyone agrees to but never reads.

burningChrome 2y ago
>> I think we should not expect privacy to meaningfully improve until the gap between end-user perceptions of the value of their privacy and corporate perceptions of the value of their customers' (or users') data shrinks, Do you see this as a generational thing? I remember working for wireless companies in the 1900's when GPS got huge and companies wanted to start using tracking apps for their fleet management and every company we talked to refused to install it because of privacy concerns and the drivers (and sometimes unions) being 100% opposed to it. Now? We have what? Two to three generations who have never valued their privacy enough to really do anything about it. While I agree with your assertion, I whole heartedly believe the road back to people seeing privacy as important may have effectively died with the Gen Xers.
- janalsncm 2y ago
  It’s a bit more nuanced than people simply not caring. All things being equal, people will choose not to be spied on. Apple showed that pretty conclusively in my mind with the “allow app to track” toggle. If your generational hypothesis is correct we should see that older people are less likely to allow the app to track. I personally doubt that but I don’t have the data. More likely is that young people are more likely to use newer technologies at all, and those technologies have other side effects. People care, imo. Companies know this so they make it a pain in the butt to opt out. Plus, the harms are very abstract and rarely materialize. So unless a person has a lot of time to spend configuring everything, they usually don’t waste time turning tracking off.
janalsncm 2y ago
I agree, I don’t expect privacy to be popular enough to drive the R&D. But models small enough to run on-device will be a byproduct of other cost cutting, which means we will be able to turn off telemetry via other means or even develop more privacy-conscious devices. For the layman, this will allow them to bypass a monthly cloud subscription fee which is a killer feature.

im3w1l 2y ago

I don't think edge ML is competitive for now. It can do simple things but not the big beefy LLMs. State of the art AI chips are so expensive that you can't afford to idle them. They need to be M:N shared - M chips for N users so they have maximum utilization, and that fits perfectly for the cloud. However, there is a middle ground: pluggable AI could potentially be a thing. The device would use an open protocol to access cloud AI. If the original company goes bankrupt then someone else can implement the protocol and the devices can be repointed.

janalsncm 2y ago
Yeah definitely, especially for more complicated things like having a conversation. State of the Art will probably always need a server. For simpler things, small models can definitely handle them. Transcription, object detection, simple classification tasks. I expect more and more to fall under the category of “things which ML can do on $X of hardware” as hardware and software get better.