4 ms·

I am completely convinced the future is local LLMs, privet, on your device. But then also fine tuned with your own data. All your documents, email, movements, b

I am completely convinced the future is local LLMs, privet, on your device. But then also fine tuned with your own data. All your documents, email, movements, browser history - maybe it even gets trained on your calls. Effectively an extension to your own brain. I really don't want my AI assistant in the cloud, controlled by a corporation, reading and seeing everything I do. I would go as far as saying I want local AI more and open source AI. Another comment said they want it to be an offline Wikipedia, we don't need that, we need it to know all our own stuff, that's what we work with every day.

sircastor 3y ago
I also don’t want my AI in the cloud favoring a corporation’s values and goals. But these tools are almost certainly going to come from them. And they want you to store all your data in the cloud too. And they want to keep you from running the software you want.
- samwillis 3y ago
  I think there is one company very well placed to do this, Apple. Their revenue is driven by hardware sales, rather than advertising, and they have committed to some level of "privacy" focussed design (although obviously there are criticisms). On top of that their hardware is looking almost like it was design for exactly these use cases. Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs. Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud. They are looking for "the next big thing" that lets them sell hardware. Some people thinks that's going to be their VR/AR headset, I don't. It's going to be local AI running privately on your device. Putting the AI in the cloud doesn't help with hardware sales. My only hesitation is that they have such a crap spell checker, surly you would solve first...
  - synctext 3y ago
    > Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs. Yes, university research from 2011 already showed a privacy focused design to machine learning[1]. But will this be private or public like Wikipedia, Bitcoin and Bittorrent? > Effectively an extension to your own brain. Correct. After 1996 fantasy/visionary publications about this, we now know roughly how to do this. Most difficult problem is who should own it, no corporation in control is desired. That means full decentralisation, federated learning is not enough. But fully decentralised learning is hard. Try to apply machine learning in permissionless, byzantine, unsupervised, decentralised, adversarial, continuous learning context. See my lab at Delft University focused for a decade already on "The Global Brain"[2]. With 2.3 million download and crowd-sourcing, we might get there.. [1] "Gossip Learning with Linear Models on Fully Distributed Data", 2011, https://arxiv.org/abs/1109.1396 https://arxiv.org/abs/1109.1396 [2] The Global Brain - the roadmap, https://github.com/Tribler/tribler/issues/7064 https://github.com/Tribler/tribler/issues/7064
  - tinus_hn 3y ago
    I think Apple will be hesitant to release an AI on their platforms that can be made to say unpredictably bad things. For now as far as I have seen there’s only censoring and that’ll only lead to a game of people trying to work around it.
  - fsflover 3y ago
    > Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud. Looking at how they promote using iCloud on iDevices, I doubt that.
    - mrtranscendence 3y ago
      I don't think iCloud is a great example, because there's a definite value add in having your files synced between multiple devices and your computer. A better example might be Siri, which does basically nothing without an internet connection.
      - fsflover 3y ago
        iCloud provides a lot of value, because storage is extremely expensive on all iDevices, and you can't upgrade it. Also Syncthing can be used for a synchronization between devices, if you control them.
  - triggerity 3y ago
    Pretty every Google, Microsoft, Amazon, Samsung and apple can do that. Everyone has a huge motivation to get this solved. But because all of them also have their cloud storage and every customer is trusting their company I'm pretty sure most will not use local ai. Until when suddenly in 10 years we have hardware chips like gpt-8 hardware version or something.
- cykros 3y ago
  Eh. Patience. In the 70s, text editors, operating systems, and just about everything came from corporations, governments, and perhaps most commonly, universities when it came to software tools, and was hosted on their systems. As the hardware became more accessible, and the power needs diminished, it moved out from the mainframe to the client side. This process will take time, but if we don't have the equivalent of chatGPT available on a home system (if not phone sized device) in 20 years, I'll eat my hat. There are already open source projects on Github that'll let you roll your own, but good luck coming up with the computing power.
  - thethimble 3y ago
    And yet more recently everything began shifting to the cloud (storage, compute). Centralization leads to better economies of scale and therefore more efficiency. It’s not clear to me this will ever change. While local compute will always improve (so local models become more viable), I believe that the largest and most capable models will always be cloud based.
    - IwantAIPlus 3y ago
      If efficiency was the only metric, then I agree. However, we don't all need the largest and most capable models all the time. For example, stockfish running on my phone would destroy me and almost every human on Earth, even if every human collaborated together to try to beat it. Also, centralization isn't always the panacea people make it out to be. Google search was very efficient and centralized. They're about to have their lunch eaten by a start-up from nowhere. McDonald's hamburger is very efficient, but I eat that sparingly. Varieties and different (inefficient) paths are still viable and usually needed to progress further. In many cases, centralization in the name of efficiency usually spells the beginning of the end for many firms (and countries).
skinkestek 3y ago
https://mitta.us https://mitta.us is one project that I find myself attracted to but haven't dare to try because it isn't local. It does 25% of what you ask (subtracted 50% because it isn't local and 25% because it is missing calls, movies etc). I wonder if at some point there will be an open source project like this?
mFixman 3y ago
Computers are becoming smaller and the internet is becoming even more ubiquous. I don't see your future happening. Why would I spend a lot of money, space, and maintenance time for a computer with a good GPU when I can just use any of the two internet connections I'm already paying for? I'm already trusting corporations to read my email, and no ChatGPT answer can get more personal than that. This is also why I think cloud gaming is only going to become more popular.
- jaimex2 3y ago
  Cloud gaming was a crash and burn story already. There's a stronger resistance to cloud than ever before after all the leaks, hacks and pushed agendas. These models aren't cheap to host and they'll lose to free local models that don't have boundaries or agendas. Cloud AIs inevitably will try and monetise by selling you crap or be funded to shape your opinions.
  - shortcake27 3y ago
    > Cloud gaming was a crash and burn story already. What? No it isn’t. One company failing doesn’t mean the industry has crashed and burned. I use Xbox cloud gaming all the time along with millions of other people.
    - Krutonium 3y ago
      Even Microsoft has admitted (between the lines) that cloud gaming isn't as profitable as just selling games. And it's not just one company that's failed either. Google, Onlive, Gaikai, there was a company that made a Cloud console a couple years back, the list goes on. Besides, the latency will never be low enough, and the bandwidth cheap enough, for a lot of people to make it a viable solution.
  - triggerity 3y ago
    Gaming is a total different field. Hard to make any assumptions out of this.most people I know definitely sync there data onto companies cloud storage. Alone for image backups and sharing and etc. No one minds that. No one minds mail analysis either. Cloud is healthy, very healthy
d3vmax 3y ago
I remember google desktop search. It was amazing, even today's win 11 search doesnt come close. So something snappy like that with LLM will be perfect.
- eurekin 3y ago
  Yes! I absolutely loved Google desktop search and hoarded much more pdfs and tex files than otherwise. Very excited for the LLM equivalent. PrivateGPT, although more of a proof of concept right now, looks like it could be a nice fit in future
outofpaper 3y ago
Yes, I whole-heartedly agree. What we need is a proliferation of AI tools. Centralization of compute in the hands of a few corporations could be worse than monoculture farming. If we don't create, mix, and match our own models we're offloading all decisions to opaque companies like the "open" OpenAI.
cookieperson 3y ago
So a hackers main goal will be to get your LLM data, and companies will be backdooring them?
- soupfordummies 3y ago
  Yeah, I would definitely be leaning into the "offline" word in the original title for something like this.
- roarcher 3y ago
  Why would this be a risk? The data is already on your machine. It would be much easier for a hacker to target that data directly, as they do today, than to try to extract some impossibly obfuscated form of it from an LLM.
  - cookieperson 3y ago
    Well if they are interested in data. If they are interested in asking a prompt "hey what's my cc number and password again?" That'd bypass an awful lot of encryption. Would also be easy to find blackmail for a lot of people thinking it's a secure way to lookup crimes.
yieldcrv 3y ago
These things will inevitably be included with the OS soon Desktop, mobile, even consoles - especially video game consoles where games can tap into the shared resource for their NPCs instead of bundling it in their package GPU based token generation will be good enough in speed
koonsolo 3y ago
It's probably the future you want. But fact is, most of peoples emails, documents, browser history, travel history, bookmarks, contacts, etc., already live in the cloud. People are obviously not concerned with it, so I would say the future of LLM's is in the cloud using all of the cloud data.

sircastor 3y ago

I also don’t want my AI in the cloud favoring a corporation’s values and goals. But these tools are almost certainly going to come from them. And they want you to store all your data in the cloud too. And they want to keep you from running the software you want.

samwillis 3y ago
I think there is one company very well placed to do this, Apple. Their revenue is driven by hardware sales, rather than advertising, and they have committed to some level of "privacy" focussed design (although obviously there are criticisms). On top of that their hardware is looking almost like it was design for exactly these use cases. Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs. Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud. They are looking for "the next big thing" that lets them sell hardware. Some people thinks that's going to be their VR/AR headset, I don't. It's going to be local AI running privately on your device. Putting the AI in the cloud doesn't help with hardware sales. My only hesitation is that they have such a crap spell checker, surly you would solve first...
- synctext 3y ago
  > Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs. Yes, university research from 2011 already showed a privacy focused design to machine learning[1]. But will this be private or public like Wikipedia, Bitcoin and Bittorrent? > Effectively an extension to your own brain. Correct. After 1996 fantasy/visionary publications about this, we now know roughly how to do this. Most difficult problem is who should own it, no corporation in control is desired. That means full decentralisation, federated learning is not enough. But fully decentralised learning is hard. Try to apply machine learning in permissionless, byzantine, unsupervised, decentralised, adversarial, continuous learning context. See my lab at Delft University focused for a decade already on "The Global Brain"[2]. With 2.3 million download and crowd-sourcing, we might get there.. [1] "Gossip Learning with Linear Models on Fully Distributed Data", 2011, https://arxiv.org/abs/1109.1396 https://arxiv.org/abs/1109.1396 [2] The Global Brain - the roadmap, https://github.com/Tribler/tribler/issues/7064 https://github.com/Tribler/tribler/issues/7064
- tinus_hn 3y ago
  I think Apple will be hesitant to release an AI on their platforms that can be made to say unpredictably bad things. For now as far as I have seen there’s only censoring and that’ll only lead to a game of people trying to work around it.
- fsflover 3y ago
  > Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud. Looking at how they promote using iCloud on iDevices, I doubt that.
  - mrtranscendence 3y ago
    I don't think iCloud is a great example, because there's a definite value add in having your files synced between multiple devices and your computer. A better example might be Siri, which does basically nothing without an internet connection.
    - fsflover 3y ago
      iCloud provides a lot of value, because storage is extremely expensive on all iDevices, and you can't upgrade it. Also Syncthing can be used for a synchronization between devices, if you control them.
- triggerity 3y ago
  Pretty every Google, Microsoft, Amazon, Samsung and apple can do that. Everyone has a huge motivation to get this solved. But because all of them also have their cloud storage and every customer is trusting their company I'm pretty sure most will not use local ai. Until when suddenly in 10 years we have hardware chips like gpt-8 hardware version or something.
cykros 3y ago
Eh. Patience. In the 70s, text editors, operating systems, and just about everything came from corporations, governments, and perhaps most commonly, universities when it came to software tools, and was hosted on their systems. As the hardware became more accessible, and the power needs diminished, it moved out from the mainframe to the client side. This process will take time, but if we don't have the equivalent of chatGPT available on a home system (if not phone sized device) in 20 years, I'll eat my hat. There are already open source projects on Github that'll let you roll your own, but good luck coming up with the computing power.
- thethimble 3y ago
  And yet more recently everything began shifting to the cloud (storage, compute). Centralization leads to better economies of scale and therefore more efficiency. It’s not clear to me this will ever change. While local compute will always improve (so local models become more viable), I believe that the largest and most capable models will always be cloud based.
  - IwantAIPlus 3y ago
    If efficiency was the only metric, then I agree. However, we don't all need the largest and most capable models all the time. For example, stockfish running on my phone would destroy me and almost every human on Earth, even if every human collaborated together to try to beat it. Also, centralization isn't always the panacea people make it out to be. Google search was very efficient and centralized. They're about to have their lunch eaten by a start-up from nowhere. McDonald's hamburger is very efficient, but I eat that sparingly. Varieties and different (inefficient) paths are still viable and usually needed to progress further. In many cases, centralization in the name of efficiency usually spells the beginning of the end for many firms (and countries).

skinkestek 3y ago

https://mitta.us https://mitta.us is one project that I find myself attracted to but haven't dare to try because it isn't local. It does 25% of what you ask (subtracted 50% because it isn't local and 25% because it is missing calls, movies etc). I wonder if at some point there will be an open source project like this?

mFixman 3y ago

Computers are becoming smaller and the internet is becoming even more ubiquous. I don't see your future happening. Why would I spend a lot of money, space, and maintenance time for a computer with a good GPU when I can just use any of the two internet connections I'm already paying for? I'm already trusting corporations to read my email, and no ChatGPT answer can get more personal than that. This is also why I think cloud gaming is only going to become more popular.

jaimex2 3y ago
Cloud gaming was a crash and burn story already. There's a stronger resistance to cloud than ever before after all the leaks, hacks and pushed agendas. These models aren't cheap to host and they'll lose to free local models that don't have boundaries or agendas. Cloud AIs inevitably will try and monetise by selling you crap or be funded to shape your opinions.
- shortcake27 3y ago
  > Cloud gaming was a crash and burn story already. What? No it isn’t. One company failing doesn’t mean the industry has crashed and burned. I use Xbox cloud gaming all the time along with millions of other people.
  - Krutonium 3y ago
    Even Microsoft has admitted (between the lines) that cloud gaming isn't as profitable as just selling games. And it's not just one company that's failed either. Google, Onlive, Gaikai, there was a company that made a Cloud console a couple years back, the list goes on. Besides, the latency will never be low enough, and the bandwidth cheap enough, for a lot of people to make it a viable solution.
- triggerity 3y ago
  Gaming is a total different field. Hard to make any assumptions out of this.most people I know definitely sync there data onto companies cloud storage. Alone for image backups and sharing and etc. No one minds that. No one minds mail analysis either. Cloud is healthy, very healthy

d3vmax 3y ago

I remember google desktop search. It was amazing, even today's win 11 search doesnt come close. So something snappy like that with LLM will be perfect.

eurekin 3y ago
Yes! I absolutely loved Google desktop search and hoarded much more pdfs and tex files than otherwise. Very excited for the LLM equivalent. PrivateGPT, although more of a proof of concept right now, looks like it could be a nice fit in future

outofpaper 3y ago

Yes, I whole-heartedly agree. What we need is a proliferation of AI tools. Centralization of compute in the hands of a few corporations could be worse than monoculture farming. If we don't create, mix, and match our own models we're offloading all decisions to opaque companies like the "open" OpenAI.

cookieperson 3y ago

So a hackers main goal will be to get your LLM data, and companies will be backdooring them?

soupfordummies 3y ago
Yeah, I would definitely be leaning into the "offline" word in the original title for something like this.
roarcher 3y ago
Why would this be a risk? The data is already on your machine. It would be much easier for a hacker to target that data directly, as they do today, than to try to extract some impossibly obfuscated form of it from an LLM.
- cookieperson 3y ago
  Well if they are interested in data. If they are interested in asking a prompt "hey what's my cc number and password again?" That'd bypass an awful lot of encryption. Would also be easy to find blackmail for a lot of people thinking it's a secure way to lookup crimes.

yieldcrv 3y ago

These things will inevitably be included with the OS soon Desktop, mobile, even consoles - especially video game consoles where games can tap into the shared resource for their NPCs instead of bundling it in their package GPU based token generation will be good enough in speed

koonsolo 3y ago

It's probably the future you want. But fact is, most of peoples emails, documents, browser history, travel history, bookmarks, contacts, etc., already live in the cloud. People are obviously not concerned with it, so I would say the future of LLM's is in the cloud using all of the cloud data.