4 ms·

Was just talking about this on reddit like two days ago Instead of data going to models, we need models come to our data which is stored locally and stay local

Was just talking about this on reddit like two days ago Instead of data going to models, we need models come to our data which is stored locally and stay locally. While there are many OSS for Loading personal data, they dont do images or videos. In the future everyone may get their own Model but for now tech is there but product/OSS is missing for everyone to get their own QLORA or RAG or Summarizer. Not just messages/docs: What we read or write, and our thoughts are part of what makes an individual unique. Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today. Was just watching this yesterday https://www.youtube.com/watch?v=zHLCKpmBeKA https://www.youtube.com/watch?v=zHLCKpmBeKA and thought, why we still don't have a computer secretary like her after almost 30 years, who is one step ahead of us.

butz 3y ago
I assume that training LLMs locally require high-end hardware. Even running a model requires a decent CPU or, even better, a high end GPU, but it is not so expensive as training a model. And usually you have to use hardware that is available on the cloud, so not much of privacy here.
- cjbprime 3y ago
  You don't need to train the model on your data: you can use retrieval augmented generation to add the relevant documents to your prompt at query time.
  - butz 3y ago
    Thank you for explanation. I see there is still a lot I have to learn about LLMs.
  - Art9681 3y ago
    This works if the document plus prompt fit in the context window. I suspect the most popular task for this workflow is summary which presumably means large documents. That's when you begin scaling out to a vector store and implementing those more advanced workflows. It does work even by sending a large document on certain local models, but even with the highest tier MacBook Pro a large document can quickly choke up any LLM and bring inference speed to a crawl. Meaning, a powerful client is still required no matter what. Even if you generate embeddings in "real-time" and dump to a vector store that process would be slow in most consumers hardware. If you're passing in smaller documents then it works pretty good for real-time feedback.
- smcleod 3y ago
  As someone else said you don't need to train any models, also - small LLMs (7b~) can run really well even on a base M1 Macbook air from 3 years ago.
simonw 3y ago
"While there are many OSS for Loading personal data, they dont do images or videos" Local models for images are getting pretty good. LLaVA is an LLM with multi-modal image capabilities that runs pretty well on my laptop: https://simonwillison.net/2023/Nov/29/llamafile/ https://simonwillison.net/2023/Nov/29/llamafile/ Models like Salesforce BLIP can be used to generate captions for images too - I built a little CLI tool far that here: https://github.com/simonw/blip-caption https://github.com/simonw/blip-caption
- orbital-decay 3y ago
  CogVLM blows LLaVA out of the water, although it needs a beefier machine (quantized low-res version barely fits into 12GB VRAM, not sure about the accuracy of that).
  - cinntaile 3y ago
    I have no actual knowledge in this area so I'm not sure if it's entirely relevant but an update from the 7th of December on the CogVLM repo says it now works with 11GB of VRAM.
csbartus 3y ago
> Instead of data going to models, we need models come to our data which is stored locally and stay locally. That's the most important idea I've read since ChatGPT / last year. I'll wait for this. Then build my own private AI. And share it / pair it for learning with other private AIs, like a blogroll. As always, there will be two 'different' AIs: a.) the mainstream, centralized, ad/revenue-driven, capitalist, political, controlling / exploiting etc. b.) personal, trustworthy, polished on peer networks, fun, profitable for one / a small community. If by chance, commercial models will be better than open source models, due to better access to computing power / data, please let me know. We can go back to SETI and share our idle computing power / existing knowledge
jakderrida 3y ago
> Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today. I was imagining something a little more ambitious. Like a model that uses our search history and behavior to derive how to best compose a search query. Bing Chat's search queries look like what my uncle would type right after I explained to him what a search engine is. Throw in some advanced operators like site: or filetype: or at least parentheses along with AND/OR. Surely, we can fine tune it to emulate the search processes of the most impressive researchers, paralegals, and teenagers on the spectrum that immediately factcheck your grandpop's Ellis Island story, with evidence he both arrived at first and was naturalized in Chicago.
amelius 3y ago
Local compute is so 80s, when people moved away from dumb terminals and mainframes, to PCs.
- simondotau 3y ago
  Yes but this time we call it “distributed computing”or “edge computing” instead.
- gpderetta 3y ago
  remote computing is so late '90s when people moved away from PCs to servers (the dot in dot com). Turns out this sort of stuff is cyclical.
conradev 3y ago
> Instead of data going to models, we need models come to our data which is stored locally and stay locally. We are building this over at https://software.inc https://software.inc! We collect data about you (from your computer and the internet) into a local database and then teach models how to use it. The models can either be local or cloud-based, and we can route requests based on the sensitivity of the data or the capabilities needed. We're also hiring if that sounds interesting!
- gardenhedge 3y ago
  Wow, nice domain. I'd work there for the name alone haha.
  - voakbasda 3y ago
    Am I cynical thinking the opposite? I can’t imagine they got that domain for a song. Spending a pile of cash on vanity such as that is a real turn off for me; it signals more flash than bang. Am I wrong to think this?
    - nickthegreek 3y ago
      I’d be worried about the ability to be in relevant searches with a name so generic.
      - otteromkram 3y ago
        I've never ever ran a query for "software inc" before. They should be okay. Plus, search engines usually catch up based in click-throughs, bounces, financial kickbacks (cough), too. Searching for Go programming language stuff was a pain a few years back, but now engines have adapted to Go or Golang. I don't use Google, so ymmv.
    - conradev 3y ago
      You are not wrong to think this – spending a pile of cash on a name is a big decision that you want to approach with rigor. We didn't do that, though. Our domain was available for like $4,000. The .inc TLD is intentionally expensive to discourage domain squatting :-)
pradn 3y ago
Yes, should have local models in addition to remote models. Remote ones are always going to be more capable, and we shouldn't throw that away. Augmentation is orthogonal - you can augment either of these with your own data.
nullc 3y ago
Just having an archiver that gives you a tradition search over every webpage you've loaded-- forget the AI stuff, would be a major advance. I don't know about everyone but a majority of searches are for stuff I've seen before, and they're often frustrated by things that have gone offline or are downranked by search engines (e.g. old documentation on HTTP only sites) or burred by SEO.
- mclightning 3y ago
  you will be shocked when you try Rewind then...
timenova 3y ago
I believe that's exactly what GitHub Copilot does. It first scans and indexes your entire codebase including dependencies (I think). So when it auto-completes, it heavily uses the context of your code, which actually makes Copilot so useful. You're absolutely right about models coming to our data! If we could have Copilot-like intelligence, completely on-device, scanning all sorts of personal breadcrumbs like messages, browsing history, even webpage content, it would be a game-changer!
deleted 3y ago
[deleted]
diceduckmonk 3y ago
Google already tried this 15 years ago https://en.m.wikipedia.org/wiki/Google_Search_Appliance https://en.m.wikipedia.org/wiki/Google_Search_Appliance

butz 3y ago

I assume that training LLMs locally require high-end hardware. Even running a model requires a decent CPU or, even better, a high end GPU, but it is not so expensive as training a model. And usually you have to use hardware that is available on the cloud, so not much of privacy here.

cjbprime 3y ago
You don't need to train the model on your data: you can use retrieval augmented generation to add the relevant documents to your prompt at query time.
- butz 3y ago
  Thank you for explanation. I see there is still a lot I have to learn about LLMs.
- Art9681 3y ago
  This works if the document plus prompt fit in the context window. I suspect the most popular task for this workflow is summary which presumably means large documents. That's when you begin scaling out to a vector store and implementing those more advanced workflows. It does work even by sending a large document on certain local models, but even with the highest tier MacBook Pro a large document can quickly choke up any LLM and bring inference speed to a crawl. Meaning, a powerful client is still required no matter what. Even if you generate embeddings in "real-time" and dump to a vector store that process would be slow in most consumers hardware. If you're passing in smaller documents then it works pretty good for real-time feedback.
smcleod 3y ago
As someone else said you don't need to train any models, also - small LLMs (7b~) can run really well even on a base M1 Macbook air from 3 years ago.

simonw 3y ago

"While there are many OSS for Loading personal data, they dont do images or videos" Local models for images are getting pretty good. LLaVA is an LLM with multi-modal image capabilities that runs pretty well on my laptop: https://simonwillison.net/2023/Nov/29/llamafile/ https://simonwillison.net/2023/Nov/29/llamafile/ Models like Salesforce BLIP can be used to generate captions for images too - I built a little CLI tool far that here: https://github.com/simonw/blip-caption https://github.com/simonw/blip-caption

orbital-decay 3y ago
CogVLM blows LLaVA out of the water, although it needs a beefier machine (quantized low-res version barely fits into 12GB VRAM, not sure about the accuracy of that).
- cinntaile 3y ago
  I have no actual knowledge in this area so I'm not sure if it's entirely relevant but an update from the 7th of December on the CogVLM repo says it now works with 11GB of VRAM.

csbartus 3y ago

> Instead of data going to models, we need models come to our data which is stored locally and stay locally. That's the most important idea I've read since ChatGPT / last year. I'll wait for this. Then build my own private AI. And share it / pair it for learning with other private AIs, like a blogroll. As always, there will be two 'different' AIs: a.) the mainstream, centralized, ad/revenue-driven, capitalist, political, controlling / exploiting etc. b.) personal, trustworthy, polished on peer networks, fun, profitable for one / a small community. If by chance, commercial models will be better than open source models, due to better access to computing power / data, please let me know. We can go back to SETI and share our idle computing power / existing knowledge

jakderrida 3y ago

> Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today. I was imagining something a little more ambitious. Like a model that uses our search history and behavior to derive how to best compose a search query. Bing Chat's search queries look like what my uncle would type right after I explained to him what a search engine is. Throw in some advanced operators like site: or filetype: or at least parentheses along with AND/OR. Surely, we can fine tune it to emulate the search processes of the most impressive researchers, paralegals, and teenagers on the spectrum that immediately factcheck your grandpop's Ellis Island story, with evidence he both arrived at first and was naturalized in Chicago.

amelius 3y ago

Local compute is so 80s, when people moved away from dumb terminals and mainframes, to PCs.

simondotau 3y ago
Yes but this time we call it “distributed computing”or “edge computing” instead.
gpderetta 3y ago
remote computing is so late '90s when people moved away from PCs to servers (the dot in dot com). Turns out this sort of stuff is cyclical.

conradev 3y ago

> Instead of data going to models, we need models come to our data which is stored locally and stay locally. We are building this over at https://software.inc https://software.inc! We collect data about you (from your computer and the internet) into a local database and then teach models how to use it. The models can either be local or cloud-based, and we can route requests based on the sensitivity of the data or the capabilities needed. We're also hiring if that sounds interesting!

gardenhedge 3y ago
Wow, nice domain. I'd work there for the name alone haha.
- voakbasda 3y ago
  Am I cynical thinking the opposite? I can’t imagine they got that domain for a song. Spending a pile of cash on vanity such as that is a real turn off for me; it signals more flash than bang. Am I wrong to think this?
  - nickthegreek 3y ago
    I’d be worried about the ability to be in relevant searches with a name so generic.
    - otteromkram 3y ago
      I've never ever ran a query for "software inc" before. They should be okay. Plus, search engines usually catch up based in click-throughs, bounces, financial kickbacks (cough), too. Searching for Go programming language stuff was a pain a few years back, but now engines have adapted to Go or Golang. I don't use Google, so ymmv.
  - conradev 3y ago
    You are not wrong to think this – spending a pile of cash on a name is a big decision that you want to approach with rigor. We didn't do that, though. Our domain was available for like $4,000. The .inc TLD is intentionally expensive to discourage domain squatting :-)

pradn 3y ago

Yes, should have local models in addition to remote models. Remote ones are always going to be more capable, and we shouldn't throw that away. Augmentation is orthogonal - you can augment either of these with your own data.

nullc 3y ago

Just having an archiver that gives you a tradition search over every webpage you've loaded-- forget the AI stuff, would be a major advance. I don't know about everyone but a majority of searches are for stuff I've seen before, and they're often frustrated by things that have gone offline or are downranked by search engines (e.g. old documentation on HTTP only sites) or burred by SEO.

mclightning 3y ago
you will be shocked when you try Rewind then...

timenova 3y ago

I believe that's exactly what GitHub Copilot does. It first scans and indexes your entire codebase including dependencies (I think). So when it auto-completes, it heavily uses the context of your code, which actually makes Copilot so useful. You're absolutely right about models coming to our data! If we could have Copilot-like intelligence, completely on-device, scanning all sorts of personal breadcrumbs like messages, browsing history, even webpage content, it would be a game-changer!

deleted 3y ago

[deleted]

diceduckmonk 3y ago

Google already tried this 15 years ago https://en.m.wikipedia.org/wiki/Google_Search_Appliance https://en.m.wikipedia.org/wiki/Google_Search_Appliance