9 ms·

Why we don’t have an LLM based search tool for our pc / smartphones? Specially for the smartphones all of your data is on the cloud anyway, instead of just scr

Why we don’t have an LLM based search tool for our pc / smartphones? Specially for the smartphones all of your data is on the cloud anyway, instead of just scraping it for advertising and the FBI they could also do something useful for the user?

dmbche 2y ago
It doesn't solve any problem, you can just search your files using your prefered file explorer (crtl-f) I'd assume most people organise their files so that they know where things are as well.
- whatever1 2y ago
  But file explorer does not read the actual files and build context. Even for pure text files that sometimes search functions can also access, I need to remember exactly the string of characters I am looking for. I was hoping an LLM would have a context of all of my content (text and visual) and for the first time use my computers data as a knowledge base. Queries like “what was my design file for that x service” ? Today it’s impossible to answer unless you have organized your data your self. Why do we still have to organize our data manually?
  - pests 2y ago
    The photos apps do this well now. Can search Apple/Google photos with questions about the content of images and videos and get useful results.
- nine_k 2y ago
  > you can just search your files using your prefered file explorer This only work if you remember specific substrings. An LLM (or some other language model) can summarize and interpolate. It can be asked to find that file that mentions a transaction for buying candy, and it has a fair chance to find it, even if none of the words "transaction", "buying" or "candy" are present in the file, e.g. it says "shelled out $17 for a huge pack of gobstoppers". > I'd assume most people organise their files You'll be shocked, but...
  - ozim 2y ago
    I think the same, people are not organized - even with things that make them money and being organized could earn them much more.
  - dmbche 2y ago
    But isn't that candy example non-sensical? In what situation do you need some information without any of the context(or without knowing any of the context)? i really believe that this is not an actual problem in need of solving, but instead creating a tool (personal ai assistant) and trying to find a usecase Edit0: note to self, rambling - assuming there exist valuable information that one needa to access in their files, but one doesn't know where it is, when it was made, it's name or other information about it(as you could find said file right away with this information). Say you need an information for some documentation like the C standard - you need precise information on some process. Is it not much simpler to just open the doc and use the index? Then again for you to be aeare of the C standard makes the query useless. If it's from something less well organised, say you want letters you wrote to your significant other, maybe the assistant could help. But then again, what are you asking? How hard is it to keep your letters in a folder? Or even simply know what you've done (I surely can't imagine forgetting things I've created but somehow finding use in a llm that finds it for me). Like asking it "what is my opinion on x" or "what's a good compliment I wrote" is nonsensical to me, but asking it about external ressources makes the idea of training it on your own data pointless. "How did I write X API" - just open your file, no? You know where it is, you made it. Like saying "get me that picture of unle tony in Florida" might save you 10 seconds instead of going into your files and thinking about when you got that picture, but it's not solving a real issue or making things more efficient. (Edit1: if you don't know Tony, when you got the picture or of what it's a picture of, why are you querying? What's the usecase for this information, is it just to prove it can be done? It feels like the user needs to contorts themselves in a small niche for this product to be useful) Either it's used for non valuable work (menial search) or you already know how to get the answer you need. I cannot imagine a query that would be useful compared to simply being aware of what's in your computer. And if you're not aware of it, how do you search for it?
    - mjlee 2y ago
- ozim 2y ago
  I think you are really wrong. Most people I see at work and outside don’t care and they want stupid machine to deal with it. That is why smartphones and tablets move away from providing „file system” access. It is super annoying for me but most people want to save their tax form or their baby photo not even understanding each is different file type - because they couldn’t care less about file types let alone making folder structure to keep them organized.
- acchow 2y ago
  Curiously, the things I search most often are not located in files: calendar, photo content/location, email, ChatGPT history, Spotify library, iMessage/whatsapp history, contacts, notes, Amazon order history
rudedogg 2y ago
This is roughly what Apple Intelligence was supposed to deliver but has yet to.
visarga 2y ago
I found that ChatGPT or Claude are really good at music and shopping suggestions. Just chat with them about your tastes for a while, then ask for suggestions. Compared to old recommender systems this method allows much better user guidance.
- josephg 2y ago
  Yeah, Claude helped me decide what to get my girlfriend for her birthday a few weeks ago. It suggested some great gift ideas I hadn’t thought of - and my girlfriend loved them.
  - Workaccount2 2y ago
    I think we can expect this to be rapidly monetized.
- KoftaBob 2y ago
  For shopping suggestions, I've had the best experience with Perplexity.
GraemeMeyer 2y ago
It's coming for PCs soon: https://www.theregister.com/2025/01/20/microsoft_unveils_windows_search_improvements/ https://www.theregister.com/2025/01/20/microsoft_unveils_win... And to a certain extent for the Microsoft cloud experience as well: https://www.theverge.com/2024/10/8/24265312/microsoft-onedrive-mobile-app-search-colored-folders-file-explorer https://www.theverge.com/2024/10/8/24265312/microsoft-onedri...
curious_cat_163 2y ago
> Why we don’t have an LLM based search tool for our pc / smartphones? I'll offer my take as an outside observer. If someone has better insights, feel free to share as well. In market terms, I think it is because Google, Microsoft and Apple are all still trying with varied success. It has to be them because that's where a big bulk of the users are. They are all also public companies with impatient investors wanting the stock to go up into the right. So, they are both cautious about what ship to billions of devices (brand protection) and cautious about "opening up" their OS beyond that they have already done (fear of disruption). In technical terms, it is taking a while because if the tool is going to use LLMs, then they need to solve for 99.999% of the reliability problems (brand protection) that come with that tech. They need to solve for power consumption (either on edge or in the data centers) due to their sheer scale. So, their choices are ship fast (which Google has been trying to do more) and iterate in public; or partner with other product companies by investing in them (which Microsoft has been doing with Open AI and Google is doing with Anthropic, etc.). Apple is taking some middle path but they just fired the person who was heading up the initiative [1] so let's see how that goes. My two cents. [1] https://www.reuters.com/technology/artificial-intelligence/apple-shakes-up-ai-executive-ranks-bid-turn-around-siri-bloomberg-news-reports-2025-03-20/ https://www.reuters.com/technology/artificial-intelligence/a...
wildrhythms 2y ago
Pixels already have the Screenshots app that indexes screenshots and makes them searchable. My assumption is the context window size is still too small for all of your data to go into it.

dmbche 2y ago

It doesn't solve any problem, you can just search your files using your prefered file explorer (crtl-f) I'd assume most people organise their files so that they know where things are as well.

whatever1 2y ago
But file explorer does not read the actual files and build context. Even for pure text files that sometimes search functions can also access, I need to remember exactly the string of characters I am looking for. I was hoping an LLM would have a context of all of my content (text and visual) and for the first time use my computers data as a knowledge base. Queries like “what was my design file for that x service” ? Today it’s impossible to answer unless you have organized your data your self. Why do we still have to organize our data manually?
- pests 2y ago
  The photos apps do this well now. Can search Apple/Google photos with questions about the content of images and videos and get useful results.
nine_k 2y ago
> you can just search your files using your prefered file explorer This only work if you remember specific substrings. An LLM (or some other language model) can summarize and interpolate. It can be asked to find that file that mentions a transaction for buying candy, and it has a fair chance to find it, even if none of the words "transaction", "buying" or "candy" are present in the file, e.g. it says "shelled out $17 for a huge pack of gobstoppers". > I'd assume most people organise their files You'll be shocked, but...
- ozim 2y ago
  I think the same, people are not organized - even with things that make them money and being organized could earn them much more.
- dmbche 2y ago
  But isn't that candy example non-sensical? In what situation do you need some information without any of the context(or without knowing any of the context)? i really believe that this is not an actual problem in need of solving, but instead creating a tool (personal ai assistant) and trying to find a usecase Edit0: note to self, rambling - assuming there exist valuable information that one needa to access in their files, but one doesn't know where it is, when it was made, it's name or other information about it(as you could find said file right away with this information). Say you need an information for some documentation like the C standard - you need precise information on some process. Is it not much simpler to just open the doc and use the index? Then again for you to be aeare of the C standard makes the query useless. If it's from something less well organised, say you want letters you wrote to your significant other, maybe the assistant could help. But then again, what are you asking? How hard is it to keep your letters in a folder? Or even simply know what you've done (I surely can't imagine forgetting things I've created but somehow finding use in a llm that finds it for me). Like asking it "what is my opinion on x" or "what's a good compliment I wrote" is nonsensical to me, but asking it about external ressources makes the idea of training it on your own data pointless. "How did I write X API" - just open your file, no? You know where it is, you made it. Like saying "get me that picture of unle tony in Florida" might save you 10 seconds instead of going into your files and thinking about when you got that picture, but it's not solving a real issue or making things more efficient. (Edit1: if you don't know Tony, when you got the picture or of what it's a picture of, why are you querying? What's the usecase for this information, is it just to prove it can be done? It feels like the user needs to contorts themselves in a small niche for this product to be useful) Either it's used for non valuable work (menial search) or you already know how to get the answer you need. I cannot imagine a query that would be useful compared to simply being aware of what's in your computer. And if you're not aware of it, how do you search for it?
  - mjlee 2y ago
ozim 2y ago
I think you are really wrong. Most people I see at work and outside don’t care and they want stupid machine to deal with it. That is why smartphones and tablets move away from providing „file system” access. It is super annoying for me but most people want to save their tax form or their baby photo not even understanding each is different file type - because they couldn’t care less about file types let alone making folder structure to keep them organized.
acchow 2y ago
Curiously, the things I search most often are not located in files: calendar, photo content/location, email, ChatGPT history, Spotify library, iMessage/whatsapp history, contacts, notes, Amazon order history

rudedogg 2y ago

This is roughly what Apple Intelligence was supposed to deliver but has yet to.

visarga 2y ago

I found that ChatGPT or Claude are really good at music and shopping suggestions. Just chat with them about your tastes for a while, then ask for suggestions. Compared to old recommender systems this method allows much better user guidance.

josephg 2y ago
Yeah, Claude helped me decide what to get my girlfriend for her birthday a few weeks ago. It suggested some great gift ideas I hadn’t thought of - and my girlfriend loved them.
- Workaccount2 2y ago
  I think we can expect this to be rapidly monetized.
KoftaBob 2y ago
For shopping suggestions, I've had the best experience with Perplexity.

GraemeMeyer 2y ago

It's coming for PCs soon: https://www.theregister.com/2025/01/20/microsoft_unveils_windows_search_improvements/ https://www.theregister.com/2025/01/20/microsoft_unveils_win... And to a certain extent for the Microsoft cloud experience as well: https://www.theverge.com/2024/10/8/24265312/microsoft-onedrive-mobile-app-search-colored-folders-file-explorer https://www.theverge.com/2024/10/8/24265312/microsoft-onedri...

curious_cat_163 2y ago

> Why we don’t have an LLM based search tool for our pc / smartphones? I'll offer my take as an outside observer. If someone has better insights, feel free to share as well. In market terms, I think it is because Google, Microsoft and Apple are all still trying with varied success. It has to be them because that's where a big bulk of the users are. They are all also public companies with impatient investors wanting the stock to go up into the right. So, they are both cautious about what ship to billions of devices (brand protection) and cautious about "opening up" their OS beyond that they have already done (fear of disruption). In technical terms, it is taking a while because if the tool is going to use LLMs, then they need to solve for 99.999% of the reliability problems (brand protection) that come with that tech. They need to solve for power consumption (either on edge or in the data centers) due to their sheer scale. So, their choices are ship fast (which Google has been trying to do more) and iterate in public; or partner with other product companies by investing in them (which Microsoft has been doing with Open AI and Google is doing with Anthropic, etc.). Apple is taking some middle path but they just fired the person who was heading up the initiative [1] so let's see how that goes. My two cents. [1] https://www.reuters.com/technology/artificial-intelligence/apple-shakes-up-ai-executive-ranks-bid-turn-around-siri-bloomberg-news-reports-2025-03-20/ https://www.reuters.com/technology/artificial-intelligence/a...

wildrhythms 2y ago

Pixels already have the Screenshots app that indexes screenshots and makes them searchable. My assumption is the context window size is still too small for all of your data to go into it.