3 ms·

Logical direction, but the real killer feature would be for an LLM to read every page you bookmark and not just categorize it once for you, but add it to a quer

Logical direction, but the real killer feature would be for an LLM to read every page you bookmark and not just categorize it once for you, but add it to a queryable knowledge-base that you can reference at any point in future conversations.

purplecats 3y ago
the ability to do this is pretty accessible nowadays, its just a little expensive for normal use for the layman
eagrwl 3y ago
Zenfetch (https://www.zenfetch.com/ https://www.zenfetch.com/) does an awesome job at this!
coolvision 3y ago
There is a nice opensource extension that does exactly that: https://github.com/memex-life/memex https://github.com/memex-life/memex I tried to use it, and it just does not work very well, I don't it's because implementation is bad, it's just RAG (retrieval based generation) does not work well outside of some simple use cases.
- danielbln 3y ago
  What are those simple use cases, and where do you see Retrieval Augmented Generation fall over?
  - coolvision 3y ago
    I think it works better if query is a larger chunk of text. Like, if you have an email from a customer and want to compose a response based on some relevant documentation, it should work well. But for a use case where you want to retrieve something from browsing history you would mainly use a short search query, just few words. in this case embeddings are too ambiguous and relevance of retrieved content is not great.
    - janalsncm 3y ago
      That’s not a problem with RAG itself that’s an issue with your retriever. In the original RAG paper they used two vanilla BERT models and cosine similarity but there’s no requirement you do that. Use any retriever that gets you high precision. Use BM25 if you want, it’s simple and cheap. You’re right in saying there’s not enough semantic meaning in the text of the query. The domain of queries and the domain of documents are very different. That’s why a real retrieval system will train the query encoder and doc encoder to be closer in their embedding space using click data. This is what Google is doing.
      - ayaanmomin 3y ago
        "train the query encoder and doc encoder to be closer in their embedding space using click data" <- Any papers/resources you know where I can learn more about this process?
        janalsncm 3y ago
        Triplet loss. https://pytorch.org/docs/stable/generated/torch.nn.TripletMarginLoss.html https://pytorch.org/docs/stable/generated/torch.nn.TripletMa... Triplet loss takes an anchor, positive, and negative. In this case the anchor is your query, the positive is a similar doc, and the negative is a dissimilar doc. When you train, backpropagate the loss to both the doc and the query encoder.
visarga 3y ago
alway wonder why browsers don't keep at least a text version of the pages for search nowadays I expect browsers to incorporate small LLMs like Mistral out of the box
- coolvision 3y ago
  if history search worked better then you would visit google/bing less, so I expect pushback from browser vendors also: https://bugs.chromium.org/p/chromium/issues/detail?id=297648 https://bugs.chromium.org/p/chromium/issues/detail?id=297648
  - TeMPOraL 3y ago
    From a comment on that issue page, dating a little over a decade ago: > Shouldn't this be left to users to decide whether they want a feature or not. Yes, those were good times. Nowadays, it's anathema to software vendors, and increasingly even to open source devs.
  - totetsu 3y ago
    Another advertising revenue win for Query Jerry from marketing.
  - poyu 3y ago
    Not every browser vendor runs a search engine…
    - janalsncm 3y ago
      Apple and Mozilla aren’t Google. Apple runs their own searches on the App Store and iPhone, but Google pays them to be default. There’s no reason Apple couldn’t also index your search history in a more meaningful way other than perhaps space requirements.
- patapong 3y ago
  Opera used to do this! And then when you searched in the top bar, it would also search your visited pages. It was very useful.
deleted 3y ago
[deleted]
jondwillis 3y ago
That gets real messy real fast!

purplecats 3y ago

the ability to do this is pretty accessible nowadays, its just a little expensive for normal use for the layman

eagrwl 3y ago

Zenfetch (https://www.zenfetch.com/ https://www.zenfetch.com/) does an awesome job at this!

coolvision 3y ago

There is a nice opensource extension that does exactly that: https://github.com/memex-life/memex https://github.com/memex-life/memex I tried to use it, and it just does not work very well, I don't it's because implementation is bad, it's just RAG (retrieval based generation) does not work well outside of some simple use cases.

danielbln 3y ago
What are those simple use cases, and where do you see Retrieval Augmented Generation fall over?
- coolvision 3y ago
  I think it works better if query is a larger chunk of text. Like, if you have an email from a customer and want to compose a response based on some relevant documentation, it should work well. But for a use case where you want to retrieve something from browsing history you would mainly use a short search query, just few words. in this case embeddings are too ambiguous and relevance of retrieved content is not great.
  - janalsncm 3y ago
    That’s not a problem with RAG itself that’s an issue with your retriever. In the original RAG paper they used two vanilla BERT models and cosine similarity but there’s no requirement you do that. Use any retriever that gets you high precision. Use BM25 if you want, it’s simple and cheap. You’re right in saying there’s not enough semantic meaning in the text of the query. The domain of queries and the domain of documents are very different. That’s why a real retrieval system will train the query encoder and doc encoder to be closer in their embedding space using click data. This is what Google is doing.
    - ayaanmomin 3y ago
      "train the query encoder and doc encoder to be closer in their embedding space using click data" <- Any papers/resources you know where I can learn more about this process?
      - janalsncm 3y ago
        Triplet loss. https://pytorch.org/docs/stable/generated/torch.nn.TripletMarginLoss.html https://pytorch.org/docs/stable/generated/torch.nn.TripletMa... Triplet loss takes an anchor, positive, and negative. In this case the anchor is your query, the positive is a similar doc, and the negative is a dissimilar doc. When you train, backpropagate the loss to both the doc and the query encoder.

visarga 3y ago

alway wonder why browsers don't keep at least a text version of the pages for search nowadays I expect browsers to incorporate small LLMs like Mistral out of the box

coolvision 3y ago
if history search worked better then you would visit google/bing less, so I expect pushback from browser vendors also: https://bugs.chromium.org/p/chromium/issues/detail?id=297648 https://bugs.chromium.org/p/chromium/issues/detail?id=297648
- TeMPOraL 3y ago
  From a comment on that issue page, dating a little over a decade ago: > Shouldn't this be left to users to decide whether they want a feature or not. Yes, those were good times. Nowadays, it's anathema to software vendors, and increasingly even to open source devs.
- totetsu 3y ago
  Another advertising revenue win for Query Jerry from marketing.
- poyu 3y ago
  Not every browser vendor runs a search engine…
  - janalsncm 3y ago
    Apple and Mozilla aren’t Google. Apple runs their own searches on the App Store and iPhone, but Google pays them to be default. There’s no reason Apple couldn’t also index your search history in a more meaningful way other than perhaps space requirements.
patapong 3y ago
Opera used to do this! And then when you searched in the top bar, it would also search your visited pages. It was very useful.

deleted 3y ago

[deleted]

jondwillis 3y ago

That gets real messy real fast!