3 ms·

Really what we need is AI parsed and driven collective memory indexing and access. Search queries, and more importantly search engines, are a very poor mechani

by kromem 3y ago

Really what we need is AI parsed and driven collective memory indexing and access. Search queries, and more importantly search engines, are a very poor mechanism for referencing existing information. We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts that are designed with a LLM in mind as the end user, and not my grandmother. With the proper "collective long term memory" in place, we're going to see a leap in LLM capabilities all over again. Right now LLMs are like humans after language but before writing - creating very clever things in a single generation, but unable to persist ideas to compound by future iterations. When that becomes a solved problem (in guessing less than 12 months) it's going to be nuts, especially with something like a 50 page prompt size to stuff relevant fragments of a collective memory within.

masswerk 3y ago
> We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts But, how could this ever be better and/or more accurate as, say, Wikipedia (as a flagship for collective curated content efforts)? A single, reliable source of truth and context is very much a dream. Moreover, a collective long-term memory for LLMs would just ring the bell for a next-generation SEO industry.
- gaogao 3y ago
  WikiData also works pretty well too - https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... I've been thinking that you probably can do something better than just embedding search on paragraphs, since you want coverage, not just narrow semantically similar things to the question. Decomposition can help with that, but you might be able to do something more clever, say pasted into the model itself. I think possibly also training a foundational model with context always available could be a potential path to getting more efficient models and might having interesting behavior around hallucinations. (You'd do unsupervised training that's also doing semantic search, not just inference)
  - masswerk 3y ago
    Mind that none of this is a technical problem. There is just no way of having a commonly agreed upon data source, especially, when it comes to short summarizing definitions, and there will be always errors involved, lack of understanding or expertise, contesting sources, coercing source using different terminology, varying world views, etc. (E.g., ask any number of economists what caused the 2008 financial crises, and you'll get about the same number of non-matching answers, from laissez-fair to state intervention, subprime credits or public debt, whatever supports the particular narrative.)

by kromem 3y ago

masswerk 3y ago

> We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts But, how could this ever be better and/or more accurate as, say, Wikipedia (as a flagship for collective curated content efforts)? A single, reliable source of truth and context is very much a dream. Moreover, a collective long-term memory for LLMs would just ring the bell for a next-generation SEO industry.

gaogao 3y ago
WikiData also works pretty well too - https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... I've been thinking that you probably can do something better than just embedding search on paragraphs, since you want coverage, not just narrow semantically similar things to the question. Decomposition can help with that, but you might be able to do something more clever, say pasted into the model itself. I think possibly also training a foundational model with context always available could be a potential path to getting more efficient models and might having interesting behavior around hallucinations. (You'd do unsupervised training that's also doing semantic search, not just inference)
- masswerk 3y ago
  Mind that none of this is a technical problem. There is just no way of having a commonly agreed upon data source, especially, when it comes to short summarizing definitions, and there will be always errors involved, lack of understanding or expertise, contesting sources, coercing source using different terminology, varying world views, etc. (E.g., ask any number of economists what caused the 2008 financial crises, and you'll get about the same number of non-matching answers, from laissez-fair to state intervention, subprime credits or public debt, whatever supports the particular narrative.)