3 ms·
> When you're dealing with ten notes, it's trivial to dump them into an LLM and generate connections. With a hundred notes, it's still manageable. But what happ
by visarga 2y ago
> When you're dealing with ten notes, it's trivial to dump them into an LLM and generate connections. With a hundred notes, it's still manageable. But what happens when you hit 10,000 notes? Or a million? We quickly run into the limitations of context windows and processing capacity. Vector search helps narrow things down, but it's prone to missing important connections that a human mind might naturally make.
I run into the same problem. I made a RAG system and imported 15 years of reddit and HN content (my own message logs) + 2 years of LLM chats, totaling about 80MB of text. I can use it to retrieve fragments but there is duplication, and almost all duplicate fragments have slightly different approaches. How do I merge all of them, and how do I get a deduplicated taxonomy? I got about 290K unique keywords extracted from the text, it doesn't fit into LLMs.
I am gravitating towards building graphs of ideas, and having a way to generate unique (non duplicate) new ideas while ingesting new text.
- austinjp 2y agoIntriguing. I've been thinking of something similar. Do you have any notion of what constitutes an 'idea' in this context?
- visarga 2y agoI am using a LLM to extract ideas from text, as a numbered list, each one self contained, about 50-100 words long. It also generates relations between ideas (supporting, contradicting, expands on, etc...) The idea extraction step is simple when starting from scratch, but when you got megabytes of text deduplication becomes a problem. I like to circle back a lot.
- nsavage 2y agoAuthor here. Glad I'm not the only one with this problem! What I'm experimenting with is building a graph out of "entities", which I'm calling people, places and things found in each card. The point I guess is to allow the LLMs to make connections between cards, but this won't be replacing user made connections. This lets me pick up all cards that reference something like, I don't know, "factories" as a concept, which isn't something the user will necessarily want to directly connect. Downside is that deduplicated taxonomy: I have, from a scan through my list, "factories", "Factory" and "Factories". My attempt to head this off is by vector searching based on the proposed entity keywords and having an LLM decide if they are the same or not. I think I need to work on my prompt for this though.
- lukev 2y agoI have a thesis that RDF is good for this sort of work (I gave a talk on it at Clojure Conj last year, https://youtu.be/OxzUjpihIH4 https://youtu.be/OxzUjpihIH4 if you're interested.) It's good for bridging the gaps between LLMs, ontologies, traditional structured data, and logic programming.
- ketedrum 2y agoInteresting, thank you! Gonna watch this asap
- uoaei 2y agoDo you know of any good RDF datasets or repositories to look for? I am familiar with linked data / semantic web but have not kept up on developments for a long while now.
- zozbot234 2y agoYou can refer to the Linked Open Data (LOD) cloud for a network of openly available RDF datasets. Wikidata currently serves as the unofficial "hub" for the cloud, a role that was formerly played by DBpedia.
- spothedog1 2y agoyup I've been working on building a platform entirely built on RDF Ontologies. Excited to try to combine it with LLMs in the not-to-distant future.