I am using a LLM to extract ideas from text, as a numbered list, each one self contained, about 50-100 words long. It also generates relations between ideas (su

I am using a LLM to extract ideas from text, as a numbered list, each one self contained, about 50-100 words long. It also generates relations between ideas (supporting, contradicting, expands on, etc...) The idea extraction step is simple when starting from scratch, but when you got megabytes of text deduplication becomes a problem. I like to circle back a lot.