3 ms·
Those are great links. I’ve been using: https://github.com/npiv/chatblade https://github.com/npiv/chatblade https://github.com/tbckr/sgpt https://github.com/tb
by alchemist1e9 3y ago
Those are great links. I’ve been using:
https://github.com/npiv/chatblade https://github.com/npiv/chatblade
https://github.com/tbckr/sgpt https://github.com/tbckr/sgpt
I totally agree with LLM+CLI are perfect fit.
One pattern I used recently was httrack + w3m dump + sgpt images with gpt vision to generate a 278K token specific knowledge base with a custom perl hack for a RAG that preserved the outline of the knowledge.
Which brings me to my question for you - have you seen anything unix philosophy aligned for processing inputs and doing RAG locally?
EDIT. Turns out OP has done quite a bit toward what I’m asking. Written up here:
https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/
Something I’m currently a bit hung up on is finding a toolchain for chunking content on which to create embeddings. Ideally it would detect location context, like section “2.1 Failover” or “Chapter 8: The dream” or whatever from the original text, also handle 80 character wide source unwrapping, smart splitting so paragraphs are kept together, etc etc.
- levidos 3y agoI'm also a very heavy daily user of a similar tool, shell_gpt. Not having it would feel like a limb missing, very odd. https://github.com/TheR1D/shell_gpt https://github.com/TheR1D/shell_gpt
- simonw 3y agoThat's the same problem I haven't figured out yet: the best strategies for chunking. I'm hoping good, well proven patterns emerge soon so I can integrate them into my various tools.
- alchemist1e9 3y agoI’ve been looking at this https://freeling-user-manual.readthedocs.io/en/v4.2/modules/splitter/ https://freeling-user-manual.readthedocs.io/en/v4.2/modules/... at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly. There is also https://github.com/Unstructured-IO/unstructured https://github.com/Unstructured-IO/unstructured But I don’t like it, can’t explain why yet. My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text. BRIO/BERT summarization could also have a role of some type. Those are my ideas so far.
- roomey 3y agoA quick glance at "embedding" reminds me a lot of some work I was doing on quantum computing. I wonder if there is some crossover potential there, in terms of calculations across vector arrays