4 ms·
what do you use to create your vectorized chunks?
by sharemywin 3y ago
what do you use to create your vectorized chunks?
- isoprophlex 3y agoOpenAI's stuff, no experience with local models. I'm curious to hear from people that have compared between those.
- kordlessagain 3y agoCheck out Instructor Large.
- J_Shelby_J 3y agoIt's pretty straight forward. Pre-process -> chunk -> openai embedding endpoint I found langchain not great, so ended up rewritting most of it for my chat bot. https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8299dd0c03d1c5576cdc84fa1377b475ed6ff/app/services/index_service.py#L348 https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8...
- sharemywin 3y agoSorry I meant the chunk part. do you do anything special with trying to break the text into manageable chunks. TextSplitter in Langchain just uses some line breaks stuff I guess. --- The default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""] --- https://python.langchain.com/docs/modules/data_connection/document_transformers/ https://python.langchain.com/docs/modules/data_connection/do...