3 ms·
It's pretty straight forward. Pre-process -> chunk -> openai embedding endpoint I found langchain not great, so ended up rewritting most of it for my chat bot
by J_Shelby_J 3y ago
It's pretty straight forward.
Pre-process -> chunk -> openai embedding endpoint
I found langchain not great, so ended up rewritting most of it for my chat bot.
https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8299dd0c03d1c5576cdc84fa1377b475ed6ff/app/services/index_service.py#L348 https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8...
- sharemywin 3y agoSorry I meant the chunk part. do you do anything special with trying to break the text into manageable chunks. TextSplitter in Langchain just uses some line breaks stuff I guess. --- The default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""] --- https://python.langchain.com/docs/modules/data_connection/document_transformers/ https://python.langchain.com/docs/modules/data_connection/do...