4 ms·
This is really cool and a much needed contribution to helping LLMs run better on large code bases.
by yding 3y ago
This is really cool and a much needed contribution to helping LLMs run better on large code bases.
- kevinlu1248 3y agoThanks! Would love to see this algorithm in LlamaIndex.
- thomasfromcdnjs 3y agoAlso thank you! I had an idea of how this all worked but this definitely concreted my thoughts.
- kevinlu1248 3y agoI'm glad to hear! Let me know if you need more questions or are interested in implementing this chunking algo.
- sandGorgon 3y agooh really ? Thats awfully kind. I'll take that in for EdgeChains as well. https://github.com/arakoodev/EdgeChains/issues/172 https://github.com/arakoodev/EdgeChains/issues/172
- kevinlu1248 3y agoFeel free to use the algo! Happy to help and reach out if you need help with implementation!
- kevinlu1248 3y agoFor others in this thread, this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89bfdc7e1c62fd456648cc59d7480/llama_index/langchain_helpers/text_splitter.py#L476-L559C1 https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...