3 ms·
How does it work for code? (Chunking code that is)
by samlinnfer 2y ago
How does it work for code? (Chunking code that is)
- nostrebored 2y agoPoorly, just like it does for text. Chunking is easily where all of these problems die beyond PoC scale. I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.
- potatoman22 2y agoWhat do they use BM25 for? RAG?
- nostrebored 2y agoCorrect -- finding the correct functions and files to include
- bhavnicksm 2y agoRight now, we haven't worked on adding support for code -- some things like comments (#, //) have punctuations that adversely affect chunking, along with indentation and other issues. But, it's on the roadmap, so please hold on!