3 ms·
You can use the AST to chunk the code: https://docs.sweep.dev/blogs/chunking-2m-files https://docs.sweep.dev/blogs/chunking-2m-files
by jshobrook 2y ago
You can use the AST to chunk the code: https://docs.sweep.dev/blogs/chunking-2m-files https://docs.sweep.dev/blogs/chunking-2m-files
- CuriousJ 2y agoWe're using an improvement over this exact blogpost actually. We started from there, but weren't happy that some of the chunks were really small (and they would undeservedly get surfaced to the top). So we added some extra logic to merge the siblings if they're small. https://github.com/Storia-AI/repo2vec/blob/1864102949e7203205c3ab855cc9a3918494a265/src/chunker.py#L112 https://github.com/Storia-AI/repo2vec/blob/1864102949e720320...