3 ms·
Sweep is mentioned as attribution in multiple places a) https://github.com/definitive-io/code-indexer-loop#attribution https://github.com/definitive-io/code-ind
by ricklamers 3y ago
Sweep is mentioned as attribution in multiple places a) https://github.com/definitive-io/code-indexer-loop#attribution https://github.com/definitive-io/code-indexer-loop#attributi... b) https://github.com/definitive-io/code-indexer-loop/blob/fd9d4a65e475e9f41e782af7cedea3d0bc675af5/code_indexer_loop/code_splitter.py#L3 https://github.com/definitive-io/code-indexer-loop/blob/fd9d...
The difference is packaging it as a consumable PyPI package that can easily be used in a project (they even call out for separating this out into a stand alone project but that they lack the time to do so: https://docs.sweep.dev/blogs/chunking-2m-files#future- https://docs.sweep.dev/blogs/chunking-2m-files#future- )
In addition, we expand and fix the implementation, for example it now supports limiting on token count instead of character count, and we fix some white space inconsistencies in parsing/chunk reconstruction.
- _boffin_ 3y agoLooking forward to diving further in and seeing how you handle white spaces issues as I’ve had massive issues with that already
- ricklamers 3y agoPlease let me know if anything isn’t as expected and I’ll try to look into it