4 ms·
The original llama uses google's tooling for that, written in C++ https://github.com/google/sentencepiece https://github.com/google/sentencepiece
by Const-me 4y ago
The original llama uses google's tooling for that, written in C++ https://github.com/google/sentencepiece https://github.com/google/sentencepiece